Tags: basically, fit, fmincon, fminsearch, matlab, programming

fminsearch and fmincon

On Programmer » Matlab

14,686 words with 7 Comments; publish: Tue, 06 May 2008 16:38:00 GMT; (200248.05, « »)

Hello,

I have a question about fminsearch vs fmincon.

Basically my problem is trying to fit some data with the equation:

(A*yval_1+B*yval_2)*exp(-C+D*x)

I was calling fminsearch with the parameters:

fminsearch('fitting_function',[A B C D], [], xvals, yval_1, yval_2)

where fitting_function.m had the equation above and basically found

the error between the function and my data for values of A, B, C and

D (which fminsearch worked to find)...

Now I'm working with a dataset in which A and B need to be

constrained:

A+B=1 and A,B={0..1}

but C and D must not be constrained.

I know there's a fmincon function, but I'm not sure of how to use it

in my situation. How can I have A and B be constrained, and let C

and D float?

I read the matlab help, but I'm still . Is there a tutorial

somewhere that covers this?

Thanks.

All Comments

Leave a comment...

  • 7 Comments
    • In article <ef0e9c7.-1.matlab.questionfor.info.webx.raydaftYaTP>,

      Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

      > Hello,

      > I have a question about fminsearch vs fmincon.

      > Basically my problem is trying to fit some data with the equation:

      > (A*yval_1+B*yval_2)*exp(-C+D*x)

      > I was calling fminsearch with the parameters:

      > fminsearch('fitting_function',[A B C D], [], xvals, yval_1, yval_2)

      > where fitting_function.m had the equation above and basically found

      > the error between the function and my data for values of A, B, C and

      > D (which fminsearch worked to find)...

      > Now I'm working with a dataset in which A and B need to be

      > constrained:

      > A+B=1 and A,B={0..1}

      > but C and D must not be constrained.

      > I know there's a fmincon function, but I'm not sure of how to use it

      > in my situation. How can I have A and B be constrained, and let C

      > and D float?

      > I read the matlab help, but I'm still . Is there a tutorial

      > somewhere that covers this?

      > Thanks.

      There is no unique solution to the coefficients you are

      trying to estimate. This will cause problems for any

      optimizer. At least this is so if you do not pose that

      sum constraint on A and B. Consider that any value of C

      can be absorbed into A and B. The problems you will see

      are lack of convergence to a consistent solution from

      different sets of starting values.

      With the constraint there is no longer a problem. But

      this constraint is trivial to impose. Since A+B=1,

      simply eliminate B in your model.

      (A*yval_1+(1-A)*yval_2)*exp(-C+D*x)

      Note that C can no longer be taken out of the exponential

      to absorb into A, because of the 1-A term.

      If you really don't want to bother with fmincon, then

      why not do one more transformtion (hack if you prefer)

      to your model? If A + B = 1, then transform the problem

      into a related one where

      A = [sin(theta)]^2

      B = [cos(theta)]^2

      Note that A + B == 1 identically, and A and B are

      always contained in the interval [0,1]. In effect

      we have turned a 4 variable problem into a 3 variable

      one.

      ([sin(theta)]^2*yval_1+[cos(theta)]^2*yv

      al_2)*exp(-C+D*x)

      Theta has no constraints upon it at all, but the

      coefficients of yval_1 and yval_2 are implicitly

      constrained. And, you can go back to fminsearch

      with no mental energy consumed.

      HTH,

      John D'Errico

      The best material model of a cat is another, or

      preferably the same, cat.

      A. Rosenblueth, Philosophy of Science, 1945

      #1; Tue, 06 May 2008 16:39:00 GMT
    • >

      > There is no unique solution to the coefficients you are

      > trying to estimate. This will cause problems for any

      > optimizer. At least this is so if you do not pose that

      > sum constraint on A and B. Consider that any value of C

      > can be absorbed into A and B. The problems you will see

      > are lack of convergence to a consistent solution from

      > different sets of starting values.

      > With the constraint there is no longer a problem. But

      > this constraint is trivial to impose. Since A+B=1,

      > simply eliminate B in your model.

      > (A*yval_1+(1-A)*yval_2)*exp(-C+D*x)

      > Note that C can no longer be taken out of the exponential

      > to absorb into A, because of the 1-A term.

      > If you really don't want to bother with fmincon, then

      > why not do one more transformtion (hack if you prefer)

      > to your model? If A + B = 1, then transform the problem

      > into a related one where

      > A = [sin(theta)]^2

      > B = [cos(theta)]^2

      > Note that A + B == 1 identically, and A and B are

      > always contained in the interval [0,1]. In effect

      > we have turned a 4 variable problem into a 3 variable

      > one.

      > ([sin(theta)]^2*yval_1+[cos(theta)]^2*yv

      al_2)*exp(-C+D*x)

      > Theta has no constraints upon it at all, but the

      > coefficients of yval_1 and yval_2 are implicitly

      > constrained. And, you can go back to fminsearch

      > with no mental energy consumed.

      > HTH,

      > John D'Errico

      >

      Thanks for responding John!

      I guess I was asking the question in the wrong way. More generally,

      what is the right way to solve an equation with say 3 (or 4) unknowns

      when I need to constrain 1 (or 2) of the unknowns within some limits

      (and let the other unknowns float)... I've never really used the

      fmincon function and am a bit unsure of how to proceed. I'm not

      looking for an answer to the specific problem I posed (in the

      original post), but rather a tutorial on the uses and differences

      between fminsearch and fmincon...(so I better know how to apply

      it)...

      thanks again!

      #2; Tue, 06 May 2008 16:40:00 GMT
    • In article <ef0e9c7.1.matlab.questionfor.info.webx.raydaftYaTP>,

      Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

      > I guess I was asking the question in the wrong way. More generally,

      > what is the right way to solve an equation with say 3 (or 4) unknowns

      > when I need to constrain 1 (or 2) of the unknowns within some limits

      > (and let the other unknowns float)... I've never really used the

      > fmincon function and am a bit unsure of how to proceed. I'm not

      > looking for an answer to the specific problem I posed (in the

      > original post), but rather a tutorial on the uses and differences

      > between fminsearch and fmincon...(so I better know how to apply

      > it)...

      This is what the lb and ub inputs to fmincon do.

      They provide bound constraints.

      If a variable is unconstrained, then use +/- inf.

      As an example, if x1 must lie in [0,3], and x3>=2,

      but x2 and x4 are unconstrained, then

      lb = [0 -inf 2 -inf]

      ub = [3 inf inf inf]

      As for the differences between the two optimizers,

      that would take a book. Fminsearch is a Nelder/Mead

      polytope method. (Some call it a simplex method,

      but then someone always gets that it does

      linear programming. Polytope has no such ambiguity.)

      These simplex/polytope methods work by reflecting

      a simplex (think triangle in 2-dimensions) around

      the space. The simplex walks around, testing the

      objective at each point. They walk AWAY from high

      values, rather than towards low ones. There is a

      difference.

      Fmincon is a more sophisticated method, which

      requires local knowledge of the gradient of the

      surface. It computes the gradient using a finite

      difference approximation, or you can supply it

      yourself.

      Fminsearch is limited in the number of unknowns you

      can solve for, because it gets very slow for high

      dimensional problems. I'd rarely (if ever) use it

      for more than about 5 or 6 unknowns. Its fine for

      2d or 3d cases that I will not need to solve often.

      Think of fminsearch as the 1960's era VW beetle.

      Its a fun little car that will get you where you

      want to go, as long as you don't want to get there

      too fast, or carry too many bags in the trunk. Its

      easy enough to repair when there are problems - no

      fancy diagnostic tools required and a simple shop

      manual. Don't expect much in the way of heat in the

      winter or power windows and seats.

      On the other hand, fmincon is a new car. Its got

      gadgets like power windows, heated seats that adjust

      at the touch of a button, dvd players in the back

      seat for the kids, gps mapping for the navigator.

      It will get you up to 75 on the expressway in a

      matter of seconds (not that I would ever exceed

      the posted speed limit.) Its a little touchy under

      the hood though.

      HTH,

      John

      The best material model of a cat is another, or

      preferably the same, cat.

      A. Rosenblueth, Philosophy of Science, 1945

      #3; Tue, 06 May 2008 16:41:00 GMT
    • Thanks for responding again, John!

      > This is what the lb and ub inputs to fmincon do.

      > They provide bound constraints.

      > If a variable is unconstrained, then use +/- inf.

      > As an example, if x1 must lie in [0,3], and x3>=2,

      > but x2 and x4 are unconstrained, then

      > lb = [0 -inf 2 -inf]

      > ub = [3 inf inf inf]

      >

      I guess I just need this kind of hand-holding... I've never used

      fmincon before, so this is all new to me. However, this is exactly

      what I was looking for!

      > As for the differences between the two optimizers,

      > that would take a book. Fminsearch is a Nelder/Mead

      > polytope method. (Some call it a simplex method,

      > but then someone always gets that it does

      > linear programming. Polytope has no such ambiguity.)

      > These simplex/polytope methods work by reflecting

      > a simplex (think triangle in 2-dimensions) around

      > the space. The simplex walks around, testing the

      > objective at each point. They walk AWAY from high

      > values, rather than towards low ones. There is a

      > difference.

      > Fmincon is a more sophisticated method, which

      > requires local knowledge of the gradient of the

      > surface. It computes the gradient using a finite

      > difference approximation, or you can supply it

      > yourself.

      > Fminsearch is limited in the number of unknowns you

      > can solve for, because it gets very slow for high

      > dimensional problems. I'd rarely (if ever) use it

      > for more than about 5 or 6 unknowns. Its fine for

      > 2d or 3d cases that I will not need to solve often.

      > Think of fminsearch as the 1960's era VW beetle.

      > Its a fun little car that will get you where you

      > want to go, as long as you don't want to get there

      > too fast, or carry too many bags in the trunk. Its

      > easy enough to repair when there are problems - no

      > fancy diagnostic tools required and a simple shop

      > manual. Don't expect much in the way of heat in the

      > winter or power windows and seats.

      > On the other hand, fmincon is a new car. Its got

      > gadgets like power windows, heated seats that adjust

      > at the touch of a button, dvd players in the back

      > seat for the kids, gps mapping for the navigator.

      > It will get you up to 75 on the expressway in a

      > matter of seconds (not that I would ever exceed

      > the posted speed limit.) Its a little touchy under

      > the hood though.

      > HTH,

      > John

      >

      This really helped! thanks again! (heh, I really enjoyed the

      analogy!) I've just always stuck with fminsearch (since that's what

      I started using way back when) and I've never had a reason to

      question it's abilities until now. It't probably time to learn some

      new tricks!

      A few more questions though. I'm getting a feel for constraining a

      single variable bewteen bounds, but I'm still unsure of how to

      express a linear combination of variables with constraint:

      for instance - four variables, a(1) a(2) a(3) a(4)

      0 <= a(1) <= 1 and 0 <= a(2) <= 1 while a(3) and a(4)

      float,

      I'll use the ub and lb as you stated above...

      but also I may need:

      a(1) + a(2) = 1

      (or some other linear combination of variables equal to a constant)

      Also, how do I pass values that are needed for fitting to fmincon.

      When using fminsearch, variables that were in the first group were

      minimized to fit, while everything else was passed as static

      variables or arguments:

      fminsearch('fitting_function',[A B C D], [], xvals, other_args,

      etc...)

      so [A B C D] are minimized from their starting values, while

      xvals, other_args, etc... are static yet needed to fit A, B, C, and D

      (or had other effects on the .m file)

      I think what I need is a tutorial which explains how to put together

      the correct syntax for fmincon vs fminsearch for specific problems

      (the matlab help didn't help me much - maybe I'm too dense for

      it!)...

      Thanks again!

      #4; Tue, 06 May 2008 16:42:00 GMT
    • John,

      I think I've gotten the hang of fmincon! Something just clicked,

      and it all seems to work now... fmincon IS extremely more

      useful/powerful as compared to fminsearch! Thanks for all your

      help. I just need to find some more constraints for my fitting now.

      Thanks again!

      #5; Tue, 06 May 2008 16:44:00 GMT
    • In article <ef0e9c7.4.matlab.questionfor.info.webx.raydaftYaTP>,

      Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

      > John,

      > I think I've gotten the hang of fmincon! Something just clicked,

      > and it all seems to work now... fmincon IS extremely more

      > useful/powerful as compared to fminsearch! Thanks for all your

      > help. I just need to find some more constraints for my fitting now.

      > Thanks again!

      Yes. An "AHA!" moment. They are nice.

      Ask again if you have any questions about using

      constraints.

      John

      The best material model of a cat is another, or

      preferably the same, cat.

      A. Rosenblueth, Philosophy of Science, 1945

      #6; Tue, 06 May 2008 16:44:00 GMT
    • You can still pass extra arguments to fmincon like you did to fminsearch tha

      t will be passed on to the fitting function when fmincon calls it. I haven't

      noticed that this is mentioned anywhere in the help files though.

      Seth

      #7; Tue, 06 May 2008 16:46:00 GMT