### Tags: basically, fit, fmincon, fminsearch, matlab, programming

# fminsearch and fmincon

On Programmer » Matlab

14,686 words with 7 Comments; publish: Tue, 06 May 2008 16:38:00 GMT; (200125.00, « »)

Hello,

I have a question about fminsearch vs fmincon.

Basically my problem is trying to fit some data with the equation:

(A*yval_1+B*yval_2)*exp(-C+D*x)

I was calling fminsearch with the parameters:

fminsearch('fitting_function',[A B C D], [], xvals, yval_1, yval_2)

where fitting_function.m had the equation above and basically found

the error between the function and my data for values of A, B, C and

D (which fminsearch worked to find)...

Now I'm working with a dataset in which A and B need to be

constrained:

A+B=1 and A,B={0..1}

but C and D must not be constrained.

I know there's a fmincon function, but I'm not sure of how to use it

in my situation. How can I have A and B be constrained, and let C

and D float?

I read the matlab help, but I'm still . Is there a tutorial

somewhere that covers this?

Thanks.

*http://matlab.questionfor.info/q_matlab_18713.html*

All Comments

Leave a comment...

- 7 Comments
- In article <ef0e9c7.-1.matlab.questionfor.info.webx.raydaftYaTP>,
Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

> Hello,

> I have a question about fminsearch vs fmincon.

> Basically my problem is trying to fit some data with the equation:

> (A*yval_1+B*yval_2)*exp(-C+D*x)

> I was calling fminsearch with the parameters:

> fminsearch('fitting_function',[A B C D], [], xvals, yval_1, yval_2)

> where fitting_function.m had the equation above and basically found

> the error between the function and my data for values of A, B, C and

> D (which fminsearch worked to find)...

> Now I'm working with a dataset in which A and B need to be

> constrained:

> A+B=1 and A,B={0..1}

> but C and D must not be constrained.

> I know there's a fmincon function, but I'm not sure of how to use it

> in my situation. How can I have A and B be constrained, and let C

> and D float?

> I read the matlab help, but I'm still . Is there a tutorial

> somewhere that covers this?

> Thanks.

There is no unique solution to the coefficients you are

trying to estimate. This will cause problems for any

optimizer. At least this is so if you do not pose that

sum constraint on A and B. Consider that any value of C

can be absorbed into A and B. The problems you will see

are lack of convergence to a consistent solution from

different sets of starting values.

With the constraint there is no longer a problem. But

this constraint is trivial to impose. Since A+B=1,

simply eliminate B in your model.

(A*yval_1+(1-A)*yval_2)*exp(-C+D*x)

Note that C can no longer be taken out of the exponential

to absorb into A, because of the 1-A term.

If you really don't want to bother with fmincon, then

why not do one more transformtion (hack if you prefer)

to your model? If A + B = 1, then transform the problem

into a related one where

A = [sin(theta)]^2

B = [cos(theta)]^2

Note that A + B == 1 identically, and A and B are

always contained in the interval [0,1]. In effect

we have turned a 4 variable problem into a 3 variable

one.

([sin(theta)]^2*yval_1+[cos(theta)]^2*yv

al_2)*exp(-C+D*x)

Theta has no constraints upon it at all, but the

coefficients of yval_1 and yval_2 are implicitly

constrained. And, you can go back to fminsearch

with no mental energy consumed.

HTH,

John D'Errico

The best material model of a cat is another, or

preferably the same, cat.

A. Rosenblueth, Philosophy of Science, 1945

#1; Tue, 06 May 2008 16:39:00 GMT

- In article <ef0e9c7.-1.matlab.questionfor.info.webx.raydaftYaTP>,
- >
> There is no unique solution to the coefficients you are

> trying to estimate. This will cause problems for any

> optimizer. At least this is so if you do not pose that

> sum constraint on A and B. Consider that any value of C

> can be absorbed into A and B. The problems you will see

> are lack of convergence to a consistent solution from

> different sets of starting values.

> With the constraint there is no longer a problem. But

> this constraint is trivial to impose. Since A+B=1,

> simply eliminate B in your model.

> (A*yval_1+(1-A)*yval_2)*exp(-C+D*x)

> Note that C can no longer be taken out of the exponential

> to absorb into A, because of the 1-A term.

> If you really don't want to bother with fmincon, then

> why not do one more transformtion (hack if you prefer)

> to your model? If A + B = 1, then transform the problem

> into a related one where

> A = [sin(theta)]^2

> B = [cos(theta)]^2

> Note that A + B == 1 identically, and A and B are

> always contained in the interval [0,1]. In effect

> we have turned a 4 variable problem into a 3 variable

> one.

> ([sin(theta)]^2*yval_1+[cos(theta)]^2*yv

al_2)*exp(-C+D*x)

> Theta has no constraints upon it at all, but the

> coefficients of yval_1 and yval_2 are implicitly

> constrained. And, you can go back to fminsearch

> with no mental energy consumed.

> HTH,

> John D'Errico

>

Thanks for responding John!

I guess I was asking the question in the wrong way. More generally,

what is the right way to solve an equation with say 3 (or 4) unknowns

when I need to constrain 1 (or 2) of the unknowns within some limits

(and let the other unknowns float)... I've never really used the

fmincon function and am a bit unsure of how to proceed. I'm not

looking for an answer to the specific problem I posed (in the

original post), but rather a tutorial on the uses and differences

between fminsearch and fmincon...(so I better know how to apply

it)...

thanks again!

#2; Tue, 06 May 2008 16:40:00 GMT

- >
- In article <ef0e9c7.1.matlab.questionfor.info.webx.raydaftYaTP>,
Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

> I guess I was asking the question in the wrong way. More generally,

> what is the right way to solve an equation with say 3 (or 4) unknowns

> when I need to constrain 1 (or 2) of the unknowns within some limits

> (and let the other unknowns float)... I've never really used the

> fmincon function and am a bit unsure of how to proceed. I'm not

> looking for an answer to the specific problem I posed (in the

> original post), but rather a tutorial on the uses and differences

> between fminsearch and fmincon...(so I better know how to apply

> it)...

This is what the lb and ub inputs to fmincon do.

They provide bound constraints.

If a variable is unconstrained, then use +/- inf.

As an example, if x1 must lie in [0,3], and x3>=2,

but x2 and x4 are unconstrained, then

lb = [0 -inf 2 -inf]

ub = [3 inf inf inf]

As for the differences between the two optimizers,

that would take a book. Fminsearch is a Nelder/Mead

polytope method. (Some call it a simplex method,

but then someone always gets that it does

linear programming. Polytope has no such ambiguity.)

These simplex/polytope methods work by reflecting

a simplex (think triangle in 2-dimensions) around

the space. The simplex walks around, testing the

objective at each point. They walk AWAY from high

values, rather than towards low ones. There is a

difference.

Fmincon is a more sophisticated method, which

requires local knowledge of the gradient of the

surface. It computes the gradient using a finite

difference approximation, or you can supply it

yourself.

Fminsearch is limited in the number of unknowns you

can solve for, because it gets very slow for high

dimensional problems. I'd rarely (if ever) use it

for more than about 5 or 6 unknowns. Its fine for

2d or 3d cases that I will not need to solve often.

Think of fminsearch as the 1960's era VW beetle.

Its a fun little car that will get you where you

want to go, as long as you don't want to get there

too fast, or carry too many bags in the trunk. Its

easy enough to repair when there are problems - no

fancy diagnostic tools required and a simple shop

manual. Don't expect much in the way of heat in the

winter or power windows and seats.

On the other hand, fmincon is a new car. Its got

gadgets like power windows, heated seats that adjust

at the touch of a button, dvd players in the back

seat for the kids, gps mapping for the navigator.

It will get you up to 75 on the expressway in a

matter of seconds (not that I would ever exceed

the posted speed limit.) Its a little touchy under

the hood though.

HTH,

John

The best material model of a cat is another, or

preferably the same, cat.

A. Rosenblueth, Philosophy of Science, 1945

#3; Tue, 06 May 2008 16:41:00 GMT

- In article <ef0e9c7.1.matlab.questionfor.info.webx.raydaftYaTP>,
- Thanks for responding again, John!
> This is what the lb and ub inputs to fmincon do.

> They provide bound constraints.

> If a variable is unconstrained, then use +/- inf.

> As an example, if x1 must lie in [0,3], and x3>=2,

> but x2 and x4 are unconstrained, then

> lb = [0 -inf 2 -inf]

> ub = [3 inf inf inf]

>

I guess I just need this kind of hand-holding... I've never used

fmincon before, so this is all new to me. However, this is exactly

what I was looking for!

> As for the differences between the two optimizers,

> that would take a book. Fminsearch is a Nelder/Mead

> polytope method. (Some call it a simplex method,

> but then someone always gets that it does

> linear programming. Polytope has no such ambiguity.)

> These simplex/polytope methods work by reflecting

> a simplex (think triangle in 2-dimensions) around

> the space. The simplex walks around, testing the

> objective at each point. They walk AWAY from high

> values, rather than towards low ones. There is a

> difference.

> Fmincon is a more sophisticated method, which

> requires local knowledge of the gradient of the

> surface. It computes the gradient using a finite

> difference approximation, or you can supply it

> yourself.

> Fminsearch is limited in the number of unknowns you

> can solve for, because it gets very slow for high

> dimensional problems. I'd rarely (if ever) use it

> for more than about 5 or 6 unknowns. Its fine for

> 2d or 3d cases that I will not need to solve often.

> Think of fminsearch as the 1960's era VW beetle.

> Its a fun little car that will get you where you

> want to go, as long as you don't want to get there

> too fast, or carry too many bags in the trunk. Its

> easy enough to repair when there are problems - no

> fancy diagnostic tools required and a simple shop

> manual. Don't expect much in the way of heat in the

> winter or power windows and seats.

> On the other hand, fmincon is a new car. Its got

> gadgets like power windows, heated seats that adjust

> at the touch of a button, dvd players in the back

> seat for the kids, gps mapping for the navigator.

> It will get you up to 75 on the expressway in a

> matter of seconds (not that I would ever exceed

> the posted speed limit.) Its a little touchy under

> the hood though.

> HTH,

> John

>

This really helped! thanks again! (heh, I really enjoyed the

analogy!) I've just always stuck with fminsearch (since that's what

I started using way back when) and I've never had a reason to

question it's abilities until now. It't probably time to learn some

new tricks!

A few more questions though. I'm getting a feel for constraining a

single variable bewteen bounds, but I'm still unsure of how to

express a linear combination of variables with constraint:

for instance - four variables, a(1) a(2) a(3) a(4)

0 <= a(1) <= 1 and 0 <= a(2) <= 1 while a(3) and a(4)

float,

I'll use the ub and lb as you stated above...

but also I may need:

a(1) + a(2) = 1

(or some other linear combination of variables equal to a constant)

Also, how do I pass values that are needed for fitting to fmincon.

When using fminsearch, variables that were in the first group were

minimized to fit, while everything else was passed as static

variables or arguments:

fminsearch('fitting_function',[A B C D], [], xvals, other_args,

etc...)

so [A B C D] are minimized from their starting values, while

xvals, other_args, etc... are static yet needed to fit A, B, C, and D

(or had other effects on the .m file)

I think what I need is a tutorial which explains how to put together

the correct syntax for fmincon vs fminsearch for specific problems

(the matlab help didn't help me much - maybe I'm too dense for

it!)...

Thanks again!

#4; Tue, 06 May 2008 16:42:00 GMT

- Thanks for responding again, John!
- John,
I think I've gotten the hang of fmincon! Something just clicked,

and it all seems to work now... fmincon IS extremely more

useful/powerful as compared to fminsearch! Thanks for all your

help. I just need to find some more constraints for my fitting now.

Thanks again!

#5; Tue, 06 May 2008 16:44:00 GMT

- John,
- In article <ef0e9c7.4.matlab.questionfor.info.webx.raydaftYaTP>,
Justin_Time <justin_time.matlab.questionfor.info.mailinator.com> wrote:

> John,

> I think I've gotten the hang of fmincon! Something just clicked,

> and it all seems to work now... fmincon IS extremely more

> useful/powerful as compared to fminsearch! Thanks for all your

> help. I just need to find some more constraints for my fitting now.

> Thanks again!

Yes. An "AHA!" moment. They are nice.

Ask again if you have any questions about using

constraints.

John

The best material model of a cat is another, or

preferably the same, cat.

A. Rosenblueth, Philosophy of Science, 1945

#6; Tue, 06 May 2008 16:44:00 GMT

- In article <ef0e9c7.4.matlab.questionfor.info.webx.raydaftYaTP>,
- You can still pass extra arguments to fmincon like you did to fminsearch tha
t will be passed on to the fitting function when fmincon calls it. I haven't

noticed that this is mentioned anywhere in the help files though.

Seth

#7; Tue, 06 May 2008 16:46:00 GMT

- You can still pass extra arguments to fmincon like you did to fminsearch tha