1. By Liyun Zhang Aug.9 th,2012

2. * Method for Single Variable Unconstraint Optimization : 1. Quadratic Interpolation method 2. Golden Section method * Method for Multivariable Constraint Optimization Method 1. Lagrange Multiplier Method 2. Exterior-point Penalty Method

3. * Solve for: unconstraint optimization problems with single variable and unimodal function * Advantage and Disadvantage: converges fast but UNRILIABLE (you will see it in a later example)

4. target function: f(x) select three points x0, x1, x2 on f(x) such that: x0<x1<x2, f2=min(f0, f1, f2) there must exist a quadratic function: q(x)=a0+a1x+a2x^2 which connects these points such that: q(x0)=a0+a1x0+a2x0^2=f(x0) q(x1)=a0+a1x1+a2x1^2=f(x1) q(x2)=a0+a1x2+a2x2^2=f(x2)

6. Initialization: Determine x0, x1 and x2 which is known to contain the minimum of the function f(x). Step 1 If x3-x1<ε (a sufficiently small number), then the minimum occurs at xo=x3 and stop iteration, else go to Step 2. Step 2 Evaluate x3 and x1 If x3<x1, go to Step 3, else go to Step 4. Step 3 Evaluate f(x3) and f(x1) If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x0 x2=x1 x1=x3 If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x3 x1=x1 x2=x2

7. Step 4 Evaluate f(x3) and f(x1) If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x1 x1=x3 x2=x2 If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x0 x1=x1 x2=x3 Step 5 Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3. code illustration

8. Example 1. Find the minimum point and value of the function f(x)=(x^2-2)^2/2-1 Solution: f1001=inline('(x.*x-2).^2/2-1','x'); a=0; b=5; TolX=1e-5; TolFun=1e-8; MaxIter=100; [xoq,foq]=Opt_Quadratic(f1001,[a,b],TolX,TolFun,MaxIter) xoq = 1.414187140007763 foq = -0.999999997207487

9. Fminbnd method attempts to solve the optimum of one variable function within a fixed interval. The most common syntaxes are: [x, f]=fminbnd (fun, x1, x2) [x, f]=fminbnd (fun, x1, x2, options) In Ex. 1, we have x1=0 x2=5 and the result is: x = 1.414214286263162 f = -0.999999999997904

11. * Solve for: unconstraint optimization problems with single variable and unimodal function * Advantages: 1. more reliable than quadratic interpolation method 2. still converges fast since: a. rely on interval approximation instead of derivative calculations. b. the rate of the convergence is fixed to avoid the wider interval being used many times which may slow down the rate of convergence. c. interior points are selected to reduce the bounds as quickly as possible.

12. Select the intermediate points x3 and x4: c/a=a/b c/(b-c)=a/b so k=a/b= (1+ )/2 x3=x1+1/(k+1)*(x2-x1) x4=x1+k/(k+1)*(x2-x1)

13. Initialization: Determine x1 and x2 which is known to contain the minimum of the function f(x). Step 1 Determine two intermediate points x3 and x4 such that x3=x1+(1-r)*h x4=x1+r*h where r=k/(k+1)= ( -1)/2 h=x2-x1 Step 2 If x4-x3 f(x4)) and stop iterating, else go to Step3. Step3 Evaluate f(x3) and f(x4) If f(x3)<f(x4), then determine new x1 and x2 as shown following, and go to Step4. x1=x1 x2=x4 If f(x3)>f(x4), then determine the new x1 and x2 as shown following, and go to Step4. x1=x3 x2=x2 Step 4 Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3, x4. code illustration

14. Example 2. Find the minimum point and value of the function x-(x^2-2)^3/2, x ∈ [0,4] Solution: f1002=inline('x-(x.*x-2).^3/2','x'); a=0; b=4; TolX=1e-4; TolFun=1e-4; MaxIter=100; [xog,fog]=Opt_Golden(f1002,a,b,TolX,TolFun,MaxIter) xog = 3.999999954235401 fog = -1.367999892407431e+003 [xoq,foq]=Opt_Quadratic(f1002,[a,b],TolX,TolFun,MaxIter) xoq = 1.938216986786059 foq = -0.772297597277559

16. * Apply primarily on: optimization problems with multivariable functions and equality constraints * Mathematical Backgrounds: Consider the function z=f(x,y) which is subject to ϕ (x,y)=0 or y=g(x), suppose x0 is the minimum point of z=f(x,g(x)), then x0 is the solution of the following equation system: f’x+ λ ϕ ’x=0 f’y+ λ ϕ ’y=0 ϕ (x,y)=0 More generally, we can introuduce a transformed function L(x1,x2…xm; λ1,λ2…λn)=f(x1,x2…xm)+ ∑λn ϕ n(x1,x2…xn) and find all x satisfying the following equation system f’xi+ ∑ λn ϕ n’xi=0 (i=1,2…m) ϕ k(x1,x2…xm) =0 (k=1,2…n) substitute {x} back into f(x1,x2…xm) and then compare their value

17. Example 3. Find the minimum point and value of the function f(x1,x2)=x1+x2, s.t x1^2+x2^2=2 Solution: codecode >> [xo,yo,fo]=Lag() xo = 1 yo = 1 fo = -2 2 so minimum is -2 at (-1,-1)

18. * Apply primarily on: optimization problems with multivariable functions and equality or inequality constraints, and the initial guess point is out of the feasible domain. * Advantage: work well when the initial guess point is out of the domain so that the computation is subjected to less restrictions compared with the interior- point penalty method.

19. Consider the function f(x) which is subject to a series of equality or inequality constraints Ci(x)>=0(i=1,2…m),to find the minimum point of f(x), we combine these conditions into a single function: F(x)=f(x)+c*∑g(Ci(x)). penalty coefficients: ci=c1*p^(i-1) where p>=2 penalty function: g(Ci(x))=min(0, Ci(x))^2 Note that the optimum point of the unconstrained function will eventually converge to the original constrained one when c*∑g(Ci(x))<= ɛ. One way to search for the optimum point of the unconstrained function is the Newton method, which resorts to the derivative method.

20. Initialization Set up the initial optimum guess point x0, c1,p(p>=2), accuracy ɛ >0, and let k=1. Step 1 Combine the given conditions into a new function F(x)=f(x)+ci*(∑hp(x)^2+∑gs(x)^2), hp(x)=0 (p=1,2…m) gs(x)>=0 (s=1,2…n) Step 2 Search for the minimum of F(x) from the point xk-1 with unconstraint optimization methods. Step 3 Let S(xk)= ck* (∑hp(x)^2+∑gs(x)^2), if S(xk)<= ɛ, stop iteration, else let k=k+1 and go to step 2 code

21. Example 4. Find the minimum point and value of the function f(x1,x2)=x1^2-x1*x2+x2-x1+1, s.t 2x1+3x2-9=0 x2^2+x1^2-6>=0 x1>=0 x2>=0 Solution: let c1=0.05, p=2, (x1,x2)=(2,2) >> syms x1 x2; >> f=x1^2-x1*x2+x2-x1+1; >> g=[x1^2+x2^2-6;x1;x2]; h=[2*x1+3*x2-9]; >> [x,minf]=minEPF(f,[2 2],g,h,0.05,2,[x1 x2]) >> x= 1.4000 2.0668 minf= 0.7333

22. Fmincon attempts to solve constraint optimization problems of the form: min f(x) subject to: A*x<=b, Aeq*x=beq (linear constraints) x subject to: c(x)<=0, ceq(x)=0 (nonlinear constraints) lb<=x<=ub (bounds) The common syntax for fmincon is [x,fval]=fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon) where x0 is the initial guess. In Ex. 4, we have >> A=[-1 0; 0 -1]; >> b=[0; 0]; >> Aeq=[2 3]; >> beq=[9]; and the result is x = 1.399999998247254 2.066666667835164 fval= 0.733333333333334