Conjugate Gradient Optimization

Outline CG Method Non-linear CG Solving Linear System of Equations Preconditioned CG and Regularization

Quasi-Newton Condition: g’ – g = Hdx’ Recall: d f(x)/dx ~ [df(x+dx)/dx-df(x)/dx]/dx 2 Step 1: Quadratic f(x+dx’) ~ f(x) + dx’ g + 1/2dx’ H dx’ Step 2: Gradient f(x+dx’) ~ g + H dx’ D Step 3: Define f(x+dx’) = g’  g’ – g = Hdx’ D Similar to FD approx. to 2nd dervative x* dx _ g’ g dx’ g’ dx Kiss point x

Conjugate Gradient dx’ = bdx– g’ s.t. (dx’, Hdx)=0 dx _ g D dx Quasi-Newton Condition: g’ – g = Hdx’ (1) Plane spanned by dx and dx’ dx’ x* dx _ g g dx’ dx’ dx’ Kiss point dx Bullseye ( f(x*),dx)=0 D Bullseye ( f(x*),dx’)=0 g’ at bullseye has no components in dx & dx’ plane If dx’ points at bullseye, then dot product of dx with eq. 1 gives -dxg = 0 = dxHdx’ (3) Conjugacy Cond. -> search dx’ (conjugate to previous dx) hits bullseye dx’ = bdx– g’ s.t. (dx’, Hdx)=0 Above is conjugate to dx if b found

{ Conjugate Gradient Conjugacy Cond : dx’Hdx = 0 (3) dx New Search Direc: New Search Direc: dx’ = bdx – g’ (4) { dx’ is conjugate to dx for b s.t. (dx’,Hdx)=0 ( bdx – g’,Hdx)=0 (5) b = g’Hdx/dxHdx (6) Solve for Conjugacy Step b Step length a : a = (g,dx)/dxHdx (7) Hit Bullseye No H!

Conjugate Gradient dx’ For k =2:Niter x* dx No H! end Fletcher-Reeves

Conjugate Gradient For k =2:Niter end dx dx’ Starting point x0 and take -g as initial direction dx’ For k =2:Niter x* dx Polack-Ribiere end

Conjugate Gradient Properties For i = 1:nit %find b p= dx +bg %find a x* g’ dx’ dx’ = dx + ap dx’’ dx= dx’ Kiss point dx x=x+ dx’ end 1. (p ,Hp )=0 for any i=j i j 2. Converges in N steps for NxN H with SPD property 3. Converges quickly if eigenvalues are clustered (i.e., round contours)

Solving Rectangular Linear Systems by Regularized SD with Scaling Given: H rectangular matrix s.t. Hx=g ill-conditioned Let CH H x = Cg s.t. C approximates inverse H H T T x = x – [CH( H x - g) + l Gx (k+1) (k) T (k) Soln: a CG will converge in 2 steps! Newton in one step! 1 1 4 5 2 x1 x2 = Classroom Exercise: Derive formula for b 2. Write CG code and solve above equations

Solving Square Linear Systems by Regularized CG HL Solving Lx = -g  e = xTg + 0.5 xT L x

Solving Rectangular Linear Systems by Regularized CG Solving LTLx = -L Tg  e = xTg + 0.5 xT LTL x Write Two Subroutines: 1). [d]=forward(m) % predict data d=Lm from model 2). [g] = adjoint(r) % predict model g=L’r from data residual r=(Lm-d)