1. Conjugate Gradient Optimization

2. Outline CG Method Non-linear CG Solving Linear System of Equations
Preconditioned CG and Regularization

3. Quasi-Newton Condition: g’ – g = Hdx’
Recall: d f(x)/dx ~ [df(x+dx)/dx-df(x)/dx]/dx 2
Step 1: Quadratic f(x+dx’) ~ f(x) + dx’ g + 1/2dx’ H dx’
Step 2: Gradient f(x+dx’) ~ g + H dx’ D
Step 3: Define f(x+dx’) = g’ 
g’ – g = Hdx’ D
Similar to FD approx. to 2nd dervative x*
dx _ g’ g
dx’ g’ dx
Kiss point x

4. Conjugate Gradient dx’ = bdx– g’ s.t. (dx’, Hdx)=0 dx _ g D dx
Quasi-Newton Condition: g’ – g = Hdx’ (1)
Plane spanned by dx and dx’
dx’ x*
dx _ g g
dx’
dx’
dx’
Kiss point dx
Bullseye ( f(x*),dx)=0 D
Bullseye ( f(x*),dx’)=0
g’ at bullseye has no components in dx & dx’ plane
If dx’ points at bullseye, then dot product of dx with eq. 1 gives
-dxg = 0 = dxHdx’ (3)
Conjugacy Cond. -> search dx’ (conjugate to previous dx) hits bullseye
dx’ = bdx– g’ s.t. (dx’, Hdx)=0
Above is conjugate to dx if b found

5. { Conjugate Gradient Conjugacy Cond : dx’Hdx = 0 (3) dx
New Search Direc:
New Search Direc:
dx’ = bdx – g’ (4) {
dx’ is conjugate to dx
for b s.t. (dx’,Hdx)=0
( bdx – g’,Hdx)= (5)
b = g’Hdx/dxHdx (6)
Solve for Conjugacy Step b
Step length a : a = (g,dx)/dxHdx (7)
Hit Bullseye
No H!

6. Conjugate Gradient
dx’
For k =2:Niter x* dx
No H!
end
Fletcher-Reeves

7. Conjugate Gradient For k =2:Niter end dx dx’
Starting point x0 and take -g as initial direction
dx’
For k =2:Niter x* dx
Polack-Ribiere
end

8. Conjugate Gradient Properties
For i = 1:nit
%find b
p= dx +bg
%find a x* g’
dx’
dx’ = dx + ap
dx’’
dx= dx’
Kiss point dx
x=x+ dx’
end
1. (p ,Hp )=0 for any i=j i j
2. Converges in N steps for NxN H with SPD property
3. Converges quickly if eigenvalues are clustered (i.e., round contours)

9. Solving Rectangular Linear Systems by Regularized SD with Scaling
Given: H rectangular matrix s.t. Hx=g ill-conditioned
Let CH H x = Cg s.t. C approximates inverse H H T T
x = x – [CH( H x - g) + l Gx
(k+1)
(k) T
(k)
Soln: a
CG will converge in 2 steps!
Newton in one step! 1 5 2 x1 x2 =
Classroom Exercise:
Derive formula for b
2. Write CG code and
solve above equations

10. Solving Square Linear Systems by
Regularized CG
HL
Solving Lx = -g  e = xTg xT L x

11. Solving Rectangular Linear Systems by Regularized CG
Solving LTLx = -L Tg  e = xTg xT LTL x
Write Two Subroutines: 1). [d]=forward(m) % predict data d=Lm from model
2). [g] = adjoint(r) % predict model g=L’r from data residual r=(Lm-d)