1. Agenda Project 2- Due this Thursday Office Hours Wed 10:30-12 Image blending Background – Constrained optimization

2. Recall: goal

3. Formulation: find the best patch f Given vector field v (pasted gradient), find the value of f in unknown region that optimize: Pasted gradient Mask Background unknown region

4. Notation Destination image: f* (table) Source image: g (table) Output image: f (table)  : list of (i,j) pixel coordinates from f* we want to replace d  : list of (i,j) pixel coordinates on border of  We’ll use p = (i,j) to denote a pixel location – g p is a pixel value at p = (i,j) from source image, – f  is the set of pixels we’re trying to find

5. Notation Destination image: f* (table) Source image: g (table) Output image: f (table)  : set of (i,j) pixel coordinates from f we want to replace (list of pairs) d  : set of (i,j) pixel coordinates on border of  (list of pairs) We’ll use p = (i,j) to denote a pixel location – g p is a pixel value at p = (i,j) from source image, – f  is the set of pixels we’re trying to find With constraint that, for p in d  sum over all pairs of neighbors in 

6. Optimization What is optimal f  without above constraint? What is known versus unknown? Variational formulation of solution: The best patch is the one that produces the lowest score, subject to the constraint Drop subscript for all p in dOmega

7. Optimization Pretend constraint wasn’t there: how to find lowest scoring f  ? 1)Brute-force search -Keep guessing different patches f and score them -Output the best-scoring one 2)Gradient descent -Guess a patch f. Update guess with f = f -

8. How to estimate gradient? In general, we can always do it numerically For above quadratic function, we can calculate in closed form

9. How to estimate gradient? In general, we can always do it numerically For above quadratic function, we can calculate in closed form

10. Constrained optimization 1)Brute-force search -Keep guessing different patches f and score them -Output the best-scoring one 2)Gradient descent -Guess a patch f. Update guess with f = f - What happens when gradient is zero?

11. Optimization 1)Brute-force search -Keep guessing different patches f and score them -Output the best-scoring one 2)Gradient descent -Guess a patch f. Update guess with f = f – 3) Closed-form solution (for simple functions)

12. Constrained optimization How to handle constraints? 1)Brute-force search -Keep guessing different patches f and score them -Output the best-scoring one 2)Gradient descent -Guess a patch f. Update guess with f = f - Correct f p = f* p after a gradient update

13. Constrained optimization How to handle constraints? 1)Brute-force search -Keep guessing different patches f and score them -Output the best-scoring one 2)Gradient descent -Guess a patch f. Update guess with f = f - What happens when gradient is zero?

14. Lagrangian optimization If there was no constraint, we’d have a closed- form solution Is there a way to get closed-form solutions using the constraint?

15. Lagrangian optimization min f(x,y) such that g(x,y) = 0 Imagine we want to synthesize a “two-pixel” patch

16. Lagrangian optimization min f(x,y) such that g(x,y) = 0 and g(x,y) = 0

17. Write conditions with single equation (just for convenience) At minimum of F, the its gradient is 0 Therefore, the following conditions hold

18. Multiple constraints min f(x,y) such that g1(x,y) = 0, g2(x,y) = 0 What is f(x,y) in our case? g1(x,y)?

19. Lagrangian optimization for p in d  (border pixels) for all other p in  Since S is quadratic in f, the above yeilds a set of linear equations Af =b f = inv(A)b