1. Image Congealing (batch/multiple) image (alignment/registration) Advanced Topics in Computer Vision (048921) Boris Kimelman

2. Introduction Dramatic increase in popularity of image and video sharing sites Hard to measure image similarity: – Illumination – Occlusion – Misalignment 2

3. Problem Definition 3

4. Applications Batch image alignment (congealing) Identification pre-processing Video stabilization Background segmentation Facial contour detection Inpainting 4

5. Congealing example Input imagesInput images realigned using the transformations computed by RASL 5

6. Unsupervised Joint Alignment of Complex Images Gary B Huang, Vidit Jain, Erik Learned-Miller ICCV 2007 6

7. Basic assumptions Input images have similar structure and shape Thus, low variability of pixel values at specific location Distribution Field: empirical density function at each pixel 7 Pixel stack

8. Basic algorithm 8 Each stage increases image likelihood

9. Funneling: new image alignment Add to training set and re-run Instead, save sequence of distribution fields and increase likelihood of new image at each iteration New ImageAligned Image Image Funnel 9

10. Congealing Color Images 10

11. Congealing with SIFT descriptor (1) Cluster SIFT descriptors using k-means Congealing on hard assignments forces pixels to take relatively small number of values Use soft assignment of pixels to clusters (GMM EM) Analogy with grayscale using binary alphabet 11

12. Congealing with SIFT descriptor (2) Window around pixel SIFT vector and clusters Posterior distribution 12

13. Mathematical formulation 13

14. Labeled Faces in the Wild database 13233 images Size: 250X250X16MB 5749 people 1680 people with two or more images 14

15. Results on faces 15

16. Align for identification 16 Hyper feature based identifier

17. Results on cars 17

18. Evaluation LFW database contribution Novel: Information theory point of view Funneling process  Demo code available Results:  No measure of alignment accuracy  Comparison only against face alignment algorithm Writing level: convincing  illustrations would help 18

19. RASL: Robust Alignment by Sparse and Low- Rank Decomposition for Linearly Correlated Images Yigang Peng, Arvind Balasubramanian, John Wright, Ma Yi CVPR 2011 19

20. How to measure image similarity? 20 Least Squares Learned-Miller Generalize: lower rank as much as possible

21. Basic Assumptions 21

22. Mathematical Formulation 22

23. Graphical Explanation Matrix of corrupted observations Underlying low-rank matrixSparse error matrix 23

24. Modeling Misalignment 24

25. Optimization Formulation (1) 25

26. Optimization Formulation (2) 26

27. Nuclear norm 27

28. Constraint Linearization 28

29. RASL Algorithm 29

30. Region of Attraction 30

31. Results on controlled data set 100 misaligned images Vedaldi CVPR 08 direct/gradient RASL 31

32. Results on LFW 32

33. Stabilizing video frames 33

34. Aligning planar surfaces despite occlusions 34

35. Evaluation Novel: Unifying framework for image congealing Rank minimization as image similarity Code available Results: Comprehensive algorithm assessment  Compare only against one algorithm Extensive site about rank minimization Writing level: Convincing  Advanced mathematics required (optimization) 35

36. Comparison of Papers Learned-MillerRASL Align similar imagesYes Align different images YesNo Trained to specific object No Robust to variationsYes Robust to occlusions Yes-Yes Occlusion removalNoYes Run timeLow?Low PerformanceHigh 36

37. Future issues Multi sensor congealing: complex relationship between corresponding pixels Learned Miller – occlusion removal by interspace alignment RASL – mix between image spaces – funneling 37

38. Thank You