Image Congealing (batch/multiple) image (alignment/registration) Advanced Topics in Computer Vision (048921) Boris Kimelman

Introduction Dramatic increase in popularity of image and video sharing sites Hard to measure image similarity: – Illumination – Occlusion – Misalignment 2

Problem Definition 3

Applications Batch image alignment (congealing) Identification pre-processing Video stabilization Background segmentation Facial contour detection Inpainting 4

Congealing example Input imagesInput images realigned using the transformations computed by RASL 5

Unsupervised Joint Alignment of Complex Images Gary B Huang, Vidit Jain, Erik Learned-Miller ICCV

Basic assumptions Input images have similar structure and shape Thus, low variability of pixel values at specific location Distribution Field: empirical density function at each pixel 7 Pixel stack

Basic algorithm 8 Each stage increases image likelihood

Funneling: new image alignment Add to training set and re-run Instead, save sequence of distribution fields and increase likelihood of new image at each iteration New ImageAligned Image Image Funnel 9

Congealing Color Images 10

Congealing with SIFT descriptor (1) Cluster SIFT descriptors using k-means Congealing on hard assignments forces pixels to take relatively small number of values Use soft assignment of pixels to clusters (GMM EM) Analogy with grayscale using binary alphabet 11

Congealing with SIFT descriptor (2) Window around pixel SIFT vector and clusters Posterior distribution 12

Mathematical formulation 13

Labeled Faces in the Wild database images Size: 250X250X16MB 5749 people 1680 people with two or more images 14

Results on faces 15

Align for identification 16 Hyper feature based identifier

Results on cars 17

Evaluation LFW database contribution Novel: Information theory point of view Funneling process  Demo code available Results:  No measure of alignment accuracy  Comparison only against face alignment algorithm Writing level: convincing  illustrations would help 18

RASL: Robust Alignment by Sparse and Low- Rank Decomposition for Linearly Correlated Images Yigang Peng, Arvind Balasubramanian, John Wright, Ma Yi CVPR

How to measure image similarity? 20 Least Squares Learned-Miller Generalize: lower rank as much as possible

Basic Assumptions 21

Mathematical Formulation 22

Graphical Explanation Matrix of corrupted observations Underlying low-rank matrixSparse error matrix 23

Modeling Misalignment 24

Optimization Formulation (1) 25

Optimization Formulation (2) 26

Nuclear norm 27

Constraint Linearization 28

RASL Algorithm 29

Region of Attraction 30

Results on controlled data set 100 misaligned images Vedaldi CVPR 08 direct/gradient RASL 31

Results on LFW 32

Stabilizing video frames 33

Aligning planar surfaces despite occlusions 34

Evaluation Novel: Unifying framework for image congealing Rank minimization as image similarity Code available Results: Comprehensive algorithm assessment  Compare only against one algorithm Extensive site about rank minimization Writing level: Convincing  Advanced mathematics required (optimization) 35

Comparison of Papers Learned-MillerRASL Align similar imagesYes Align different images YesNo Trained to specific object No Robust to variationsYes Robust to occlusions Yes-Yes Occlusion removalNoYes Run timeLow?Low PerformanceHigh 36

Future issues Multi sensor congealing: complex relationship between corresponding pixels Learned Miller – occlusion removal by interspace alignment RASL – mix between image spaces – funneling 37

Thank You