Image Alignment and Mosaicing
Image Alignment Applications Local alignment:Local alignment: – Tracking – Stereo Global alignment:Global alignment: – Camera jitter elimination – Image enhancement – Panoramic mosaicing
Image Stitching Problems Motion modelMotion model ParameterizationParameterization CorrespondenceCorrespondence WarpingWarping BlendingBlending
Motion Model How do images relate to each other?How do images relate to each other? Possibilities:Possibilities: – Pure translation – Rigid body (translation + rotation) – Affine – Parallax – General motion
Motion Model Common case: images taken with perspective cameraCommon case: images taken with perspective camera Motion around center of projectionMotion around center of projection
Parameterization Single parameterization for all imagesSingle parameterization for all images Cylindrical parameterization often used for panoramic mosaicsCylindrical parameterization often used for panoramic mosaics
Cylindrical Parameterization After image warping, motion model is pure translationAfter image warping, motion model is pure translation
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Optical Flow for Image Registration Compute local matchesCompute local matches Least-squares fit to motion modelLeast-squares fit to motion model Problem: outliersProblem: outliers
Outlier Rejection Least squares very sensitive to outliersLeast squares very sensitive to outliers
Outlier Rejection Robust estimation: tolerant of outliersRobust estimation: tolerant of outliers In general, methods based on absolute value rather than square: minimize |x i -f|, not (x i -f ) 2In general, methods based on absolute value rather than square: minimize |x i -f|, not (x i -f ) 2
Robust Fitting In general, hard to computeIn general, hard to compute Analytic solutions in a few casesAnalytic solutions in a few cases – Mean Median – Solution for fitting lines
Approximation to Robust Fitting Compute least squaresCompute least squares Assign weights based on average deviationAssign weights based on average deviation – Simplest: weight = 1 if deviation < 2.5 , 0 otherwise – Smoother version: weight = G (deviation) Weighted least squaresWeighted least squares IterateIterate
Alternative Approximation to Robust Fitting Least median of squares:Least median of squares: – Select n random subsets of data points – Perform least-squares fit on each – Compute average deviation for each – Select fit with median deviation
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Correlation / Search Methods Assume translation onlyAssume translation only Given images I 1, I 2Given images I 1, I 2 For each translation (t x, t y ) computeFor each translation (t x, t y ) compute Select translation that maximizes cSelect translation that maximizes c Depending on window size, local or globalDepending on window size, local or global
Cross-Correlation Statistical definition of correlation:Statistical definition of correlation: Disadvantage: sensitive to local variations in image brightnessDisadvantage: sensitive to local variations in image brightness
Normalized Cross-Correlation Normalize to eliminate brightness sensitivity: whereNormalize to eliminate brightness sensitivity: where
Sum of Squared Differences More intuitive measure:More intuitive measure: Negative sign so that higher values mean greater similarityNegative sign so that higher values mean greater similarity Expand:Expand:
Correlation vs. Optical Flow Intuitively similar to functional minimization: search vs. Newton’s methodIntuitively similar to functional minimization: search vs. Newton’s method
Local vs. Global Correlation with local windows not too expensiveCorrelation with local windows not too expensive High cost if window size = whole imageHigh cost if window size = whole image But computation looks like convolutionBut computation looks like convolution – FFT to the rescue!
Correlation
Fourier Transform with Translation
Therefore, if I 1 and I 2 differ by translation,Therefore, if I 1 and I 2 differ by translation, So, F -1 (F 1 /F 2 ) will have a peak at ( x, y)So, F -1 (F 1 /F 2 ) will have a peak at ( x, y)
Phase Correlation In practice, use cross power spectrumIn practice, use cross power spectrum Compute inverse FFT, look for peaksCompute inverse FFT, look for peaks [Kuglin & Hines 1975][Kuglin & Hines 1975]
Phase Correlation AdvantagesAdvantages – Fast computation – Low sensitivity to global brightness changes (since equally sensitive to all frequencies)
Phase Correlation DisadvantagesDisadvantages – Sensitive to white noise – No robust version – Translation only Extensions to rotation, scaleExtensions to rotation, scale But not local motionBut not local motion Not too bad in practice with small local motionsNot too bad in practice with small local motions
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Correlation plus Optical Flow Use e.g. phase correlation to find average translation (may be large)Use e.g. phase correlation to find average translation (may be large) Use optical flow to find local motionsUse optical flow to find local motions
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Iteration Local refinement of optical flow estimateLocal refinement of optical flow estimate Sort of equivalent to multiple iterations of Newton’s methodSort of equivalent to multiple iterations of Newton’s method
Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine
Image Pyramids Pre-filter images to collect information at different scalesPre-filter images to collect information at different scales More efficient computation, allows larger motionsMore efficient computation, allows larger motions
Image Pyramids Szeliski
Pyramid Creation “Gaussian” Pyramid“Gaussian” Pyramid “Laplacian” Pyramid“Laplacian” Pyramid – Created from Gaussian pyramid by subtraction L i = G i – expand(G i+1 ) Szeliski
Octaves in the Spatial Domain Bandpass Images Lowpass Images Szeliski
Image Warping Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))? f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski
Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) What if pixel lands “between” two pixels?What if pixel lands “between” two pixels? f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski
Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) What if pixel lands “between” two pixels?What if pixel lands “between” two pixels? Answer: add “contribution” to several pixels, normalize later (splatting)Answer: add “contribution” to several pixels, normalize later (splatting) f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski
Inverse Warping Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x) What if pixel comes from “between” two pixels?What if pixel comes from “between” two pixels? f(x)f(x)f(x)f(x) g(x’) xx’ h - 1 (x’) Szeliski
Inverse Warping Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x) What if pixel comes from “between” two pixels?What if pixel comes from “between” two pixels? Answer: resample color value from interpolated (prefiltered) source imageAnswer: resample color value from interpolated (prefiltered) source image Szeliski
Blending Blend over too small a region: seamsBlend over too small a region: seams Blend over too large a region: ghostingBlend over too large a region: ghosting
Multiresolution Blending Different blending regions for different levels in a pyramid [Burt & Adelson]Different blending regions for different levels in a pyramid [Burt & Adelson] – Blend low frequencies over large regions (minimize seams due to brightness variations) – Blend high frequencies over small regions (minimize ghosting)
Pyramid Blending Szeliski
Minimum-Cost Cuts Instead of blending high frequencies along a straight line, blend along line of minimum differences in image intensitiesInstead of blending high frequencies along a straight line, blend along line of minimum differences in image intensities
Minimum-Cost Cuts Moving object, simple blending blur [Davis 98]
Minimum-Cost Cuts Minimum-cost cut no blur [Davis 98]