Image Alignment and Mosaicing. Image Alignment Applications Local alignment:Local alignment: – Tracking – Stereo Global alignment:Global alignment: –

Slides:



Advertisements
Similar presentations
Image Registration  Mapping of Evolution. Registration Goals Assume the correspondences are known Find such f() and g() such that the images are best.
Advertisements

CSCE 643 Computer Vision: Template Matching, Image Pyramids and Denoising Jinxiang Chai.
Transforming images to images
Templates and Image Pyramids Computational Photography Derek Hoiem, University of Illinois 09/06/11.
Computer Vision Optical Flow
Mosaics con’t CSE 455, Winter 2010 February 10, 2010.
Lecture 11: Transformations CS4670/5760: Computer Vision Kavita Bala.
Automatic Image Alignment (direct) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and Rick.
Motion Analysis (contd.) Slides are from RPI Registration Class.
Announcements Quiz Thursday Quiz Review Tomorrow: AV Williams 4424, 4pm. Practice Quiz handout.
Lecture 8: Geometric transformations CS4670: Computer Vision Noah Snavely.
CSCE 641 Computer Graphics: Image Sampling and Reconstruction Jinxiang Chai.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
CSCE 641 Computer Graphics: Image Registration Jinxiang Chai.
CSCE 641:Computer Graphics Image Warping/Registration Jinxiang Chai.
Announcements Project1 artifact reminder counts towards your grade Demos this Thursday, 12-2:30 sign up! Extra office hours this week David (T 12-1, W/F.
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Computing motion between images
Optical flow and Tracking CISC 649/849 Spring 2009 University of Delaware.
(1) Feature-point matching by D.J.Duff for CompVis Online: Feature Point Matching Detection, Extraction.
Lecture 9: Image alignment CS4670: Computer Vision Noah Snavely
Image Pyramids Pre-filter images to collect information at different scalesPre-filter images to collect information at different scales More efficient.
2D Fourier Theory for Image Analysis Mani Thomas CISC 489/689.
Optical Flow Estimation
Lecture 8: Image Alignment and RANSAC
Lecture 19: Optical flow CS6670: Computer Vision Noah Snavely
Matching Compare region of image to region of image. –We talked about this for stereo. –Important for motion. Epipolar constraint unknown. But motion small.
Computational Photography: Image Registration Jinxiang Chai.
Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)
CSCE 641 Computer Graphics: Image Registration Jinxiang Chai.
Announcements Project 1 artifact voting Project 2 out today (help session at end of class)
Image warping/morphing Digital Video Special Effects Fall /10/17 with slides by Y.Y. Chuang,Richard Szeliski, Steve Seitz and Alexei Efros.
Image Alignment and Mosaicing Feature Tracking and the Kalman Filter.
Image Stitching Ali Farhadi CSEP 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.
Announcements Midterm due Friday, beginning of lecture Guest lecture on Friday: Antonio Criminisi, Microsoft Research.
Image Representation Gaussian pyramids Laplacian Pyramids
Image Stitching Ali Farhadi CSE 455
Applications of Image Filters Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/04/10.
Generating panorama using translational movement model.
The Measurement of Visual Motion P. Anandan Microsoft Research.
Templates and Image Pyramids Computational Photography Derek Hoiem, University of Illinois 09/03/15.
Image Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher.
Effective Optical Flow Estimation
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm.
Templates, Image Pyramids, and Filter Banks
Mosaics part 3 CSE 455, Winter 2010 February 12, 2010.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Computer Vision – Sampling Hanyang University Jong-Il Park.
Announcements Final is Thursday, March 18, 10:30-12:20 –MGH 287 Sample final out today.
Lecture 15: Transforms and Alignment CS4670/5670: Computer Vision Kavita Bala.
Image Stitching Computer Vision CS 691E Some slides from Richard Szeliski.
CPSC 641: Image Registration
COSC579: Image Align, Mosaic, Stitch
Announcements Final is Thursday, March 20, 10:30-12:20pm
Motion and Optical Flow
Lecture 7: Image alignment
Image Stitching Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi.
Announcements Project 2 out today (help session at end of class)
Image Stitching Computer Vision CS 678
Filtering Things to take away from this lecture An image as a function
Announcements more panorama slots available now
Announcements Project 1 artifact voting Project 2 out today
Announcements more panorama slots available now
Intensity Transformation
Filtering An image as a function Digital vs. continuous images
Photometric Processing
Image Registration  Mapping of Evolution
Image Stitching Linda Shapiro ECE/CSE 576.
Image Stitching Linda Shapiro ECE P 596.
Presentation transcript:

Image Alignment and Mosaicing

Image Alignment Applications Local alignment:Local alignment: – Tracking – Stereo Global alignment:Global alignment: – Camera jitter elimination – Image enhancement – Panoramic mosaicing

Image Stitching Problems Motion modelMotion model ParameterizationParameterization CorrespondenceCorrespondence WarpingWarping BlendingBlending

Motion Model How do images relate to each other?How do images relate to each other? Possibilities:Possibilities: – Pure translation – Rigid body (translation + rotation) – Affine – Parallax – General motion

Motion Model Common case: images taken with perspective cameraCommon case: images taken with perspective camera Motion around center of projectionMotion around center of projection

Parameterization Single parameterization for all imagesSingle parameterization for all images Cylindrical parameterization often used for panoramic mosaicsCylindrical parameterization often used for panoramic mosaics

Cylindrical Parameterization After image warping, motion model is pure translationAfter image warping, motion model is pure translation

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Optical Flow for Image Registration Compute local matchesCompute local matches Least-squares fit to motion modelLeast-squares fit to motion model Problem: outliersProblem: outliers

Outlier Rejection Least squares very sensitive to outliersLeast squares very sensitive to outliers

Outlier Rejection Robust estimation: tolerant of outliersRobust estimation: tolerant of outliers In general, methods based on absolute value rather than square: minimize  |x i -f|, not  (x i -f ) 2In general, methods based on absolute value rather than square: minimize  |x i -f|, not  (x i -f ) 2

Robust Fitting In general, hard to computeIn general, hard to compute Analytic solutions in a few casesAnalytic solutions in a few cases – Mean  Median – Solution for fitting lines

Approximation to Robust Fitting Compute least squaresCompute least squares Assign weights based on average deviationAssign weights based on average deviation – Simplest: weight = 1 if deviation < 2.5 , 0 otherwise – Smoother version: weight = G  (deviation) Weighted least squaresWeighted least squares IterateIterate

Alternative Approximation to Robust Fitting Least median of squares:Least median of squares: – Select n random subsets of data points – Perform least-squares fit on each – Compute average deviation for each – Select fit with median deviation

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Correlation / Search Methods Assume translation onlyAssume translation only Given images I 1, I 2Given images I 1, I 2 For each translation (t x, t y ) computeFor each translation (t x, t y ) compute Select translation that maximizes cSelect translation that maximizes c Depending on window size, local or globalDepending on window size, local or global

Cross-Correlation Statistical definition of correlation:Statistical definition of correlation: Disadvantage: sensitive to local variations in image brightnessDisadvantage: sensitive to local variations in image brightness

Normalized Cross-Correlation Normalize to eliminate brightness sensitivity: whereNormalize to eliminate brightness sensitivity: where

Sum of Squared Differences More intuitive measure:More intuitive measure: Negative sign so that higher values mean greater similarityNegative sign so that higher values mean greater similarity Expand:Expand:

Correlation vs. Optical Flow Intuitively similar to functional minimization: search vs. Newton’s methodIntuitively similar to functional minimization: search vs. Newton’s method

Local vs. Global Correlation with local windows not too expensiveCorrelation with local windows not too expensive High cost if window size = whole imageHigh cost if window size = whole image But computation looks like convolutionBut computation looks like convolution – FFT to the rescue!

Correlation

Fourier Transform with Translation

Therefore, if I 1 and I 2 differ by translation,Therefore, if I 1 and I 2 differ by translation, So, F -1 (F 1 /F 2 ) will have a peak at (  x,  y)So, F -1 (F 1 /F 2 ) will have a peak at (  x,  y)

Phase Correlation In practice, use cross power spectrumIn practice, use cross power spectrum Compute inverse FFT, look for peaksCompute inverse FFT, look for peaks [Kuglin & Hines 1975][Kuglin & Hines 1975]

Phase Correlation AdvantagesAdvantages – Fast computation – Low sensitivity to global brightness changes (since equally sensitive to all frequencies)

Phase Correlation DisadvantagesDisadvantages – Sensitive to white noise – No robust version – Translation only Extensions to rotation, scaleExtensions to rotation, scale But not local motionBut not local motion Not too bad in practice with small local motionsNot too bad in practice with small local motions

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Correlation plus Optical Flow Use e.g. phase correlation to find average translation (may be large)Use e.g. phase correlation to find average translation (may be large) Use optical flow to find local motionsUse optical flow to find local motions

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Iteration Local refinement of optical flow estimateLocal refinement of optical flow estimate Sort of equivalent to multiple iterations of Newton’s methodSort of equivalent to multiple iterations of Newton’s method

Correspondence Approaches Optical flowOptical flow CorrelationCorrelation Correlation + optical flowCorrelation + optical flow Any of the above, iterated (e.g. Lucas- Kanade)Any of the above, iterated (e.g. Lucas- Kanade) Any of the above, coarse-to-fineAny of the above, coarse-to-fine

Image Pyramids Pre-filter images to collect information at different scalesPre-filter images to collect information at different scales More efficient computation, allows larger motionsMore efficient computation, allows larger motions

Image Pyramids Szeliski

Pyramid Creation “Gaussian” Pyramid“Gaussian” Pyramid “Laplacian” Pyramid“Laplacian” Pyramid – Created from Gaussian pyramid by subtraction L i = G i – expand(G i+1 ) Szeliski

Octaves in the Spatial Domain Bandpass Images Lowpass Images Szeliski

Image Warping Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))? f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski

Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) What if pixel lands “between” two pixels?What if pixel lands “between” two pixels? f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski

Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) What if pixel lands “between” two pixels?What if pixel lands “between” two pixels? Answer: add “contribution” to several pixels, normalize later (splatting)Answer: add “contribution” to several pixels, normalize later (splatting) f(x)f(x)f(x)f(x) g(x’) xx’ h(x)h(x)h(x)h(x) Szeliski

Inverse Warping Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x) What if pixel comes from “between” two pixels?What if pixel comes from “between” two pixels? f(x)f(x)f(x)f(x) g(x’) xx’ h - 1 (x’) Szeliski

Inverse Warping Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x) What if pixel comes from “between” two pixels?What if pixel comes from “between” two pixels? Answer: resample color value from interpolated (prefiltered) source imageAnswer: resample color value from interpolated (prefiltered) source image Szeliski

Blending Blend over too small a region: seamsBlend over too small a region: seams Blend over too large a region: ghostingBlend over too large a region: ghosting

Multiresolution Blending Different blending regions for different levels in a pyramid [Burt & Adelson]Different blending regions for different levels in a pyramid [Burt & Adelson] – Blend low frequencies over large regions (minimize seams due to brightness variations) – Blend high frequencies over small regions (minimize ghosting)

Pyramid Blending Szeliski

Minimum-Cost Cuts Instead of blending high frequencies along a straight line, blend along line of minimum differences in image intensitiesInstead of blending high frequencies along a straight line, blend along line of minimum differences in image intensities

Minimum-Cost Cuts Moving object, simple blending  blur [Davis 98]

Minimum-Cost Cuts Minimum-cost cut  no blur [Davis 98]