Image Stitching Shangliang Jiang Kate Harrison

What is image stitching?

Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35°

Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135°

Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° –Panoramic Mosaic = 360 x 180°

Recognizing Panoramas 1D Rotations () –Ordering  matching images

Recognizing Panoramas 1D Rotations () –Ordering  matching images

Recognizing Panoramas 1D Rotations () –Ordering  matching images

Recognizing Panoramas 2D Rotations (, ) –Ordering  matching images 1D Rotations () –Ordering  matching images

Recognizing Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

Recognizing Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

Recognizing Panoramas

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

SIFT Features SIFT features are… –Geometrically invariant to similarity transforms, some robustness to affine change –Photometrically invariant to affine changes in intensity

Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Nearest Neighbor Matching Find k nearest neighbors for each feature –k  number of overlapping images (we use k = 4) Use k-d tree –k-d tree recursively bi-partitions data at mean in the dimension of maximum variance –Approximate nearest neighbors found in O(nlogn)

Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

RANSAC for Homography

Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

Probabilistic model for verification

Finding the panoramas

Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

Bundle Adjustment New images initialised with rotation, focal length of best matching image

Bundle Adjustment New images initialised with rotation, focal length of best matching image

Error function Sum of squared projection errors –n = #images –I(i) = set of image matches to image i –F(i, j) = set of feature matches between images i,j –r ij k = residual of k th feature match between images i,j Robust error function

Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Blending

Gain compensation

How do we blend? Linear blending Multi-band blending

Multi-band Blending Burt & Adelson 1983 –Blend frequency bands over range 

Low frequency ( > 2 pixels) High frequency ( < 2 pixels) 2-band Blending

3-band blending Band 1: high frequencies

3-band blending Band 2: mid-range frequencies

3-band blending Band 3: low frequencies

Panorama straightening Heuristic: people tend to shoot pictures in a certain way

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

Conclusion

Algorithm

AutoStitch program AutoStitch.net

Open questions Advanced camera modeling –radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash … Full 3D case – recognizing 3D objects/scenes in unordered datasets

Credits Automatic Panoramic Image Stitching Using Invariant Features, 2007 –Matthew Brown and David G. Lowe (Uni. of British Columbia) Recognising Panoramas, 2003 –Matthew Brown and David G. Lowe (Uni. of British Columbia) –2003 –Thanks for the slides! Image Alignment and Stitching: A Tutorial, 2006 –Richard Szeliski (Microsoft)

Questions?