1. Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia

2. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35°

3. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135°

4. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° –Panoramic Mosaic = 360 x 180°

5. Recognising Panoramas

6. 1D Rotations () –Ordering  matching images

7. Recognising Panoramas 1D Rotations () –Ordering  matching images

8. Recognising Panoramas 1D Rotations () –Ordering  matching images

9. Recognising Panoramas 2D Rotations (, ) –Ordering  matching images 1D Rotations () –Ordering  matching images

10. Recognising Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

11. Recognising Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

12. Recognising Panoramas

13. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

14. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

15. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

16. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

17. Invariant Features Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

18. SIFT Features Invariant Features –Establish invariant frame Maxima/minima of scale-space DOG  x, y, s Maximum of distribution of local gradients   –Form descriptor vector Histogram of smoothed local gradients 128 dimensions SIFT features are… –Geometrically invariant to similarity transforms, some robustness to affine change –Photometrically invariant to affine changes in intensity

19. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

20. Nearest Neighbour Matching Nearest neighbour matching Use k-d tree –k-d tree recursively bi-partitions data at mean in the dimension of maximum variance –Approximate nearest neighbours found in O(n log n) Find k-NN for each feature –k  number of overlapping images (we use k = 4) [ Beis Lowe 1997, Nene Nayar 1997, Gray Moore 2000, Shakhnarovich 2003 ]

21. K-d tree

23. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

24. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

25. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

26. Overview Feature Matching Image Matching –RANSAC for Homography Bundle Adjustment Multi-band Blending Results Conclusions

27. Overview Feature Matching Image Matching –RANSAC for Homography Bundle Adjustment Multi-band Blending Results Conclusions

28. RANSAC for Homography

31. RANSAC: 1D Line Fitting least squares line

32. RANSAC: 1D Line Fitting

33. RANSAC line

34. The RANSAC Algorithm function H = RANSAC(points, nIterations) { bestInliers = 0; bestH = zeros(3, 3); for (i = 0; i < nIterations; i++) { samplePoints = RandomSample(points); H = ComputeTransform(samplePoints); nInliers = Consistent(H); if (nInliers > bestInliers) { bestInliers = nInliers; bestH = H; } // end if } // end for } // end RANSAC

35. 2D Transforms Linear (affine) Homography

36. Finding the panoramas

40. Connected Components

41. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

42. Bundle Adjustment Adjust rotation, focal length of each image to minimise error in matched features

43. Bundle Adjustment Adjust rotation, focal length of each image to minimise error in matched features

44. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

45. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

46. Multi-band Blending Burt & Adelson 1983 –Blend frequency bands over range 

47. Low frequency ( > 2 pixels) High frequency ( < 2 pixels) 2-band Blending

48. Linear Blending

49. 2-band Blending

52. Multi-band Blending No blending

53. Multi-band Blending Linear blending –Each pixel is a weighted sum

54. Multi-band Blending Multi-band blending –Each pixel is a weighted sum (for each band)

55. Multi-band Blending Linear blending Multi-band blending

56. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

57. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

58. Results

60. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

61. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

62. Fully automatic panoramas –A recognition problem… Invariant feature based method –SIFT features, RANSAC, Bundle Adjustment, Multi- band Blending –O(nlogn) Future Work –Advanced camera modelling radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash … –Full 3D case – recognising 3D objects/scenes in unordered datasets. “PhotoTourism”. http://www.autostitch.net