1. Recognising Panoramas M. Brown and D. Lowe, University of British Columbia

2. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35°

3. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135°

4. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° –Panoramic Mosaic = 360 x 180°

5. Why Recognising Panoramas?

6. 1D Rotations () –Ordering matching images

7. Why Recognising Panoramas? 1D Rotations () –Ordering matching images

8. Why Recognising Panoramas? 1D Rotations () –Ordering matching images

9. Why Recognising Panoramas? 2D Rotations (, ) –Ordering matching images 1D Rotations () –Ordering matching images

10. Why Recognising Panoramas? 1D Rotations () –Ordering matching images 2D Rotations (, ) –Ordering matching images

11. Why Recognising Panoramas? 1D Rotations () –Ordering matching images 2D Rotations (, ) –Ordering matching images

12. Why Recognising Panoramas?

13. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

14. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

15. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

16. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

17. Invariant Features Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

18. SIFT Features Invariant Features –Establish invariant frame Maxima/minima of scale-space DOG x, y, s Maximum of distribution of local gradients –Form descriptor vector Histogram of smoothed local gradients 128 dimensions SIFT features are… –Geometrically invariant to similarity transforms, some robustness to affine change –Photometrically invariant to affine changes in intensity

19. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

20. Nearest Neighbour Matching Find k-NN for each feature –k number of overlapping images (we use k = 4) Use k-d tree –k-d tree recursively bi-partitions data at mean in the dimension of maximum variance –Approximate nearest neighbours found in O(nlogn)

21. Overview Feature Matching –SIFT Features –Nearest Neighbour Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

22. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

23. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

24. Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Multi-band Blending Results Conclusions

25. Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Multi-band Blending Results Conclusions

26. RANSAC for Homography

29. Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Multi-band Blending Results Conclusions

30. Probabilistic model for verification

31. Compare probability that this set of RANSAC inliers/outliers was generated by a correct/false image match –n i = #inliers, n f = #features –p 1 = p(inlier | match), p 0 = p(inlier | ~match) –p min = acceptance probability Choosing values for p 1, p 0 and p min

32. Finding the panoramas

36. Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Multi-band Blending Results Conclusions

37. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

38. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

39. Overview Feature Matching Image Matching Bundle Adjustment –Error function Multi-band Blending Results Conclusions

40. Overview Feature Matching Image Matching Bundle Adjustment –Error function Multi-band Blending Results Conclusions

41. Error function Sum of squared projection errors –n = #images –I(i) = set of image matches to image i –F(i, j) = set of feature matches between images i,j –r ij k = residual of k th feature match between images i,j Robust error function

42. Parameterise each camera by rotation and focal length This gives pairwise homographies Homography for Rotation

43. Bundle Adjustment New images initialised with rotation, focal length of best matching image

44. Bundle Adjustment New images initialised with rotation, focal length of best matching image

45. Overview Feature Matching Image Matching Bundle Adjustment –Error function Multi-band Blending Results Conclusions

46. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

47. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

48. Multi-band Blending Burt & Adelson 1983 –Blend frequency bands over range

49. Low frequency ( > 2 pixels) High frequency ( < 2 pixels) 2-band Blending

50. Linear Blending

51. 2-band Blending

54. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

55. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

56. Results

58. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

59. Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

60. Fully automatic panoramas –A recognition problem… Invariant feature based method –SIFT features, RANSAC, Bundle Adjustment, Multi- band Blending –O(nlogn) Future Work –Advanced camera modelling radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash … –Full 3D case – recognising 3D objects/scenes in unordered datasets

61. Questions?

62. Analytical computation of derivatives

63. Levenberg-Marquardt Iteration step of form