1. Image Stitching Shangliang Jiang Kate Harrison

2. What is image stitching?

4. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35°

5. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135°

6. Introduction Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° –Panoramic Mosaic = 360 x 180°

7. Recognizing Panoramas 1D Rotations () –Ordering  matching images

8. Recognizing Panoramas 1D Rotations () –Ordering  matching images

9. Recognizing Panoramas 1D Rotations () –Ordering  matching images

10. Recognizing Panoramas 2D Rotations (, ) –Ordering  matching images 1D Rotations () –Ordering  matching images

11. Recognizing Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

12. Recognizing Panoramas 1D Rotations () –Ordering  matching images 2D Rotations (, ) –Ordering  matching images

13. Recognizing Panoramas

14. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

15. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

16. Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

17. SIFT Features SIFT features are… –Geometrically invariant to similarity transforms, some robustness to affine change –Photometrically invariant to affine changes in intensity

18. Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

19. Nearest Neighbor Matching Find k nearest neighbors for each feature –k  number of overlapping images (we use k = 4) Use k-d tree –k-d tree recursively bi-partitions data at mean in the dimension of maximum variance –Approximate nearest neighbors found in O(nlogn)

20. Overview Feature Matching –SIFT Features –Nearest Neighbor Matching Image Matching Bundle Adjustment Image Compositing Conclusions

21. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

22. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

23. Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

24. Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

25. RANSAC for Homography

30. Overview Feature Matching Image Matching –Random Sample Consensus (RANSAC) for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

31. Probabilistic model for verification

32. Finding the panoramas

35. Overview Feature Matching Image Matching –RANSAC for Homography –Probabilistic model for verification Bundle Adjustment Image Compositing Conclusions

36. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

37. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

38. Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

39. Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

40. Bundle Adjustment New images initialised with rotation, focal length of best matching image

41. Bundle Adjustment New images initialised with rotation, focal length of best matching image

42. Error function Sum of squared projection errors –n = #images –I(i) = set of image matches to image i –F(i, j) = set of feature matches between images i,j –r ij k = residual of k th feature match between images i,j Robust error function

43. Overview Feature Matching Image Matching Bundle Adjustment –Error function Image Compositing Conclusions

44. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

46. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

47. Blending

48. Gain compensation

49. How do we blend? Linear blending Multi-band blending

50. Multi-band Blending Burt & Adelson 1983 –Blend frequency bands over range 

51. Low frequency ( > 2 pixels) High frequency ( < 2 pixels) 2-band Blending

52. 3-band blending Band 1: high frequencies

53. 3-band blending Band 2: mid-range frequencies

54. 3-band blending Band 3: low frequencies

55. Panorama straightening Heuristic: people tend to shoot pictures in a certain way

56. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

57. Overview Feature Matching Image Matching Bundle Adjustment Image Compositing Conclusions

58. Conclusion

59. Algorithm

60. AutoStitch program AutoStitch.net

61. Open questions Advanced camera modeling –radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash … Full 3D case – recognizing 3D objects/scenes in unordered datasets

62. Credits Automatic Panoramic Image Stitching Using Invariant Features, 2007 –Matthew Brown and David G. Lowe (Uni. of British Columbia) Recognising Panoramas, 2003 –Matthew Brown and David G. Lowe (Uni. of British Columbia) –2003 –Thanks for the slides! Image Alignment and Stitching: A Tutorial, 2006 –Richard Szeliski (Microsoft)

63. Questions?