1. Motion from image and inertial measurements Dennis Strelow Carnegie Mellon University

2. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 2 On the web Related materials: these and related slides related papers movies VRML models at: http://www.cs.cmu.edu/~dstrelow/epson

3. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 3 Introduction (1) From an image sequence, we can recover: 6 degree of freedom (DOF) camera motion without knowledge of the camera’s surroundings without GPS

4. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 4 Introduction (2) Fitzgibbon

5. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 5 Introduction (3) Potential applications include: modeling from video Yuji Uchida

6. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 6 Introduction (4) micro air vehicles (MAVs) AeroVironment Black WidowAeroVironment Microbat

7. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 7 Introduction (5) rover navigation Hyperion Nister, et al.

8. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 8 Introduction (6) search and rescue robots Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)

9. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 9 Introduction (7) NASA Personal Satellite Assistant (PSA)

10. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 10 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning

11. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 11 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors

12. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 12 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors over the long term

13. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 13 Introduction (9) Long-term motion estimation: absolute distance or time is long only a small fraction of the scene is visible at any one time

14. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 14 Introduction (10) given these requirements, cameras are promising sensors… …and many algorithms for motion from images already exist

15. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 15 Introduction (11) But, where are the systems for estimating the motion of: over the long term?

16. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 16 Introduction (12) …and for automatically modeling rooms buildings cities from a handheld camera?

17. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 17 Introduction (13) Motion from images suffers from some long- standing difficulties This work attacks these problems by… exploiting omnidirectional images exploiting image and inertial measurements robust image feature tracking recognizing previously mapped locations

18. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 18 Outline Motion from images refresher bundle adjustment difficulties Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation Conclusion

19. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 19 Motion from images: refresher (1) A two-step process is common: sparse feature tracking estimation

20. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 20 Motion from images: refresher (1) A two-step process is common: sparse feature tracking estimation Sparse feature tracking: inputs: raw images outputs: projections

21. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 21 Motion from images: refresher (2)

22. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 22 Motion from images: refresher (3) Template matching: correlation tracking Lucas-Kanade (Lucas and Kanade, 1981) Extraction and matching: Harris features (Harris, 1992) Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)

23. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 23 Motion from images: refresher (4) The second step is estimation: inputs: projections outputs: 6 DOF camera position at the time of each image 3D position of each tracked point

24. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 24 Motion from images: refresher (5)

25. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 25 Motion from images: refresher (6) bundle adjustment (various, 1950’s) Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990) variable state dimension filter (VSDF) (McLauchlan, 1996) two- and three-frame methods (Hartley and Zisserman, 2000; Nister, et al. 2004)

26. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 26 Motion from images: bundle adjustment (1) From tracking, we have the image locations x ij for each point j and each image i

27. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 27 Motion from images: bundle adjustment (2) Suppose we also have estimates of: the camera rotation ρ i and translation t i at time of each image 3D point positions X j of each tracked point Then, we can compute reprojections:

28. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 28 Motion from images: bundle adjustment (3)

29. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 29 Motion from images: bundle adjustment (4)

30. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 30 Motion from images: bundle adjustment (5) So, minimize: with respect to all the ρ i, t i, X j

31. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 31 Motion from images: bundle adjustment (5) So, minimize: with respect to all the ρ i, t i, X j

32. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 32 Motion from images: difficulties (1) Estimation step can be very sensitive to… incorrect or insufficient image feature tracking camera modeling and calibration errors outlier detection thresholds sequences with degenerate camera motions

33. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 33 Motion from images: difficulties (2) Iterative batch methods have poor convergence or may fail to converge if: observations are missing the initial estimate is poor

34. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 34 Motion from images: difficulties (3) Recursive methods suffer from: poor prior assumptions on the motion poor approximations in state error modeling

35. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 35 Motion from images: difficulties (4) Resulting errors are: gross local errors long term drift

36. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 36 Motion from images: difficulties (5)

37. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 37 Motion from images: difficulties (6) 151 images, 23 points manually corrected Lucas-Kanade

38. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 38 Motion from images: difficulties (7)  squares: ground truth points  dash-dotted line: accurate estimate  solid line: image-only, bundle adjustment estimate

39. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 39 Outline Motion from images Motion from image and inertial measurements inertial sensors algorithms and results related work Robust image feature tracking Long-term motion estimation Conclusion

40. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 40 Motion from image and inertial measurements: inertial sensors (1) inertial sensors can be integrated to estimate six degree of freedom motion

41. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 41 Motion from image and inertial measurements: inertial sensors (2) But many applications require small, light, and cheap sensors

42. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 42 Motion from image and inertial measurements: inertial sensors (3) Integrating the outputs of these low grade sensors will produce drifting motion because of: noise unmodeled nonlinearities

43. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 43 Motion from image and inertial measurements: inertial sensors (4) And, we can’t even integrate until we can separate the effects of… rotation ρ gravity g acceleration a slowly changing bias b a noise n …in the accelerometer measurements

44. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 44 Motion from image and inertial measurements: inertial sensors (5) Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale

45. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 45 Motion from image and inertial measurements: inertial sensors (5) Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale …even with our low-grade sensors

46. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 46 Motion from image and inertial measurements: inertial sensors (6) With image measurements, we can: reduce the drift in integrating inertial data distinguish between… rotation ρ gravity g acceleration a bias b a noise n …in accelerometer measurements

47. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 47 Motion from image and inertial measurements: algorithms and results (1) This work has developed both: batch recursive algorithms for motion from image and inertial measurements

48. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 48 Motion from image and inertial measurements: algorithms and results (2) Gyro measurements: ω’, ω: measured and actual angular velocity b ω : gyro bias n: gaussian noise

49. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 49 Motion from image and inertial measurements: algorithms and results (3) Accelerometer measurements: ρ: rotation a’, a: measured and actual acceleration g: gravity vector b a : accelerometer bias n: gaussian noise

50. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 50 Motion from image and inertial measurements: algorithms and results (4) batch algorithm minimizes a combined error:

51. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 51 Motion from image and inertial measurements: algorithms and results (5) image term E image is the same as before

52. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 52 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

53. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 53 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

54. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 54 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

55. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 55 Motion from image and inertial measurements: algorithms and results (7) timeτ i-1 (time of image i - 1) t i-1 titi I(t i-1, …) τ i (time of image i) translation ( : translation estimate for image i – 1) ( : translation estimate for image i) ( : translation integrated from previous estimate)

56. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 56 Motion from image and inertial measurements: algorithms and results (8) time τ0τ0 translation τ1τ1 τ2τ2 τ5τ5 τ3τ3 τ4τ4 τ f-3 τ f-2 τ f-1

57. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 57 Motion from image and inertial measurements: algorithms and results (9)

58. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 58 Motion from image and inertial measurements: algorithms and results (10) I t (τ i-1, τ i,…, t i-1 ) depends on: τ i-1, τ i (known) all inertial measurements for times τ i-1 < τ < τ i (known) ρ i-1, t i-1 g b ω, b a camera linear velocities: v i

59. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 59 Motion from image and inertial measurements: algorithms and results (12)  dash-dotted line: batch estimate from image and inertial  solid line: image-only, bundle adjustment estimate  squares: ground truth points

60. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 60 Motion from image and inertial measurements: algorithms and results (13) IEKF for the same sensors, unknowns  dash-dotted line: batch estimate  solid line: IEKF estimate

61. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 61 Motion from image and inertial measurements: algorithms and results (14) Difficulties with IEKF for this application: prior assumptions about motion smoothness cannot model relative error between adjacent camera positions So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction

62. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 62 Motion from image and inertial measurements: algorithms and results (15) IEKF assumptions on motion smoothness  dash-dotted line: batch estimate  solid line: IEKF estimate  right: IEKF propagation variances too strict  left: IEKF propagation variances just right

63. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 63 Motion from image and inertial measurements Recap: image, gyro, and accelerometer measurements batch algorithm recursive algorithm experiments evaluate batch and recursive algorithms establish basic facts about motion from image and inertial measurements

64. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 64 Outline Motion from images Motion from image and inertial measurements Robust image feature tracking smalls in brief Long-term motion estimation Conclusion

65. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 65 Robust image feature tracking: smalls in brief (1) Lucas-Kanade has been the go-to feature tracker for shape-from-motion suitable for real-time subpixel accuracy general heuristics for handling large image motions

66. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 66 Robust image feature tracking: smalls in brief (1) Lucas-Kanade has been the go-to feature tracker for shape-from-motion suitable for real-time subpixel accuracy general heuristics for handling large image motions …but not robust enough for “hands-free” motion estimation

67. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 67 Robust image feature tracking: smalls in brief (2) smalls is a new feature tracker targeted at 6 DOF motion estimation combines aspects of correlation tracking and “extract and match” trackers exploits the rigid scene assumption eliminates the heuristics normally used with Lucas-Kanade SIFT is an enabling technology here

68. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 68 Robust image feature tracking: smalls in brief (3) End analysis: allows hands-free SFM for many hard sequences can still be defeated by repeated texture or lack of texture Pointers to more information on smalls on the web page

69. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 69 Outline Motion from images Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation proof of concept system experiment Conclusion

70. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 70 Long-term motion estimation: proof of concept system (1) Image-based motion estimates from any system will drift: if the features we see are always changing given sufficient time if we don’t recognize when we’ve revisited a location

71. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 71 Long-term motion estimation: proof of concept system (2)

72. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 72 Long-term motion estimation: proof of concept system (3) To limit drift: recognize when we’ve returned to a previous location exploit the return A proof of concept system demonstrates these capabilities

73. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 73 Long-term motion estimation: proof of concept system (4) “smalls” tracker state: 2D feature history for images in I variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I SIFT keypoints for image i n system state S image indices: I = {i 1, …, i n }

74. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 74 Long-term motion estimation: proof of concept system (5) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8}

75. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 75 rollback Long-term motion estimation: proof of concept system (6) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8} non-rollback States:

76. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 76 rollback Long-term motion estimation: proof of concept system (7) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8} 8 non-rollback States:

77. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 77 rollback Long-term motion estimation: proof of concept system (8) 012345678 {0, 1} {0}{0, 1, 2} 8 {0, 1, 2, 3, 8} non-rollback pruned States:

78. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 78 rollback Long-term motion estimation: proof of concept system (9) 012345678 891011 121314 151617 181920 {0, …, 6, 11, 12, 17, …, 20} non-rollback pruned States:

79. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 79 Long-term motion estimation: proof of concept system (10) When to “roll back”? examine the camera covariances for the current state and the candidate rollback state check the number of SIFT matches extend from the candidate state examine the camera covariances for the current state and the resulting extended state

80. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 80 Long-term motion estimation: experiment (1) CMU FRC highbay views; 945 images total

81. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 81 Long-term motion estimation: experiment (2) CMU FRC highbay

82. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 82 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass: images 0-213)

83. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 83 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass: images 0-213)

84. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 84 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass: images 0-213)

85. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 85 Long-term motion estimation: experiment (2) CMU FRC highbay (first backward pass: images 214-380)

86. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 86 Long-term motion estimation: experiment (2) CMU FRC highbay (second forward pass: images 381-493)

87. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 87 Long-term motion estimation: experiment (2) CMU FRC highbay (second backward pass: images 494-609)

88. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 88 Long-term motion estimation: experiment (2) CMU FRC highbay (third forward pass: images 610-762)

89. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 89 Long-term motion estimation: experiment (2) CMU FRC highbay (third backward pass: images 763-944)

90. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 90 rollback Long-term motion estimation: experiment (3) 012345678 891011 121314 151617 181920 non-rollback pruned States: normally, the system produces a general tree of states

91. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 91 Long-term motion estimation: experiment (4) … 01234567 13141514 16171817 non-rollback rollback pruned States: for this example, the “rollback” states are restricted to the first forward pass 89 10111214 213

92. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 92 Long-term motion estimation: experiment (5) movie…bottom half is smalls output:

93. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 93 Long-term motion estimation: experiment (6) movie…top half is motion estimates:

94. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 94 Long-term motion estimation: experiment (7) movie…top half is motion estimates:

95. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 95 Outline Motion from images Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation Conclusion remaining issues some previous work

96. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 96 Conclusion: remaining issues all: system is experimental, not optimized for speed image and inertial: VSDF “smalls”: integration of gyro, more robustness to poor texture needed long-term: “roll back” space, computation grow with sequence length

97. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 97 Conclusion: some previous work (1) 1998-99 (CMU): trinocular stereo for Honda humanoid and Toyota highway obstacle detection 1996-1998 (K 2 T, Inc.): architectural models from still images 1996 (U. of Illinois): Masters thesis, visualizing fMRI data with virtual reality

98. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 98 Conclusion: some previous work (2) 1995 (Los Alamos): automatically delineating rib cage volumes in CT volumes 1994 (National Solar Observatory): tracking sunspot motion, measuring solar flare intensity 1993 (U. of Nebraska): AVHRR satellite image restoration

99. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 99 Thanks! Related materials: these and related slides related papers movies VRML models at: http://www.cs.cmu.edu/~dstrelow/epson