1. Motion from image and inertial measurements (additional slides) Dennis Strelow Carnegie Mellon University

2. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 2 Outline Robust image feature tracking (in detail) Lucas-Kanade and real sequences The “smalls” tracker Motion from omnidirectional images

3. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 3 Robust image feature tracking: Lucas- Kanade and real sequences (1) Combining image and inertial measurements improves our situation, but… we still need accurate feature tracking tracking some sequences do not come with inertial measurements

4. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 4 Robust image feature tracking: Lucas- Kanade and real sequences (2) better feature tracking for improved 6 DOF motion estimation remaining results will be image-only

5. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 5 Robust image feature tracking: Lucas- Kanade and real sequences (3) Lucas-Kanade has been the go-to feature tracker for shape-from-motion minimizes a correlation-like matching error using general minimization evaluates the matching error at only a few locations subpixel resolution

6. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 6 Robust image feature tracking: Lucas- Kanade and real sequences (4) Additional heuristics used to apply Lucas- Kanade to shape-from-motion: task:heuristic: choose features to trackhigh image texture identify mistracked, occluded, no-longer-visible convergence, matching error handle large motionsimage pyramid

7. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 7 Robust image feature tracking: Lucas- Kanade and real sequences (5) But Lucas-Kanade performs poorly on many real sequences…

8. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 8 Robust image feature tracking: the “smalls” tracker (1) smalls is a new feature tracker targeted at 6 DOF motion estimation exploits the rigid scene assumption eliminates the heuristics normally used with Lucas-Kanade SIFT is an enabling technology here

9. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 9 Robust image feature tracking: the “smalls” tracker (2) First step: epipolar geometry estimation use SIFT to establish matches between the two images get the 6 DOF camera motion between the two images get the epipolar geometry relating the two images

10. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 10 Robust image feature tracking: the “smalls” tracker (3)

11. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 11 Robust image feature tracking: the “smalls” tracker (4)

12. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 12 Robust image feature tracking: the “smalls” tracker (5) Second step: track along epipolar lines use nearby SIFT matches to get initial position on epipolar line exploits the rigid scene assumption eliminates heuristic: pyramid

13. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 13 Robust image feature tracking: the “smalls” tracker (6) Third step: prune features geometrically inconsistent features are marked as mistracked and removed clumped features are pruned eliminates heuristic: detecting mistracked features based on convergence, error

14. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 14 Robust image feature tracking: the “smalls” tracker (7) Fourth step: extract new features spatial image coverage is the main criterion required texture is minimal when tracking is restricted to the epipolar lines eliminates heuristic: extracting only textured features

15. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 15 Robust image feature tracking: the “smalls” tracker (8)

16. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 16 Robust image feature tracking: the “smalls” tracker (9) left: odometry onlyright: images only average error: 1.74 m maximum error: 5.14 m total distance: 230 m

17. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 17 Robust image feature tracking: the “smalls” tracker (10) Recap: exploits the rigid scene and eliminates heuristics allows hands-free tracking for real sequences can still be defeated by textureless areas or repetitive texture

18. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 18 Outline Robust image feature tracking (in detail) Motion from omnidirectional images

19. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 19 Motion from omnidirectional images (1)

20. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 20 Motion from omnidirectional images (2)

21. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 21 Motion from omnidirectional images (3)

22. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 22 Motion from omnidirectional images (4)

23. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 23 Motion from omnidirectional images (5) left: non-rigid cameraright: rigid camera squares: ground truth points solid: image-only estimates dash-dotted: image-and-inertial estimates

24. Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 24 Motion from omnidirectional images (6) In this experiment: omni images conventional images + inertial have roughly the same advantages But in general: inertial has some advantages that omni images alone can’t produce omni images can be harder to use