1. Real-time Dense Visual Odometry for Quadrocopters Christian Kerl 11.05.20121

2. Outline Motivation Hardware & Software Approach Problems Ideas 11.05.20122

3. Motivation Quadrocopters need sensors to fly in unknown environments – Motion – Position – Obstacles Restricted on-board sensors – IMU – Visual navigation (no GPS) Restricted computing resources  Autonomous system 11.05.20123

4. Motivation Standard approach to visual odometry: – Sparse feature tracking in intensity / color images – Examples: Jakob, ETH Zurich, TU Graz, MIT – On-board frame rates 10 Hz Our approach: – Using full RGB-D image information – No feature tracking 11.05.20124

5. Hardware – Asctec Pelican 11.05.20125

6. Hardware – Asctec Pelican IMU AutoPilot Board Atom Board 11.05.20126

7. Hardware – Asctec Pelican IMU – 3 axis magnetometer, gyroscope, accelerometer AutoPilot Board – Highlevel + Lowlevel Processor (ARM) Atom Board – Intel Atom Z530 1.6 GHz – 1 GB RAM – 7 Mini-USB Ports – WirelessLAN 600 g payload 11.05.20127

8. Software – Asctec Pelican ROS drivers for Asctec Pelican from ETH Zurich Nonlinear dynamic inversion for position control Luenberger Observer for data fusion Updated version using Extended Kalman Filter to be presented on ICRA 2012 Needs absolute position input from external source Allows to command accelerations, velocities or positions 11.05.20128

9. Hardware – Asus Xtion Pro Live 24 bit RGB image 16 bit depth image 640x480 @ 30 Hz 150 g +On-camera RGB and depth image registration +Time synchronized depth and RGB image -Rolling shutter -Auto exposure 11.05.20129

10. Approach Estimate transformation minimizing squared intensity error (energy minimization) 11.05.201210

11. Approach Linearization with minimize => solve normal equations 11.05.201211

12. Analysis Estimate transformation minimizing squared intensity error (energy minimization) X translation Y translation 11.05.201212

13. Hovering Image Data from Quadrocopter 11.05.201213

14. Trajectory along camera z-axis Image Data from Quadrocopter 11.05.201214

15. Problems Motion blur Auto exposure Dynamic objects (humans) 11.05.201215

16. Problems – Motion Blur 11.05.201216

17. Problems – Motion Blur 11.05.201217

18. Problems – Motion Blur 11.05.201218

19. Problems – Motion Blur 11.05.201219

20. Problems – Motion Blur 11.05.201220

21. Problems – Motion Blur 11.05.201221

22. Problems – Motion Blur 11.05.201222

23. Problems – Auto Exposure 11.05.201223

24. Problems – Auto Exposure 11.05.201224

25. Problems – Dynamic Objects 11.05.201225

26. Ideas Weighted Least Squares Initial motion estimate between 2 consecutive frames from IMU data fusion Multiple iterations per level, convergence checks Regularization term to minimize / constrain least squares solution Minimization of intensity and depth error 11.05.201226

27. Ideas – Weighted Least Squares Assign smaller weight to residual outliers => Weight calculation 11.05.201227

28. Ideas – Weighted Least Squares Influence function – Tukey weight – Huber weight 11.05.201228

29. Ideas – Weighted Least Squares Weighted error 11.05.201229

30. Ideas – Weighted Least Squares Influence on energy function X translation w/o weightsX translation w/ Huber weights 11.05.201230

31. Ideas – Weighted Least Squares Influence on energy function Y translation w/o weightsY translation w/ Huber weights 11.05.201231

32. Ideas – Weighted Least Squares 11.05.201232

33. Ideas – Weighted Least Squares Robustification with respect to dynamic objects Slightly degrades tracking performance How to choose parameter b? 11.05.201233

34. Ideas – Initialization from IMU Use transformation from IMU data fusion as initial estimate 11.05.201234

35. Ideas – Initialization from IMU Use transformation from IMU data fusion as initial estimate 11.05.201235

36. Ideas – Initialization from IMU Use transformation from IMU data fusion as initial estimate 11.05.201236

37. Ideas – Multiple Iterations Perform multiple optimization steps per image pyramid level Stop when increment below threshold Bad frames / diverging results can be recognized and skipped 11.05.201237

38. Summary/Discussion Weighted Least Squares needs more work (especially weight calculation) Initialization from IMU promising Multiple Iterations for increased accuracy and divergence detection promising, but computationally expensive Jumps in trajectory are really problematic! => Ideas welcome! 11.05.201238