1. John A. Bender Applications in Real-time 3D Tracking Collaborators: Pietro Perona Luis Goncalves Ken Goldberg Karl Chen Ilan Lobel Steve Nowlin

2. Inspiration

3. Application (image: Lorenzo Wang Graphics)

4. Tracking Problem

5. Camera Design Orange Micro iBOT IEEE 1394 (FireWire) webcam - $100 custom modifications

6. Camera Design Orange Micro iBOT IEEE 1394 (FireWire) webcam - $100 custom modifications 3-port FireWire PCI card has total bandwidth of 400 Mb/s - $100 320x240 pixels, 3 color channels, 8 bits/pixel, 30 fps, 3 cameras = 166 Mb/s isochronous sampling?? processing time??

7. Image Processing

8. Camera Modeling real-world image lens projected image

9. Camera Modeling real-world image lens projected image refraction boundary

10. Camera Modeling focal length principal point parameters of interest… camera matrix: F x 0P x 0F y P y 001 point on image plane: [ (x-P x )/F x (y-P y )/F y 1 ] (homogeneous coordinate)

11. Camera Calibration

12. 3D Modeling transformations between cameras are modeled with rotation/translation components translation vector: [ T x T y T z ] rotation vector: [ R x R y R z ] rotation matrix: X 0 X 1 X 0 Y 1 X 0 Z 1 Y 0 X 1 Y 0 Y 1 Y 0 Z 1 Z 0 X 1 Z 0 Y 1 Z 0 Z 1

13. 3D Modeling transform rays into a common coordinate system and intersect transformation for a homogeneous 3D point (z=1): X 0 = R 10 *X 1 + T 10

14. Rig Calibration

15. Image Processing Microsoft DirectX (DirectShow) interfaces with hardware image callback executed on every frame with timestamp, each camera runs in a separate thread background modeling / foreground masking with Intel Performance Primitives optimized image processing libraries ellipse fitting using Intel Open Computer Vision libraries triangulation including ellipse size fish determination TCP network streaming executes at ~20 Hz on a Dell 2.8 GHz Pentium 4 with 1.5 GB of 533 MHz DDR SDRAM

16. Path Determination each point is a 3D point which could contain a fish each column is a different time point

17. Path Determination draw paths from each point in first frame to each point in second frame

18. Path Determination draw paths from each point in first frame to each point in second frame

19. extend each path to all possible positions Path Determination

20. extend each path to all possible positions cull paths with too-low traversal probability Path Determination

21. add dummy node with fixed transition probability in case of disappearances Path Determination

22. add dummy node with fixed transition probability in case of disappearances place dummy node at last known position Path Determination

23. for each ending node, choose the unique path that minimizes traversal probability Path Determination

24. Rendering virtual fish tank each virtual fish follows the position of a real fish Newtonian motion model for virtual fish ensures smooth movement camera position in virtual tank set to head of gold fish

25. Results – from Vision…Results – from Vision to Reality