1. Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team http://www.eng.cam.ac.uk/~cipolla/people.html

2. 1. Real-time hand detection and tracking

3. Why is it hard? Highly articulated object, 27 model parameters Shape variation and self- occlusions Unreliable point features Ambiguities in single view lead to multi-modal distributions (local minima)

4. Why is it hard? Background clutter Potentially fast motion Lighting changes Partial / full occlusion

5. A Solved Problem? 3D tracking, 6/7 DOF Model: 3D quadrics Cost Function: Edges or colour-edges Tracking: Unscented Kalman filtering Single or dual view Single hypothesis filter, no recovery strategy

6. A Robust Tracker Should work in scenes with complex background and varying illumination –Important: Cost function design –Optimization strategy Should handle multi-modality –Examples: Particle filters, multi-hypotheses filters Should have a recovery strategy when track is lost –Trigger search algorithm

7. 3D Pose Recovery 3D hand model constructed from cones and ellipsoids Contour projection, handling self-occlusions 27 motion parameters

8. Hierarchy of classifiers

9. Likelihood : Edges Edge DetectionProjected Contours Robust Edge Matching Input Image 3D Model

10. Chamfer Matching Input imageCanny edges Distance transform Projected Contours

11. Likelihood : Colour Skin Colour Model Projected Silhouette Input Image 3D Model Template Matching

12. Tree-based bayesian filtering

13. Matching Multiple Templates Use tree structure to efficiently match many templates (>50,000) Arrange templates in tree based on their similarity Traverse tree using breadth-first search, several ‘active’ leaves possible Search Tree Grid-based partitioning of parameter space

14. Bayesian-Tree The search-tree is brought into a Bayesian framework by adding the prior knowledge from previous frame. The Bayesian-Tree can be thought as approximating the posterior probability at different resolutions. State space partitioning Estimation of posterior pdf

15. Experiments Global Motion 3D motions limited to hemisphere Dynamics: First-order Gaussian process 3 level tree with 16,000 templates at leaf level 5 scales, divisions of 15 degrees in 3D rotation and divisions of 10 degrees in image plane rotation Translation search at 20, 5, 2-pixel resolution

16. Tracking Results

18. Experiments Finger Articulation Opening and closing of thumb and fingers approximated by 2 parameters Global motion restricted to smaller range, but still with 6 DOF 35,000 templates at the leaf level

19. Opening and closing

20. Hand detection system

21. Ongoing work Large number of templates required Examples shown here show only constrained motion Number of templates required for fully articulated motion? Tracking rates at 5 fps to 0.2 fps For 400 - 35,000 templates (on a 2.4 GHz Pentium IV) Error introduced by geometric model No palm deformation, no skin deformation, no arm model

22. Detecting people

23. 2. Building 3D models of cities

24. Trumpington Street Data

25. Camera pose determination

26. 3D reconstruction

27. Reconstruction texture mapped

28. 3. Where am I?

29. Image-based localisation...

30. Image-based localisation

31. … …

38. Summary and deliverables Realtime hand detection in clutter 3D models from uncalibrated images Image-based localisation for augmented reality