1. Silhouette Lookup for Automatic Pose Tracking N ICK H OWE

2. Goal: 3D Pose Tracking Full 3D “motion capture” from 2D video Single camera Unmarked video Difficulties:  3D ambiguity  Self-occlusion  Foreshortening  Appearance changes  Shadowing ↑ (Uses hand-entered data)

3. The “Old” Way: Incremental Tracking Previous frame with Compare with Image Refine 2D Pose 2D Pose + Appearance NumericalOptimization Next Next frame

4. Creeping Error Incremental  Errors accumulate and grow. May be mitigated by:  Better motion models (more guidance)  Better appearance models (3D)  Better tracking (multiple hypotheses) [Sidenbladh, et. al.; Sminchisescu, et. al.] Intrinsic problems still remain. (initialization, error recovery)

5. Direct Pose Estimation Consider human abilities:  Estimate pose from still photo  Estimate pose from stick figure  Estimate pose from silhouette [Brand ’99; Rosales et. al. ’01)

6. Recognition/Retrieval Hypothesis: Humans can recognize pose by recalling similar examples.  Pose Recognition  Retrieval

7. Recognition/Retrieval Hypothesis: Humans can recognize pose by recalling similar examples.  Pose Recognition  Retrieval New Approach: 1. Store many silhouettes with known poses 2. Given video, extract silhouettes 3. Retrieve best candidate matches 4. Look for plausible series of poses over time

8. Some Related Work Estimating Human Body Configuration Using Shape Context Matching Mori & Malik, ECCV 2002 3D Tracking = Classification+Interpolation Tomasi, Petrov, & Sastry, ICCV 2003 Temporal Integration of Multiple Silhouette-based Body-part Hypotheses Kwatra, Bobick, & Johnson, CVPR 2001 3D Human Pose from Silhouettes by Relevance Vector Regression Agarwal & Triggs, CVPR 2004

9. Silhouette Comparison Turning angle (Captures morphology) Chamfer distance (Captures overlap) Combine using Belkin technique (score = sum of individual ranks)

10. Sample Retrievals (Hits from a small library of 1600 poses)

11. Coordination Between Frames Need to pick from top matches at each frame.  Want good image match at all frames  Want small change between frames  Markov chain minimization! Best local choices minimize global error etc. frame i-1frame iframe i+1

12. Too Much Coffee? Initial solution shows “twitches”

13. Smoothing it Out Jitters in motion parameters smoothed via polynomial splines

14. Making it Match Problem: poor overlap between observed silhouette & smoothed solution  Work with 11-frame splines  Optimize spline parameters to reduce chamfer distance Result: better match to observations, still smooth

15. Walking Sequence Result

16. Re-rendering Same scene, different viewpoint.

17. Another Example Tracked using library of ballet poses

18. Incremental Tracking Markov chain is best for offline use But: Convergence after ~10 frames  Incremental tracking with latency

19. Key Points Silhouette lookup provides set of potential poses for each frame Markov chain selects best temporal pose sequence (HMM) Smoothing & optimization based upon temporal splines Result: simple tracker, tolerates errors

20. Thank you! Questions?

21. Continuing Challenges Mistakes in rotational direction No data for parts not on silhouette  Incorporate optical flow Some unrealistic motions generated  Incorporate motion model Correct pose not always retrieved  Improve library coverage, retrieval

22. Future Research People carrying objects Multiple overlapping people (sports) Time considerations  Optimization slow  Chaining currently slow  Holy Grail: Real-time tracking

23. 2. Identify best (least expensive) result Markov Chain Minimization Frame 1Frame 2Frame n... 1. Compute least expense to reach each state from previous frame (cost = estimate of plausibility) State 2A State 2C State 2B State 1A State 1C State 1B State n A State n C State n B 3. Backtrack, picking out path that gave best result.

24. Silhouette Extraction Many candidate approaches.  Moving & fixed camera This work:  Static camera  Graph-based segmentation

25. Making it Match Solution doesn’t match exactly yet.