1. POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of Computing Oxford Brookes University

2. Objective ImageSegmentationPose Estimate [Images courtesy: M. Black, L. Sigal]

3. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

4. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

5. The Image Segmentation Problem Segments Image

6. Problem – MRF Formulation n Notation Labelling x over the set of pixels The observed pixel intensity values y (constitute data D) n Energy E (x) = - log Pr(x|D) + constant n Unary term Likelihood based on colour n Pairwise terms Prior Contrast term n Find best labelling x* = arg min E(x)

7. MRF for Image Segmentation D (pixels) x (labels) Image Plane i j xixi xjxj Unary Potential i (D|x i ) Pairwise Potential ij (x i, x j ) x i = {segment 1, …, segment k }for instance {obj, bkg}

8. Can be solved using graph cuts MRF for Image Segmentation MAP Solution Pair-wise Terms Contrast Term Ising Model Data (D) Unary likelihood Maximum a-posteriori (MAP) solution x* =

9. MRF for Image Segmentation Pair-wise Terms MAP Solution Unary likelihoodData (D) Unary likelihood Contrast Term Uniform Prior Maximum-a-posteriori (MAP) solution x* = Need for a human like segmentation

10. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

11. Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR 05] – Shape-Prior: Layered Pictorial Structure (LPS) – Learned exemplars for parts of the LPS model – Obtained impressive results Layer 2 Layer 1 Spatial Layout (Pairwise Configuration) + =

12. Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR 05] – Shape-Prior: Layered Pictorial Structure (LPS) – Learned exemplars for parts of the LPS model – Obtained impressive results Shape-Prior Colour + Shape Unary likelihood colour Image

13. Problems in using shape priors n Intra-class variability Need to learn an enormous exemplar set Infeasible for complex subjects (Humans) n Multiple Aspects? n Inference of pose parameters

14. Do we really need accurate models? n Interactive Image Segmentation [Boykov & Jolly, ICCV01] Rough region cues sufficient Segmentation boundary can be extracted from edges additional segmentation cues user segmentation cues

15. Do we really need accurate models? n Interactive Image Segmentation Rough region cues sufficient Segmentation boundary can be extracted from edges

16. Rough Shape Prior - The Stickman Model n 26 degrees of freedom Can be rendered extremely efficiently Over-comes problems of learning a huge exemplar set Gives accurate segmentation results

17. Pose-specific MRF Formulation D (pixels) x (labels) Image Plane i j xixi xjxj Unary Potential i (D|x i ) Pairwise Potential ij (x i, x j ) (pose parameters) Unary Potential i (x i | )

18. Pose-specific MRF Energy to be minimized Unary term Shape prior Pairwise potential Potts model distance transform

19. Pose-specific MRF Energy to be minimized Unary term Shape prior Pairwise potential Potts model += Shape Prior MAP Solution Colour likelihood Data (D) colour+ shape

20. What is the shape prior? Energy to be minimized Unary term Shape prior Pairwise potential Potts model How to find the value of ө ?

21. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

22. Formulating the Pose Inference Problem

24. Resolving ambiguity using multiple views Pose specific Segmentation Energy

25. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

26. Solving the Minimization Problem Minimize F( ө ) using Powell Minimization To solve: Let F( ө ) = Computational Problem: Each evaluation of F( ө ) requires a graph cut to be computed. (computationally expensive!!) BUT.. Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

27. Dynamic Graph Cuts PBPB SBSB cheaper operation computationally expensive operation Simpler problem P B* differences between A and B similar PAPA SASA solve

28. Dynamic Graph Cuts 20 msec Simpler problem P B* differences between A and B similar xaxa solve xbxb 400 msec

29. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

30. Segmentation Results Colour + Smoothness Colour + Smoothness + Shape Prior Only Colour Image [Images courtesy: M. Black, L. Sigal]

31. Segmentation Results - Accuracy Information used % of object pixels correctly marked Accuracy (% of pixels correctly classified) Colour45.7395.2 Colour + GMM82.4896.9 Colour + GMM + Shape 97.4399.4

32. Segmentation + Pose inference [Images courtesy: M. Black, L. Sigal]

33. Segmentation + Pose inference [Images courtesy: Vicon]

34. Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

35. Conclusions Efficient method for using shape priors for object- specific segmentation Efficient Inference of pose parameters using dynamic graph cuts Good segmentation results Pose inference - Needs further evaluation - Segmentation results could be used for silhouette intersection

36. Future Work Use dimensionality reduction to reduce the number of pose parameters. - results in less number of pose parameteres to optimize - would speed up inference Use of features based on texture Appearance models for individual part of the articulated model (instead of using a single appearance model).

37. Thank You