1. Ankit Gupta CSE 590V: Vision Seminar

2. Goal Articulated pose estimation ( ) recovers the pose of an articulated object which consists of joints and rigid parts Slide taken from authors, Yang et al.

3. Classic Approach Part Representation Head, Torso, Arm, Leg Location, Rotation, Scale Marr & Nishihara 1978 Slide taken from authors, Yang et al.

4. Classic Approach Part Representation Head, Torso, Arm, Leg Location, Rotation, Scale Fischler & Elschlager 1973 Felzenszwalb & Huttenlocher 2005 Marr & Nishihara 1978 Pictorial Structure Unary Templates Pairwise Springs Slide taken from authors, Yang et al.

5. Classic Approach Part Representation Head, Torso, Arm, Leg Location, Rotation, Scale Fischler & Elschlager 1973 Felzenszwalb & Huttenlocher 2005 Marr & Nishihara 1978 Andriluka etc. 2009 Eichner etc. 2009 Johnson & Everingham 2010 Sapp etc. 2010 Singh etc. 2010 Tran & Forsyth 2010 Epshteian & Ullman 2007 Ferrari etc. 2008 Lan & Huttenlocher 2005 Ramanan 2007 Sigal & Black 2006 Wang & Mori 2008 Pictorial Structure Unary Templates Pairwise Springs Slide taken from authors, Yang et al.

6. Pictorial Structures for Object Recognition Pedro F. Felzenszwalb, Daniel P. Huttenlocher IJCV, 2005

7. Pictorial structure for Face

8. Pictorial Structure Model Slide taken from authors, Yang et al.

9. Pictorial Structure Model Slide taken from authors, Yang et al.

10. Pictorial Structure Model Slide taken from authors, Yang et al.

11. Using this Model

12. Test phaseTrain phase

13. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features Test phaseTrain phase

14. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features Test phaseTrain phase Standard Structural SVM formulation - Standard solvers available (SVMStruct)

15. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features Test phaseTrain phase Standard Structural SVM formulation - Standard solvers available (SVMStruct) Given: - Image (I) Need to compute - Part locations (L) Algorithm - L* = arg max (S(I,L))

16. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features Test phaseTrain phase Standard Structural SVM formulation - Standard solvers available (SVMStruct) Given: - Image (I) Need to compute - Part locations (L) Algorithm - L* = arg max (S(I,L)) Standard inference problem - For tree graphs, can be exactly computed using belief propagation

17. Yi Yang & Deva Ramanan University of California, Irvine

18. Problems with previous methods: Wide Variations In-plane rotationForeshortening ScalingOut-of-plane rotation Intra-category variationAspect ratio Slide taken from authors, Yang et al.

19. Problems with previous methods: Wide Variations In-plane rotationForeshortening ScalingOut-of-plane rotation Intra-category variationAspect ratio Naïve brute-force evaluation is expensive Slide taken from authors, Yang et al.

20. Our Method – “Mini-Parts” Key idea: “mini part” model can approximate deformations Slide taken from authors, Yang et al.

21. Example: Arm Approximation Slide taken from authors, Yang et al.

22. Pictorial Structure Model Slide taken from authors, Yang et al.

23. The Flexible Mixture Model Slide taken from authors, Yang et al.

24. Our Flexible Mixture Model Slide taken from authors, Yang et al.

25. Our Flexible Mixture Model Slide taken from authors, Yang et al.

26. Our Flexible Mixture Model Slide taken from authors, Yang et al.

27. Co-occurrence “Bias” Slide taken from authors, Yang et al.

28. Co-occurrence “Bias”: Example Let part i : eyes, mixture m i = {open, closed} part j : mouth,mixture m j = {smile, frown}

29. Co-occurrence “Bias”: Example b (closed eyes, smiling mouth) b (open eyes, smiling mouth) vs Let part i : eyes, mixture m i = {open, closed} part j : mouth,mixture m j = {smile, frown}

30. Co-occurrence “Bias”: Example b (closed eyes, smiling mouth) b (open eyes, smiling mouth) < learnt Let part i : eyes, mixture m i = {open, closed} part j : mouth,mixture m j = {smile, frown}

31. Using this Model

32. Test phaseTrain phase

33. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features - Co-occurrence Test phaseTrain phase Standard Structural SVM formulation - Standard solvers available (SVMStruct)

34. Given: - Images (I) - Known locations of the parts (L) Need to learn - Unary templates - Spatial features - Co-occurrence Test phaseTrain phase Standard Structural SVM formulation - Standard solvers available (SVMStruct) Given: - Image (I) Need to compute - Part locations (L) - Part mixtures (M) Algorithm - (L*,M*) = arg max (S(I,L,M)) Standard inference problem - For tree graphs, can be exactly computed using belief propagation

35. Results

36. Achieving articulation Slide taken from authors, Yang et al.

37. Achieving articulation Slide taken from authors, Yang et al.

38. Qualitative Results

41. Failure cases To be put

42. Benchmark Datasets PARSE Full-body http://www.ics.uci.edu/~drama nan/papers/parse/index.html BUFFY Upper-body http://www.robots.ox.ac.uk/~v gg/data/stickmen/index.html Slide taken from authors, Yang et al.

43. Quantitative Results on PARSE 1 second per image Image Parse Testset MethodHeadTorsoU. LegsL. LegsU. ArmsL. ArmsTotal Ramanan 200752.137.531.029.017.513.627.2 Andrikluka 200981.475.663.255.147.631.755.2 Johnson 200977.668.861.554.953.239.356.4 Singh 201091.276.671.564.950.034.260.9 Johnson 201085.476.173.465.464.746.966.2 Our Model97.693.283.975.172.048.374.9 % of correctly localized limbs Slide taken from authors, Yang et al.

44. Quantitative Results on BUFFY All previous work use explicitly articulated models Subset of Buffy Testset MethodHeadTorsoU. ArmsL. ArmsTotal Tran 2010 --- 62.3 Andrikluka 200990.795.579.341.273.5 Eichner 200998.797.982.859.880.1 Sapp 2010a100 91.165.785.9 Sapp 2010b10096.295.363.085.5 Our Model10099.696.670.989.1 % of correctly localized limbs Slide taken from authors, Yang et al.

45. More Parts and Mixtures Help 14 parts (joints)27 parts (joints + midpoints) Slide taken from authors, Yang et al.

46. Discussion Possible limitations? Something other than human pose estimation? Can do more useful things with the Kinect? Can this encode occlusions well?

47. References http://phoenix.ics.uci.edu/software/pose/ Code and benchmark datasets available