POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of Computing Oxford Brookes University

Objective ImageSegmentationPose Estimate [Images courtesy: M. Black, L. Sigal]

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

The Image Segmentation Problem Segments Image

Problem – MRF Formulation n Notation Labelling x over the set of pixels The observed pixel intensity values y (constitute data D) n Energy E (x) = - log Pr(x|D) + constant n Unary term Likelihood based on colour n Pairwise terms Prior Contrast term n Find best labelling x* = arg min E(x)

MRF for Image Segmentation D (pixels) x (labels) Image Plane i j xixi xjxj Unary Potential i (D|x i ) Pairwise Potential ij (x i, x j ) x i = {segment 1, …, segment k }for instance {obj, bkg}

Can be solved using graph cuts MRF for Image Segmentation MAP Solution Pair-wise Terms Contrast Term Ising Model Data (D) Unary likelihood Maximum a-posteriori (MAP) solution x* =

MRF for Image Segmentation Pair-wise Terms MAP Solution Unary likelihoodData (D) Unary likelihood Contrast Term Uniform Prior Maximum-a-posteriori (MAP) solution x* = Need for a human like segmentation

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR 05] – Shape-Prior: Layered Pictorial Structure (LPS) – Learned exemplars for parts of the LPS model – Obtained impressive results Layer 2 Layer 1 Spatial Layout (Pairwise Configuration) + =

Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR 05] – Shape-Prior: Layered Pictorial Structure (LPS) – Learned exemplars for parts of the LPS model – Obtained impressive results Shape-Prior Colour + Shape Unary likelihood colour Image

Problems in using shape priors n Intra-class variability Need to learn an enormous exemplar set Infeasible for complex subjects (Humans) n Multiple Aspects? n Inference of pose parameters

Do we really need accurate models? n Interactive Image Segmentation [Boykov & Jolly, ICCV01] Rough region cues sufficient Segmentation boundary can be extracted from edges additional segmentation cues user segmentation cues

Do we really need accurate models? n Interactive Image Segmentation Rough region cues sufficient Segmentation boundary can be extracted from edges

Rough Shape Prior - The Stickman Model n 26 degrees of freedom Can be rendered extremely efficiently Over-comes problems of learning a huge exemplar set Gives accurate segmentation results

Pose-specific MRF Formulation D (pixels) x (labels) Image Plane i j xixi xjxj Unary Potential i (D|x i ) Pairwise Potential ij (x i, x j ) (pose parameters) Unary Potential i (x i | )

Pose-specific MRF Energy to be minimized Unary term Shape prior Pairwise potential Potts model distance transform

Pose-specific MRF Energy to be minimized Unary term Shape prior Pairwise potential Potts model += Shape Prior MAP Solution Colour likelihood Data (D) colour+ shape

What is the shape prior? Energy to be minimized Unary term Shape prior Pairwise potential Potts model How to find the value of ө ?

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Formulating the Pose Inference Problem

Resolving ambiguity using multiple views Pose specific Segmentation Energy

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Solving the Minimization Problem Minimize F( ө ) using Powell Minimization To solve: Let F( ө ) = Computational Problem: Each evaluation of F( ө ) requires a graph cut to be computed. (computationally expensive!!) BUT.. Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

Dynamic Graph Cuts PBPB SBSB cheaper operation computationally expensive operation Simpler problem P B* differences between A and B similar PAPA SASA solve

Dynamic Graph Cuts 20 msec Simpler problem P B* differences between A and B similar xaxa solve xbxb 400 msec

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Segmentation Results Colour + Smoothness Colour + Smoothness + Shape Prior Only Colour Image [Images courtesy: M. Black, L. Sigal]

Segmentation Results - Accuracy Information used % of object pixels correctly marked Accuracy (% of pixels correctly classified) Colour Colour + GMM Colour + GMM + Shape

Segmentation + Pose inference [Images courtesy: M. Black, L. Sigal]

Segmentation + Pose inference [Images courtesy: Vicon]

Outline n Image Segmentation Problem n Pose-Specific Segmentation n The Pose Inference Problem n Optimization n Results n Conclusion and Future Work

Conclusions Efficient method for using shape priors for object- specific segmentation Efficient Inference of pose parameters using dynamic graph cuts Good segmentation results Pose inference - Needs further evaluation - Segmentation results could be used for silhouette intersection

Future Work Use dimensionality reduction to reduce the number of pose parameters. - results in less number of pose parameteres to optimize - would speed up inference Use of features based on texture Appearance models for individual part of the articulated model (instead of using a single appearance model).

Thank You