MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra Malik

MSRI University of California Berkeley 2 Finding People Challenges: Pose, Clothing, Lighting, Clutter, …

MSRI University of California Berkeley 3 Previous Work Related Domains –Tracking People –Detecting Pedestrians –... … Localizing Human Figures –Exemplar-based: [Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], … –Part-based: [Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], … –… … Related Domains –Tracking People –Detecting Pedestrians –... … Localizing Human Figures –Exemplar-based: [Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], … –Part-based: [Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], … –… …

MSRI University of California Berkeley 4 Beyond “Trees” A hard problem! More information is needed. Important cues that are NOT in the tree model: –Symmetry of clothing/color –“V-shape” formed by the upper legs –Distance/smooth connection between arms and legs –…… A hard problem! More information is needed. Important cues that are NOT in the tree model: –Symmetry of clothing/color –“V-shape” formed by the upper legs –Distance/smooth connection between arms and legs –…… ?

MSRI University of California Berkeley 5 Our Approach Preprocessing with Constrained Delaunay Triangulation Detecting Candidate Parts from Bottom-up Learning Pairwise Constraints between Parts Assembling Parts by Integer Quadratic Programming (IQP) Preprocessing with Constrained Delaunay Triangulation Detecting Candidate Parts from Bottom-up Learning Pairwise Constraints between Parts Assembling Parts by Integer Quadratic Programming (IQP)

MSRI University of California Berkeley 6 Constrained Delaunay Triangulation –Detect edges with Pb (Probability of Boundary) –Trace contours with Canny’s hysteresis –Recursively split contours into piecewise straight lines –Complete the partial graph with Constrained Delaunay Triangulation –Detect edges with Pb (Probability of Boundary) –Trace contours with Canny’s hysteresis –Recursively split contours into piecewise straight lines –Complete the partial graph with Constrained Delaunay Triangulation

MSRI University of California Berkeley 7 Detecting Parts using Parallelism (L 1,  1 ) (L 2,  2 ) T N Candidate parts as parallel line segments (Ebenbreite) (Scale-invariant) Features for parallelism: | Pb 1 + Pb 2 |/2, |  1 -  2 |, | L 1 - L 2 |/| L 1 + L 2 |, |( C 1 - C 2 )  T |/| L 1 + L 2 |, |( C 1 - C 2 )  N |/| L 1 + L 2 | Logistic Classifier Candidate parts as parallel line segments (Ebenbreite) (Scale-invariant) Features for parallelism: | Pb 1 + Pb 2 |/2, |  1 -  2 |, | L 1 - L 2 |/| L 1 + L 2 |, |( C 1 - C 2 )  T |/| L 1 + L 2 |, |( C 1 - C 2 )  N |/| L 1 + L 2 | Logistic Classifier C1C1 C2C2

MSRI University of California Berkeley 8 Pairwise Constraints between Parts Scale (width) consistency –Use anthropometric data as groundtruth Symmetry of appearance (color) Orientation consistency Connectivity –Short distance between adjacent parts –“Smooth” connection between non-adjacent parts short “gaps” on shortest path (on CDT graph) small maximum angle on the shortest path few T-junctions/turns on the shortest path Scale (width) consistency –Use anthropometric data as groundtruth Symmetry of appearance (color) Orientation consistency Connectivity –Short distance between adjacent parts –“Smooth” connection between non-adjacent parts short “gaps” on shortest path (on CDT graph) small maximum angle on the shortest path few T-junctions/turns on the shortest path C1C1 C2C2

MSRI University of California Berkeley 9 Learning Pairwise Constraints 15 hand-labeled images from a skating sequence Empirical distributions of some pairwise features For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian)

MSRI University of California Berkeley 10 Assembling Parts as Assignment Candidates {C i }Parts {L j } ( L j1,C i1 =  (L j1 ) ) ( L j2,C i2 =  (L j2 ) ) Cost for a partial assignment {(L j1,C i1 ), (L j2,C i2 )}: assignment 

MSRI University of California Berkeley 11 Assignment by IQP Suppose there are m parts and n candidates, the optimal assignment  minimizes a quadratic function Q(x)=x T Hx where x is a mn  1 indicator vector and H is of size mn  mn. This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions. We choose an approximation scheme which solves mn linear programs followed by gradient descent. The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems. We have m=9 and n~150; the total time is less than a minute. This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions. We choose an approximation scheme which solves mn linear programs followed by gradient descent. The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems. We have m=9 and n~150; the total time is less than a minute.

MSRI University of California Berkeley 12

MSRI University of California Berkeley 13

MSRI University of California Berkeley 14

MSRI University of California Berkeley 15 Conclusion To find people under general conditions, we need to go beyond the traditional tree-based model; Most important constraints for the human body are between pairs of body parts; Pairwise constraints may be learned from a small set of training examples; Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints. To find people under general conditions, we need to go beyond the traditional tree-based model; Most important constraints for the human body are between pairs of body parts; Pairwise constraints may be learned from a small set of training examples; Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.

MSRI University of California Berkeley 16