Computer Vision Group University of California Berkeley Estimating Human Body Configurations using Shape Context Matching Greg Mori and Jitendra Malik

Computer Vision Group University of California Berkeley Problem

Computer Vision Group University of California Berkeley Approach: Exemplar-based Matching Set of stored exemplars with hand-labeled keypoints Obtain sample points Deformable matching to exemplars: –Shape context matching to get correspondences –Kinematic chain deformation model Estimate 3D body configuration

Computer Vision Group University of California Berkeley Comparing Pointsets

Computer Vision Group University of California Berkeley Shape Context Count the number of points inside each bin, e.g.: Count = 4 Count = FCompact representation of distribution of points relative to each point

Computer Vision Group University of California Berkeley Shape Context

Computer Vision Group University of California Berkeley Comparing Shape Contexts Compute matching costs using Chi Squared distance: Recover correspondences by solving linear assignment problem with costs C ij [Jonker & Volgenant 1987]

Computer Vision Group University of California Berkeley Deformable Matching Kinematic chain-based deformation model Use iterations of correspondence and deformation Keypoints on exemplars are deformed to locations on query image

Computer Vision Group University of California Berkeley

Computer Vision Group University of California Berkeley Problem

Computer Vision Group University of California Berkeley Estimate 3D Body Configuration [Taylor ’00] Known: –Relative lengths of body segments –(x,y) Image locations of keypoints –“closer endpoint” labels for each segment –Scaled orthographic camera model Solve for 3D locations of keypoints up to some scale factor –Scale factor can be estimated automatically

Computer Vision Group University of California Berkeley Solving for Foreshortening

Computer Vision Group University of California Berkeley Choosing Scale

Computer Vision Group University of California Berkeley Results

Computer Vision Group University of California Berkeley

Computer Vision Group University of California Berkeley Multiple Exemplars Parts-based approach –Use a combination of keypoints/whole limbs from different exemplars –Reduces the number of exemplars needed Compute a matching cost for each limb from every exemplar Compute pairwise “consistency” costs for neighbouring limbs Use dynamic programming to find best K configurations

Computer Vision Group University of California Berkeley Parts-based Approach

Computer Vision Group University of California Berkeley Tracking by Repeated Finding