Poselets: Body Part Detectors Trained Using 3D Human Pose Annotations ZUO ZHEN 27 SEP 2011

Outline Introduction Related work Methods Experiments Conclusion and future work

Introduction The proposed poselet classifiers are directly trained to handle the visual variation associated with a common underlying semantics.

Introduction What is poselet? A poselet describes a particular part of the human pose under a given viewpoint. It is defined with a set of examples that are close in 3D configuration space. Two criteria of “good” Poselets 1.Easy to find the poselet given the input image. (Tightly clustered in appearance space) 2.Easy to localize the 3D configuration of the person conditioned on the detection of a poselet. (Tightly clustered in configuration space) Contribution 1.Propose a new notion of part, a “poselet”, and an algorithm for selecting good poselets. 2.Develop a novel dataset H3D(Humans in 3D) which is annotated with 3D configuration information.

Related work 1.Work in the pictorial structure tradition Disadvantage: most natural to construct kinematic simulations of a moving person, while may not correspond to the most salient features for visual recognition. 2.Work in the appearance based window classification tradition Disadvantage: not suitable for pose extraction or localization of the anatomical body parts or joints. 3.Work of hybrid approach which have stages of one type followed by a stage of another type Disadvantage: the parts themselves are not jointly optimized with respect to combined appearance and configuration space criteria

Left Hip Left Shoulder Method This paper use keypoints to annotate the joints, eyes, nose, etc. of people to find correspondence at training time

Method(H3D dataset) H3D dataset:  2000 human annotations  Images from Flickr with Creative Commons Attributions License4.  Provides annotation of 15 types of regions of a person, and 19 types of keypoint annotations.

Method (H3D dataset) Why 3D not 2D? 3D2D Use ratio of annotations contribute to the statistics Every annotationOnly frontal view annotations Sensitivity to foreshortening Not strongly affectedStrongly affected Whether allow for decomposing camera view point YesNA Whether allow for query for the appearance of poselets YesNA

Method (H3D dataset) Left: H3D can generate conditional region probability masks. Right: H3D can generate scatter plots of the 2D screen locations of the right elbow and left ankle given the locations of both shoulders.

Method (Finding Candidates) Define the (asymmetric) distance in configuration space from example s to example r as: Where = [x, y, z] are the normalized 3D coordinates of the i-th keypoint of the example s. The weight term is a Gaussian with mean at the center of the patch. The term is a penalty based on the visibility mismatch of keypoint i in the two examples.

Method (Generate Poselet Candidates) Example query regions (left column) and the corresponding closest matches in configuration space generated by H3D.

Method (Training Poselet classifiers) 1.Given a seed patch 2.Find the closest patch (search by running a scanning window over all positions and scales of all annotations) 3.Sort them by residual error 4.Threshold them 5.Select a small set of poselets that are: Individually effective and complementary 6.Use them as positive training examples to train a linear SVM with HOG features

Method (For Detection & Localization) The probability of detecting the object O at position x is: Where is the score that a poselet classifier assigns to location x and is the weight of the poselet, and the author use the Max Margin Hough Transform to learn the weight.

Experiments (1) Detecting Human Torsos ROC curve comparing the proposed torso detection performance together with other published detectors on the H3D test set

Experiments Examples of torso detections using poselets

Experiments (2) Detecting People on PASCAL VOC 2007 Outperform the part-based deformable detector on H3D but get comparable performance on VOC2007.

Experiments (3) Detecting Keypoints Detection rate of some keypoints conditioned on true positive torso detection.

Conclusion & Future Work Conclusion The authors propose a two-layer classification/ regression model for detecting people and localizing body components. And the 3D annotation guides the search for good parts. Future work Use H3D more widely.

Birdlets: Subordinate Categorization Using Volumetric Primitives and Pose-Normalized Appearance

Outline Introduction Related work Methods Experiments Conclusion

Introduction Application background Current research: two extremes of individuals and basic-level categories Few research on subordinate categorization What is subordinate categorization? Distinguish by the differing properties of parts.

Introduction Overview of the Proposed approach

Introduction Contribution 1.A framework for detecting volumetric part models 2.A pose-normalized appearance model for comparing part appearance 3.A classification model for aggregating information about part properties

Related work Image features Disadvantages: view-dependent, pose variation Part model Disadvantages: high intra-class variability, significant articulation Hierarchy model Disadvantages: subordinate categories have both subtle and drastic appearance variation Attribute model Disadvantages: Insufficient to model subtle differences between parts

Method Why birds? 1.Exist largest subordinate-level dataset (CUB- 200) 2.Conform with the definition of subordinate- level (share common structure & parts with many subtle part distinctions) 3.Involving highly variable appearances and articulations (challenging)

Method (PNAD) Post-normalized appearance descriptor (PNAD) 1.Map points on a unit sphere onto the ellipsoid’s surface for patch sampling 2.Project patches on ellipsoid surface to original image plane 3.Extend the projected patches for extracting SIFT descriptor 4.Concatenate the location and appearance information for forming PAND descriptor

Method (PNAD)

Method (Birdlet) Volumetric primitive templates 1.Two parts (head & body) 2.Two ellipsoids (parameters: location center, 3D orientation, scale) 3.Alignment (assisted by visible point features: beaktips, eyes, wingtips, feet and tails)

Method (Training & Testing) 1.Get selection windows for detecting objects and parts in testing image(both positive and negative examples for SVM classifier) 2.Get birdlets for integrated classification

Method (Integrated Classification) Stacked Evidence Trees model The Stacked Evidence Tree takes a test feature and finding a set of training features that are similar both in appearance and surface location, and ultimately returning the class label distribution across this similar set

Experiments Classification Confusion Matrices (a) the PHOW/SVM Baseline (37.12% MAP), (b) the PNAD-RF performance on the top 20% of detections (40.25% MAP), and (c) the PNAD-RF performance on the ground truth part locations (66.58% MAP).

Experiments Example Volumetric Primitive Detections Top two images: the bird is detected and localized with reasonable accuracy Low two images: false positive detections

Experiments Classification of Volumetric Detections. For the k top ranked detections, this plots the corresponding PNAD-RF classification performance (using mean-average precision)

Conclusion This paper presented an approach for subordinate categorization using a pose- normalized appearance representation founded upon a volumetric part model.