Unsupervised Category Modeling, Recognition and Segmentation Sinisa Todorovic and Narendra Ahuja

What is Common in a Set of Images? Images possibly contain an object of interest Which objects appear frequently in the set? What properties are shared by similar objects in the set? Where are the objects?

Objective: Car Example occlusionno carmultiple cars learn car model unseen image segment all cars RESULT occlusion

Training Problem Definition GIVEN Images possibly containing frequent occurrences of similar objects DETERMINE If similar objects are present AND IF YES LEARN The model of similar objects GIVEN An unseen image DETECT, RECOGNIZE AND SEGMENT All occurrences of the learned category Testing

Prior Work Dominated By: Statistical modeling of local features: patches or curve fragments Trend: Object detection = Image classification Trend: Object segmentation = Object localization Trend: Object segmentation = Binary thresholding of a probabilistic map Hypothesize the number of objects and their parts Hypothesize the topology of object parts Each training image must contain a category of interest Modeling background Require typically hundreds of training images

Explicit modeling of recursive embedding of object subparts Regions vs. local features open questions: More informative? More stable and robust to noise? Regions allow: simultaneous object detection and segmentation explicit representation of the recursive embedding property Category Modeling is Very Difficult

Our Approach SIMILAR OBJECTS PRESENT IN THE SET MANY SUBIMAGES WITH SIMILAR REGION PROPERTIES ABUNDANT DATA ROBUST LEARNING IS FEASIBLE find ? do ? image matching structural learning

Region Properties Geometric Region area Boundary shape Photometric Gray-level contrast with the surround Topology Recursive containment of regions Layout - relative region locations

Feature Extraction = Image Segmentation Image Homogeneous regions at ALL contrasts and sizes segmentation [N. Ahuja TPAMI ‘96, Tabb & Ahuja TIP ‘97, Arora & Ahuja ICPR ‘06] Example segmentations for several contrasts

Multiscale Segmentation to Segmentation Tree Sample cutsets Segmentation tree Contrast level ≠ Tree level Example segmentations

Image = Tree and Object = Subtree

Outline of Our Approach Images = Trees Category present = Many similar subtrees Extracting similar subtrees = Tree matching Category model = Union of similar subtrees Simultaneous detection, recognition and segmentation of ALL category instances by Matching the model with an image

Tree Matching: Structural Noise Edit-distance tree matching [Pelillo et a. PAMI‘99, Torsello&Hancock ECCV’02, PRL’03]

Matching Algorithm Input treesMatched subtrees

which MAXIMIZES their similarity measure Matching Algorithm FIND bijection while PRESERVING ancestor-descendant relationships GIVEN two trees: node saliency cost of node matching

Matching Algorithm: Recursive Solution Maximum clique over all descendant pairs descendants SOLUTION Select all pairs with > threshold.

Outline LEARNING

Category Model = Tree Union Learning algorithm estimates: 1) Model structure 2) Model parameters Tree intersection: Tree union:

Simultaneous Detection and Segmentation MATCHING

Performance Evaluation Criteria Ground Truth (GT) Matched Subtrees (MST) DETECTION ERROR False positive: “intersection of MST and GT” < 0.5 “union of MST and GT” SEGMENTATION ERROR Matched Subtrees (MST) Ground Truth (GT) “XOR of MST and GT”

10 positive out of 20 training images 5 positive out of 10 training images Results on test images: Results: UIUC Cars Side View

6 positive out of 12 training images 3 positive out of 6 training images Results on test images: Results: Faces -- Caltech 101 Database

Results: Caltech Cars Rear View 10 positive out of 20 training images

Recall-Precision Training from a small-size dataset Varying tradeoff recall vs. precision

Complexity and Runtime on 2.4GHZ 2GB RAM PC Training on 20 images of UIUC CARS: < 2 hours # of tree nodes Extracting similar subtrees:per image pair # of subtree nodes Learning on 32 subtrees extracted for UIUC CARS: < 1 hour Learning: # of model nodes Processing time for UIUC CARS: < 10 sec, regardless of the total number of target objects Detection, recognition and segmentation:

Summary and Conclusion Unsupervised learning of an unknown category frequently occurring in a given set of images Region-based, structural approach Simultaneous detection, recognition, and segmentation of all category instances in unseen images NO multiple detections on the same object NO hypotheses on the number of objects and their parts NO hypotheses on the topology of object parts Small number of training images Complexity comparable with standard methods

Acknowledgment THANK YOU! {sintod,