Multiscale Symmetric Part Detection and Grouping Alex Levinshtein, Sven Dickinson, University of Toronto and Cristian Sminchisescu, University of Bonn Introduction MotivationDemonstration Input image 1a. Multiscale superpixel segmentation 1b. Superpixel affinity (for each scale) 1c. Group superpixels into symmetric parts using the affinities 2a. Part affinity2b. Group parts 1. Part detection 2. Part grouping Key Idea Symmetric shape decompositions offer an effective representation for shape indexing and matching. Skeleton-based approaches assume correct figure-ground segmentation, precluding their application to cluttered scenes. Filter-based approaches are imprecise and offer poor precision- recall in both part detection and part grouping. Contour-based methods can suffer from high computational complexity and typically stop short of part grouping. How can we extract and group the symmetric parts of an object from a cluttered scene in a computationally efficient manner? Traditional skeletonization algorithms decompose a closed contour into symmetric parts and attachment relations – a highly top-down process. In contrast, we will detect symmetric parts locally and then assemble them into an object – a highly bottom-up process. We begin by segmenting the image into compact regions at multiple scales, representing a large set of (deformable) maximal disc hypotheses from which skeletal parts can be detected. Adjacent maximal disc hypotheses that form a symmetric part with strong image evidence are grouped. Finally, detected parts whose attachments are deemed nonaccidental are assembled to form objects. Features to compute part attachment affinity: Boundary evidence Appearance similarity Attachment category (based on main part axes) Cluster parts using affinities. Use standard graph clustering algorithm. (We use Felzenszwalb’s and Huttenlocher’s greedy clustering algorithm (2004) ) Select non-redundant parts from each cluster. Formulate as a quadratic program that minimizes overlap, # of selected parts, while maximizing the covered area. Probability part not redundant Define an adjacency graph for each superpixel scale. Each edge is assigned an affinity representing the likelihood that the two superpixels represent maximal discs of the same symmetric part. Affinity has shape and appearance components: –Shape: check the presence of symmetric edges –Appearance: homogeneity Shape features: Learn the affinity function from training data. Trained part detection and grouping on images from Weizmann Horse dataset (Borenstein and Ullman 2002). Quantitative evaluation of part detection on images from the Weizmann Horse dataset (compare to Lindeberg and Bretzner, 2003). Qualitative evaluation on images from other domains. 1. Part Detection Object partMaximal discs Superpixel approximation SVM Logistic Shape features Superpixel affinity Logistic Appearance features Redundant parts affinity Probability that part is redundant Attached parts affinity Part affinity ∙ ∙+ Results image main part clusters Too coarseToo fineGood scale Parts appear at different scales in the image Need at least one superpixel scale that captures each part well Solution: Use multiscale superpixel segmentation 1a. Multiscale superpixel segmentation 1b. Superpixel affinity 1c. Grouping superpixels into parts edge histogramedgeshistogram bins Cluster superpixels using affinities. Use standard graph clustering algorithm. (We use Felzenszwalb’s and Huttenlocher’s greedy clustering algorithm (2004) ) 2a. Part affinity 2b. Grouping parts into objects Features to determine part redundancy: Areaoverlap Boundaryoverlap Appearance similarity image main part clusters Approach A symmetric part is the locus of maximal disc centers (Blum 1967). Use superpixels at multiple scales as data- driven hypotheses of maximal discs. Identify groups of superpixels that have strong symmetric edge support. Input imageRegion skeleton Blobs/Ridges (Lindeberg and Bretzner) Contour Groups (Stahl and Wang) Approach Part affinity defined differently for redundant and attached parts. Redundant parts are assigned high affinity, while non-redundant parts are assigned an affinity based on evidence of attachment. Learn redundancy and attachment affinity from labeled training data. Goal: Group together parts likely to belong to the same object, based on detecting part attachments. Issue: Since the same part can be detected at different scales, resulting groups may contain redundant parts. Solution: –Greedily cluster parts based on computed part attachment affinities. –Decide which redundant parts to remove in the context of each cluster. 2. Part Grouping