Efficiently Selecting Regions for Scene Understanding M. Pawan Kumar Daphne Koller http://ai.stanford.edu/~pawan http://ai.stanford.edu/~koller Aim: To efficiently select accurate, discriminative regions for a high-level vision task Results Integer Program Semantic Segmentation Integer constraints yr(i)  {0,1} Background Dataset - 715 images S - set of super-pixels (intersection of segments) 80/20 train/test split, 4 folds Over-segmentations as Regions Simple Inference Minimize Energy r,r’  D miny r,i r(i)yr(i) + (r,r’),i,j rr’(i,j)yrr’(i,j) Baselines Pixel-based Model Uniqueness i yr(i) = 1 i  L’ = {0} (not selected)  L SEGMENTS RESULT SEGMENTS RESULT Intersection of over-segmentations j  L’, (r,r’) - neighbors Marginalization j yrr’(i,j) = yr(i) Lowest energy over-segmentation r “covers” s iL yr(i) = 1 Each super-pixel is covered Covering Gould et al., 2009 I M A G E Linear Programming Relaxation Relax yr(i)  [0,1] SCENE LAYOUT, Hoiem et al, 2005 MONOCULAR 3D, Saxena et al, 2008 Standard pairwise energy. Dual decomposition with tree slaves. SEGMENTS RESULT Segments are not accurate Do not align with scene boundaries Sub-dictionary DT  D that forms a tree. Union of DT = D. P I X E L Segments are not discriminative Too small to capture useful cues Minimize energy such that uniqueness, marginalization and integer constraints are satisfied. r1 r5 Belief propagation r2 r3 r4 SEGMENTATION, Gould et al, 2008 I N T E R Overview Region Selection using Energy Minimization Tree slaves do not enforce the covering constraint. Large number of possible pixel-to-region assignments Sub-dictionary DC  D that covers s. One slave for each s  S. S E G M Exact inference is intractable Move-making algorithm s Minimize energy such that exactly one region in DC is selected. Linear Search Over Regions DICTIONARY OF REGIONS D MERGE AND INTERSECT WITH SEGMENTS TO FORM PUTATIVE REGIONS r1 r3 r2 G O U L D Standard linear programming relaxation is not tight. Clique constraints Choose a subset of 3 super-pixels SQ  S. CURRENT REGIONS SEGMENTATIONS SELECT REGIONS Sub-dictionary DQ  D, each region covers at least one s  SQ ITERATE UNTIL CONVERGENCE O U R r1 r2 r3 r4 r5 r6 Minimize energy such that uniqueness, marginalization, covering and integer constraints are satisfied. Efficient Enumeration Select regions from D (with overlapping regions) such that No regions overlap Entire image is covered Integer Program Energy of the vision task is minimized Shrinking Strategy Maintain a Small Active Set of Regions De-activate regions not selected in any slave for T iterations Two dictionary moves Merge neighboring regions Merge and intersect regions with segments Iteratively introduce cliques with maximum active regions Statistically significant improvement