1 Occlusions – the world is flat without them! : Learning-Based Methods in Vision A. Efros, CMU, Spring 2009

2 Our Main Challenge Recovering 3D geometry from single 2D projection Infinite number of possible solutions! Need to know which boundaries are depth discontinuities i.e. occlusions …

Contour ownership  A contour belongs to one of the two (but not both) abutting regions. Figure (face) Ground (shapeless) Figure (Goblet) Ground (Shapeless) Important for the perception of shape

© Stephen E. Palmer, 2002 Properties of figures vs. grounds FigureGround Thing-likeNot thing-like CloserFarther ShapedExtends behind Figure-Ground Organization

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Surroundedness Figure-Ground Organization Surrounded region --> Figure Surrounding region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Size Figure-Ground Organization Smaller region --> Figure Larger region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Orientation Figure-Ground Organization Horizontal/vertical region --> Figure Oblique region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Contrast Figure-Ground Organization Higher contrast region --> Figure Lower contrast region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Symmetry Figure-Ground Organization Symmetrical region --> Figure Asymmetrical region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Convexity Figure-Ground Organization More convex region --> Figure Less convex region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Parallelism Figure-Ground Organization More parallel region --> Figure Less parallel region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Lower region Figure-Ground Organization Lower region --> Figure Upper region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Meaningfulness Figure-Ground Organization More meaningful region --> Figure Less meaningful region --> Ground

© Stephen E. Palmer, 2002 Relation to Depth Factors Figure-Ground Organization Figure-ground organization as edge assignment: To which side does the edge belong? Depth cues can also be figure-ground factors and Figure-ground factors can be depth cues. To the closer side. This fact connects figure-ground organization with depth perception.

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Occlusion Figure-Ground Organization Occluding region --> Figure Occluded region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Cast Shadows Figure-Ground Organization Shadowing region --> Figure Shadowed region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Shading Figure-Ground Organization Shaded region --> Figure Nonshaded region --> Ground

Line Labeling > : contour direction + : convex edge - : concave edge possible junctions (constraints) Constraint Propagation [Clowes 1971, Huffman 1971; Waltz 1972; Malik 1986]

19

Junctions in Real Images

Are Junctions local evidence? J McDermott, 2004

Figure/Ground in Natural Images

Xiaofeng Ren, Charless Fowlkes and Jitendra Malik University of California, Berkeley ECCV 2006 GrazComputer Vision Group Figure/Ground Assignment in Natural Images

Figure/Ground: Groundtruth F G

Figure/Ground Dataset [Martin, Fowlkes & Malik; ECVP 2003]

 Local Cues  Gestalt Principles of Figure/Ground  Global Cues  Label Consistency at T-junctions Cues for Figure/Ground [Kienker, Sejnowski, Hinton & Schumacher 1986] [Heitger & von der Heydt 1993] [Geiger, Kumaran & Parida 1996] [Saund 1999] [Yu, Lee and Kanade 2001] …

Shapemes: Prototypical Local Shapes …… local shapes collect cluster Use Geometric Blur [Berg & Malik 2001]

parallelism convexity straight line corner line ending

Gestalt Principles for Figure/Ground  Convexity  Parallelism  Surroundedness  Symmetry  Lower Region  Common Fate …… F G F G G

Shapemes for F/G Discrimination LR L:93.8% L:49.8% L:89.6% L:11.7% L:66.5% L: 5.0% Which side is Figure? Train a logistic classifier to linearly combine the shapeme cues

Global Consistency F G F F G G common F G F G G F uncommon

Building a CRF Model  What are the features?  edge features:  Shapemes  junction features:  Junction type  How to make inference?  Loopy Belief Propagation  How to learn the parameters?  Gradient Descent on Max. Likelihood  What are the features?  edge features:  Shapemes  junction features:  Junction type  How to make inference?  Loopy Belief Propagation  How to learn the parameters?  Gradient Descent on Max. Likelihood X={X 1,X 2,…,X m } Estimate P(X i |  )

Junction Features One feature for each junction type F G F G { (G,F),(F,G) } F G F G G F { (F,G),(F,G),(F,G) } Junction potentials: { (F,G),(G,F),(F,G) } F G F F G G

Continuity in Figure/Ground If a pair of edges belong to the same foreground, they should have a smooth connection. F G F F G G G F F F G G  

Learning Junction Weights F G F G  = F G G F  = F G F F G G  = F G F G G F  =

Experiments  Using human-marked segmentations  Using edges computed by an edge detector

Results Human-marked Segmentation Edges computed with an edge detector Chance Baseline Size/Convexity Local Shapemes Averaging shapemes on boundaries Shapemes + CRF Dataset Consistency 50% 88%

Baseline: Size/Convexity A B Size(A) < Size(B): A is figure; B is ground Size(A) > Size(B): B is figure; A is ground

Results Human-marked Segmentation Edges computed with an edge detector Chance Baseline Size/Convexity Local Shapemes Averaging shapemes on boundaries Shapemes + CRF Dataset Consistency 50% 88% 55.6% 64.8% 72.0% 78.3% % 66.5% 68.9%

Image GroundtruthLocalGlobal Using human segmentations

Image Edge MapLocalGlobal Using edge maps computed from an edge detector

43 Recovering Occlusion Boundaries from a Single Image Derek Hoiem* Andrew Stein Alexei Efros Martial Hebert Carnegie Mellon University Robotics Institute * Now at University of Illinois

44 Recover Major Occlusions

45 Prior Work: Finding Boundaries NCuts Segmentation NCuts: [Cour et al. 2004] Input ImagePb Boundaries Pb: [Martin et al. 2002]

46 Segmentation into Physical Boundaries

47 Recover Major Occlusions Occlusion Boundaries Inferred Depth

48 Start with Oversegmentation Initial Segmentation Occlusion boundary? R1R1 R2R2

49 2D Cues for Occlusions Region: Color and TextureBoundaries: Strength and Continuity

50 2D Junctions Image 2D Boundary T-Junction 1 2 3

51 3D Surface Clues for Occlusions Support Planar PorousSky Surface Labels Geometric T-Junction Solid

52 3D Depth Cues for Occlusion Surfaces Initial Boundaries Depth Underestimate Depth Overestimate

53 Illustration of Depth Range SKY SUPPORT Image Depth (Max) Depth (Min)

54 Gradual Occlusion Inference Initial SegmentationFinal Boundaries ? Initial Depth (Min) Initial Depth (Max)

55 Gradual Occlusion Inference P(occlusion) Soft Boundary MapStage 1 Result

56 Gradual Occlusion Inference P(occlusion) Soft Boundary MapStage 1 Result

57 Gradual Occlusion Inference P(occlusion) + CRF(continuity, closure) Soft-Max Boundary MapStage 2 Result

58 Gradual Occlusion Inference Stage 3 Result P(occlusion) + CRF(continuity, closure, surfaces) Soft-Max Boundary Map

59 Final Estimate Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

60 Evaluation … Training: 50 images Testing: 250 images (50 quantitative)

61 Occlusion vs. Non-Occlusion

62 Foreground/Background Accuracy Edge/Region Cues+ 3D CuesWith CRF Stage 158.7%71.7% Stage 265.4%75.6%77.3% Stage 368.2%77.1%79.9% Ours Shapemes + CRF Pb Boundaries68.9% Human Boundaries78.3% Ren et al. 2006, Corel Images

63 Occlusion Result Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

64 Occlusion Result Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

65 3D Model with Occlusions 3D Model without Occlusion Reasoning 3D Model with Occlusion Reasoning