1. 1 Occlusions – the world is flat without them! 16-721: Learning-Based Methods in Vision A. Efros, CMU, Spring 2009

2. 2 Our Main Challenge Recovering 3D geometry from single 2D projection Infinite number of possible solutions! Need to know which boundaries are depth discontinuities i.e. occlusions …

3. Contour ownership  A contour belongs to one of the two (but not both) abutting regions. Figure (face) Ground (shapeless) Figure (Goblet) Ground (Shapeless) Important for the perception of shape

4. © Stephen E. Palmer, 2002 Properties of figures vs. grounds 15.18 FigureGround Thing-likeNot thing-like CloserFarther ShapedExtends behind Figure-Ground Organization

5. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Surroundedness 15.19 Figure-Ground Organization Surrounded region --> Figure Surrounding region --> Ground

6. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Size 15.20 Figure-Ground Organization Smaller region --> Figure Larger region --> Ground

7. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Orientation 15.21 Figure-Ground Organization Horizontal/vertical region --> Figure Oblique region --> Ground

8. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Contrast 15.22 Figure-Ground Organization Higher contrast region --> Figure Lower contrast region --> Ground

9. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Symmetry 15.23 Figure-Ground Organization Symmetrical region --> Figure Asymmetrical region --> Ground

10. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Convexity 15.24 Figure-Ground Organization More convex region --> Figure Less convex region --> Ground

11. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Parallelism 15.25 Figure-Ground Organization More parallel region --> Figure Less parallel region --> Ground

12. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Lower region 15.26 Figure-Ground Organization Lower region --> Figure Upper region --> Ground

13. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Meaningfulness 15.27 Figure-Ground Organization More meaningful region --> Figure Less meaningful region --> Ground

14. © Stephen E. Palmer, 2002 Relation to Depth Factors 15.28 Figure-Ground Organization Figure-ground organization as edge assignment: To which side does the edge belong? Depth cues can also be figure-ground factors and Figure-ground factors can be depth cues. To the closer side. This fact connects figure-ground organization with depth perception.

15. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Occlusion 15.29 Figure-Ground Organization Occluding region --> Figure Occluded region --> Ground

16. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Cast Shadows 15.30 Figure-Ground Organization Shadowing region --> Figure Shadowed region --> Ground

17. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Shading 15.32 Figure-Ground Organization Shaded region --> Figure Nonshaded region --> Ground

18. Line Labeling > : contour direction + : convex edge - : concave edge possible junctions (constraints) Constraint Propagation [Clowes 1971, Huffman 1971; Waltz 1972; Malik 1986]

19. 19

20. Junctions in Real Images

21. Are Junctions local evidence? J McDermott, 2004

22. Figure/Ground in Natural Images

23. Xiaofeng Ren, Charless Fowlkes and Jitendra Malik University of California, Berkeley ECCV 2006 GrazComputer Vision Group Figure/Ground Assignment in Natural Images

24. Figure/Ground: Groundtruth F G

26. Figure/Ground Dataset [Martin, Fowlkes & Malik; ECVP 2003]

27.  Local Cues  Gestalt Principles of Figure/Ground  Global Cues  Label Consistency at T-junctions Cues for Figure/Ground [Kienker, Sejnowski, Hinton & Schumacher 1986] [Heitger & von der Heydt 1993] [Geiger, Kumaran & Parida 1996] [Saund 1999] [Yu, Lee and Kanade 2001] …

28. Shapemes: Prototypical Local Shapes …… local shapes collect cluster Use Geometric Blur [Berg & Malik 2001]

29. parallelism convexity straight line corner line ending

30. Gestalt Principles for Figure/Ground  Convexity  Parallelism  Surroundedness  Symmetry  Lower Region  Common Fate …… F G F G G

31. Shapemes for F/G Discrimination LR L:93.8% L:49.8% L:89.6% L:11.7% L:66.5% L: 5.0% Which side is Figure? Train a logistic classifier to linearly combine the shapeme cues

32. Global Consistency F G F F G G common F G F G G F uncommon

33. Building a CRF Model  What are the features?  edge features:  Shapemes  junction features:  Junction type  How to make inference?  Loopy Belief Propagation  How to learn the parameters?  Gradient Descent on Max. Likelihood  What are the features?  edge features:  Shapemes  junction features:  Junction type  How to make inference?  Loopy Belief Propagation  How to learn the parameters?  Gradient Descent on Max. Likelihood X={X 1,X 2,…,X m } Estimate P(X i |  )

34. Junction Features One feature for each junction type F G F G { (G,F),(F,G) } F G F G G F { (F,G),(F,G),(F,G) } Junction potentials: { (F,G),(G,F),(F,G) } F G F F G G

35. Continuity in Figure/Ground If a pair of edges belong to the same foreground, they should have a smooth connection. F G F F G G G F F F G G  

36. Learning Junction Weights F G F G  = 0.185 F G G F  = -0.611 F G F F G G  = 0.428 F G F G G F  = -0.857

37. Experiments  Using human-marked segmentations  Using edges computed by an edge detector

38. Results Human-marked Segmentation Edges computed with an edge detector Chance Baseline Size/Convexity Local Shapemes Averaging shapemes on boundaries Shapemes + CRF Dataset Consistency 50% 88%

39. Baseline: Size/Convexity A B Size(A) < Size(B): A is figure; B is ground Size(A) > Size(B): B is figure; A is ground

40. Results Human-marked Segmentation Edges computed with an edge detector Chance Baseline Size/Convexity Local Shapemes Averaging shapemes on boundaries Shapemes + CRF Dataset Consistency 50% 88% 55.6% 64.8% 72.0% 78.3% ------ 64.9% 66.5% 68.9%

41. Image GroundtruthLocalGlobal Using human segmentations

42. Image Edge MapLocalGlobal Using edge maps computed from an edge detector

43. 43 Recovering Occlusion Boundaries from a Single Image Derek Hoiem* Andrew Stein Alexei Efros Martial Hebert Carnegie Mellon University Robotics Institute * Now at University of Illinois

44. 44 Recover Major Occlusions

45. 45 Prior Work: Finding Boundaries NCuts Segmentation NCuts: [Cour et al. 2004] Input ImagePb Boundaries Pb: [Martin et al. 2002]

46. 46 Segmentation into Physical Boundaries

47. 47 Recover Major Occlusions Occlusion Boundaries Inferred Depth

48. 48 Start with Oversegmentation Initial Segmentation Occlusion boundary? R1R1 R2R2

49. 49 2D Cues for Occlusions Region: Color and TextureBoundaries: Strength and Continuity

50. 50 2D Junctions Image 2D Boundary T-Junction 1 2 3

51. 51 3D Surface Clues for Occlusions Support Planar PorousSky Surface Labels Geometric T-Junction 1 2 3 Solid

52. 52 3D Depth Cues for Occlusion Surfaces Initial Boundaries Depth Underestimate Depth Overestimate

53. 53 Illustration of Depth Range SKY SUPPORT Image Depth (Max) Depth (Min)

54. 54 Gradual Occlusion Inference Initial SegmentationFinal Boundaries ? Initial Depth (Min) Initial Depth (Max)

55. 55 Gradual Occlusion Inference P(occlusion) Soft Boundary MapStage 1 Result

56. 56 Gradual Occlusion Inference P(occlusion) Soft Boundary MapStage 1 Result

57. 57 Gradual Occlusion Inference P(occlusion) + CRF(continuity, closure) Soft-Max Boundary MapStage 2 Result

58. 58 Gradual Occlusion Inference Stage 3 Result P(occlusion) + CRF(continuity, closure, surfaces) Soft-Max Boundary Map

59. 59 Final Estimate Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

60. 60 Evaluation … Training: 50 images Testing: 250 images (50 quantitative)

61. 61 Occlusion vs. Non-Occlusion

62. 62 Foreground/Background Accuracy Edge/Region Cues+ 3D CuesWith CRF Stage 158.7%71.7% Stage 265.4%75.6%77.3% Stage 368.2%77.1%79.9% Ours Shapemes + CRF Pb Boundaries68.9% Human Boundaries78.3% Ren et al. 2006, Corel Images

63. 63 Occlusion Result Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

64. 64 Occlusion Result Boundaries, Foreground/Background, Contact Depth (Max) Depth (Min)

65. 65 3D Model with Occlusions 3D Model without Occlusion Reasoning 3D Model with Occlusion Reasoning