Download presentation
Presentation is loading. Please wait.
Published byJuliana Randall Modified over 9 years ago
1
Recovering Surface Layout from a Single Image D. Hoiem, A.A. Efros, M. Hebert Robotics Institute, CMU Presenter: Derek Hoiem CS 598, Spring 2009 Jan 29, 2009
2
Why worry about 3d scenes?
3
Reason 1: We may want to interact with the scene NavigationManipulation
4
4 Reason 2: We need context
5
5
6
2D Object Detection
7
What the 2D Detector Sees
8
Computers need context too True Detection True Detections Missed False Detections Local Detector: [Dalal-Triggs 2005]
9
9 Context in Image Space [Kumar Hebert 2005] [Torralba Murphy Freeman 2004] [He Zemel Cerreira-Perpiñán 2004]
10
We need 3d info to reason about 3d relationships Close Not Close
11
How to represent scene space?
12
Holistic Scene Space: “Gist” Oliva & Torralba 2001 Torralba & Oliva 2002
13
How to represent scene space? Depth Map Saxena, Chung & Ng 2005, 2007
14
Gibson’s Surface Layout slide from Aude Oliva Gibson: “The elementary impressions of a visual world are those of surface and edge.” The Perception of the Visual World (1950) Focus on texture gradients
15
Surface Layout (Gibson cont.) slide from Aude Oliva Gibson’s Surface Layout
16
Surface Layout (Gibson cont.) slide from Aude Oliva Gibson’s Surface Layout
17
Marr’s 2½D Sketch Marr’s 2½-D Sketch Figs from Aude Oliva slide
18
Surface Layout (this paper) Goal: Label image into 7 Geometric Classes: Support Vertical – Planar: facing Left ( ), Center ( ), Right ( ) – Non-planar: Solid (X), Porous or wiry (O) Sky
19
Our Main Challenge Recovering 3D geometry from single 2D projection Infinite number of possible solutions! …
20
Our World is Structured Abstract World Our World Image Credit (left): F. Cunin and M.J. Sailor, UCSD
21
Most Early Work Tried to Manually Specify the Structure Hansen & Riseman 1978 (VISIONS) Barrow & Tenenbaum 1978 (Intrinsic Images) Brooks 1979 (ACRONYM) Marr 1982 (2½ D Sketch) Ohta & Kanade 1978 Guzman 1968
22
Learn the Structure of the World …
23
Infer Most Likely Scene Unlikely Likely
24
1. Use All Available Cues Vanishing points, lines Color, texture, image location Texture gradient
25
Use All Available Cues
26
2. Get Good Spatial Support 50x50 Patch
27
Image Segmentation Single segmentation won’t work Solution: multiple segmentations …
28
… … For each segment: - Get P(good segment | data) P(label | good segment, data) Labeling Segments
29
Image Labeling … Labeled Segmentations Labeled Pixels
30
30 … Gray? High in Image? Many Long Lines? Yes No Yes Very High Vanishing Point? High in Image? Smooth?Green? Blue? Yes No Yes Decision Trees + Adaboost Ground Vertical Sky Collins et al. 2002
31
Surface Confidence Maps P(Support)P(Vertical)P(Sky) P(Planar Left)P(Planar Center)P(Planar Right) P(Non-Planar Porous) P(Non-Planar Solid) Test Image
32
Experiments: Input Image
33
Experiments: Ground Truth
34
Experiments: Our Result
35
Surface Estimates: Outdoor Input ImageGround TruthOur Result Avg. Accuracy Main Class: 88% Subclass: 62%
36
Input Image Ground TruthOur Result Surface Estimates: Outdoor
37
Input Image Ground TruthOur Result Surface Estimates: Outdoor
38
Surface Estimates: Paintings Input Image Our Result
39
Surface Estimates: Indoor Avg. Accuracy Main Class: 93% Subclass: 76% Input ImageGround TruthOur Result
40
Failures: Reflections and Shadows Input Image Our Result
41
Average Accuracy Main Class: 88% Subclasses: 61%
42
Importance of Many Cues AllPosition Only Color Only Texture Only Perspective Only Main 88% 83%72%80%68% Subclass 61% 43% 55%52% AllAll But Position All But Color All But Texture All But Perspective Main 88% 84%87% 88% Subclass 61% 60% 58%57%
43
Importance of Many Cues
44
Spatial Support Matters
45
Automatic Photo Popup Labeled Image Fit Ground-Vertical Boundary with Line Segments Form Segments into Polylines Cut and Fold Final Pop-up Model [Hoiem Efros Hebert 2005]
46
video
47
Surfaces Not Enough – Need Occlusion Reasoning ImageSurface Labels 3D Model
48
Surfaces + Occlusions + Objects = Better 3D Models Surfaces Occlusions Objects and Viewpoint Support Horizon, Object Maps Surface Maps Depth, Boundaries Boundaries Horizon, Object Maps Viewpoint/Size Reasoning
49
video 2
50
Contributions General principles – Learn the structure of the world – Use all available cues – Spatial support matters – Use redundancy to deal with unreliable processes (segmentation) Results include entire spread of failure and success First work to convincingly demonstrate single-view reconstruction
51
Criticisms Still just 2D pattern recognition? Not clear how to generalize to arbitrary 3d angles Restricted to visible portion of scene Coarse layout: not clear if applicable to personal space or object shapes
52
Ideas for improvement Try improving features (e.g., add bag of words) Extend to characterize object shapes? Combine this surface-based layout with depth estimates from Saxena et al.
53
Discussion Use for context (Eamon) Multiple segmentations (Duan, Sanketh) Subcategories (Duan, Sanketh) Global info, use of object knowledge (Binbin) Combination with multiview cues (Mani) Landmarks (Gang)
54
Thank you
55
Things to cover when you present Background Overview of method Results Things you like Things you don’t Ideas for improvement Address bulletin board postings
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.