Unfolding an Indoor Origami World David Fouhey, Abhinav Gupta, Martial Hebert 1

2

3

Local Evidence 4 Hoiem et al. 2005, Saxena et al. 2005, Fouhey et al. 2013, etc.

Constraints 5

Constraints for Single Image 3D 6 Local Smoothness Low Level, Generic Hoiem et al. 2005, Saxena et al. 2005, 2008, Munoz et al., 2009, etc.

Constraints for Single Image 3D 7 Local Smoothness Low Level, Generic Hoiem et al. 2005, Saxena et al. 2005, 2008, Munoz et al., 2009, etc.

Constraints for Single Image 3D 8 Low Level, Generic High Level, Physical

High Level, Physical Constraints for Single Image 3D 9 Local Smoothness Low Level, Generic Coughlan and Yuille 2000, etc.

High Level, Physical Constraints for Single Image 3D 10 Local Smoothness Low Level, Generic Hedau et al. 2009, Del Pero et al., 2011, Wang et al., 2012, Schwing et al. 2012, etc.

High Level, Physical Constraints for Single Image 3D 11 Local Smoothness Low Level, Generic Lee et al. 2010, Xiao et al. 2012, Zhao et al. 2013, Schwing et al., 2013, etc.

High Level, Physical Constraints for Single Image 3D 12 Low Level, Generic

Mid-level in the Past 13 Huffman 71, Clowes 71, Kanade 80, 81 Sugihara 86, Malik 87, etc.

Our Mid-Level Constraints 14

This Work 15 Input: Single Image Output: Discrete Scene Parse

Overview 16 ParameterizationFormulation Experimental Results

Overview 17 ParameterizationFormulation Experimental Results

Parameterization 18

Parameterization 19 vp 1 vp 2 vp 3 VP Estimator from Hedau et al., 2009

Parameterization 20 Two VPs give grid cell

Encoding Surface Normals 21

Encoding Surface Normals 22

Encoding Surface Normals 23

Encoding Surface Normals 24 x 1,…, x 400 x 401,…, x 800 x 801,…, x 1200

Related Parameterizations 25 vp 1 vp 3 x2x2 x1x1 x3x3 x4x4 Hedau et al., 2009; Wang et al. 2010, Schwing et al., 2012, 2013 vp 2

Overview 26 ParameterizationFormulation Experimental Results

Parameterization 27

Formulation 28

Unaries 29

Unaries 30 Low c Any 3D Evidence High c

Unaries 31 … Input3D Primitive Bank Local: Data-Driven 3D Primitives Fouhey, Gupta, Hebert, 2013

Unaries 32 Hedau, Hoiem, Forsyth, 2009 Global: Cuboid Fit + Clutter Mask InputPredicted WallsClutter Mask

Binaries 33

Convex/Concave Constraints 34 Convex (+) Concave (-)

Convex/Concave Constraints 35 Detected Concave (-)

Convex/Concave Constraints 36 Detected Concave (-)

Convex/Concave Constraints 37 Detected Concave (-)

Convex/Concave Constraints 38 Detected Concave (-)

Convex/Concave Constraints 39 Detected Concave (-)

Detecting Convex/Concave 40 Ground-Truth Discontinuities similar to Gupta, Arbelaez, Malik, DP from Fouhey, Gupta, Hebert, 2013 Input3D Primitive Bank … Use 3DP to Transfer Discontinuities

Smoothness 41

Constraints 42

Solving the Model 43

Overview 44 ParameterizationFormulation Experimental Results

Dataset NYU Depth v2: 795 Train, 654 Test 45

Qualitative Results 46

Qualitative Results 47

Qualitative Results 48

Qualitative Results 49

Surface Connection Graphs 50 ConvexConcave

Baseline Primary Baseline: 3D Primitives (3DP) 51 Fouhey, Gupta, Hebert, 2013 OutputInput

Quantitative Results 52 Summary Stats (⁰) (Lower Better) % Good Pixels (Higher Better) 11.25⁰22.5⁰30⁰MeanMedian 3DP Hedau et al RMSE Lee et al Proposed

Quantitative Results 53

Failure Modes Mistaken but Confident Evidence 54

Failure Modes 55 Missing High-Level Modeling

Conclusions 56 Single Image Parameterization Formulation Discrete Parse

Thank You 57