Coherent Scene Understanding with 3D Geometric Reasoning Jiyan Pan 12/3/2012

Task Detect objects Identify surface regions Estimate ground plane Infer gravity direction Geometrically coherent in the 3D world 3D geometric context

xbxb dbdb dtdt γ nvnv θ xtxt npnp hphp ngng α H f ground plane image plane (inverse) gravity ground plane orientation ground plane height object vertical orientation real world height object depth camera center focal length object pitch and roll angles object landmarks Coordinate system Deterministic relationships Variables of global 3D geometries: n g, n p, h p

xbxb dbdb dtdt γ nvnv θ xtxt npnp hphp ngng α H f ground plane image plane (inverse) gravity ground plane orientation ground plane height object vertical orientation real world height object depth camera center focal length object pitch and roll angles object landmarks Coordinate system Probabilistic relationships Derived from appearance Prior knowledge

Can we solve them all for a coherent solution? Non-linear Non-deterministic Even invalid equations from false detections

√ √ √ √ X Global 3D context Local 3D context

√ √ √ √ X “Chicken and egg” problem:  Local entities could be validated by global 3D context  Global 3D context is induced from local entities Global 3D context Local 3D context ?

Possible solution (All in PGM) Put both global 3D geometries and local entities in a PGM [1] – Precision issue: Have to quantize continuous variables – Complexity issue: Pairwise potential would contain up to ~1e6 entries [1] D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008 Ground o1o1 o2o2 okok Gravity 100(pitch) × 100 (roll) × 100 (height)

Possible solution (Fixed global geometries as hypotheses) Task much easier under a fixed hypothesis of global 3D geometries Ground o1o1 o2o2 okok Gravity × × × × ××

Task much easier under a fixed hypothesis of global 3D geometries Possible solution (Fixed global geometries as hypotheses) o1o1 o2o2 okok ω1ω1 ω2ω2 ω3ω3 How to generate global 3D geometry hypotheses?

Possible solution (Hypotheses by exhaustive search) Exhaustive search over the quantized space of global 3D geometries [2] – Computational complexity tends to limit search space [2] S. Bao et al. Toward coherent object detection and scene layout understanding. IVC, 2011

Possible solution (Hypotheses by Hough voting) Each local entity casts vote to the Hough voting space of the global 3D geometries and peaks are selected [3] – False detections could corrupt the votes – Would applying EM help? Not likely, if false detections overwhelm [3] M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities – Compared to averaging over all local entities: More robust against outliers – Compared to directly using estimates from each single local entity: More robust against noise L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

Number of random mixtures Minimum hypothesis error Gravity Direction Individual Mixture Average

Number of random mixtures Minimum hypothesis error Ground Plane Orientation Individual Mixture Average

√ √ √ √ X Local 3D context Global 3D context

3D geometric context ground plane orientation valid invalid (#1) ground plane #1: Common ground (global)

3D geometric context #2: Gravity direction (global) (inverse) gravity ground plane orientation invalid (#2) ground plane

3D geometric context #3: Depth ordering (local) (inverse) gravity ground plane orientation incompatible (#3) ground plane

3D geometric context #4: Space occupancy (local) (inverse) gravity ground plane orientation incompatible (#4) ground plane

Global geometric compatibility for an object: Orientation: Given a global 3D geometry hypothesis

Global geometric compatibility for an object: Orientation: Height: Given a global 3D geometry hypothesis

Global geometric compatibility for a surface: Orientation: local estimates vs. or Location: horizontal surface region vs. ground horizon Given a global 3D geometry hypothesis

Local geometric compatibility for two objects: Depth ordering: Space occupancy: Given a global 3D geometry hypothesis

Objective function of the CRF: Given a global 3D geometry hypothesis

√ √ √ √ X Local 3D context Global 3D context Best hypothesis

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

False Positive per Image True Positive Rate Deformable Part Model Detector Baseline Hoiem Ours 3D geometric reasoning improves object detection performance D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

False Positive per Image True Positive Rate Dalal-Triggs Detector Baseline Hoiem Ours 3D geometric reasoning improves object detection performance D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

Improvement in AP over baseline detector Ours 10.4% Hoiem 4.8% Sun 5.1% M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, D geometric reasoning improves object detection performance

Horizon estimation median error Ours 2.05⁰ Hoiem 3.15⁰ Sun 2.41⁰ M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

√ √ √ √ X Local 3D context Global 3D context Best hypothesis

Contributions of different geometric context False Positive per Image True Positive Rate Detection ROC Curve Det Det+IdvlGeo Det+PairGeo Det+FullGeo

Benefit is mutual Error in gravity direction Error in ground orientation Vanishing points alone 2.62⁰4.85⁰ Whole system 2.05⁰2.21⁰

Extensions – Improved depth ordering constraint – Local geometric constraints involving vertical surfaces – Multiple supporting planes – Using more prior knowledge of objects – Utilizing semantic categories of surface regions

closer object farther object closer object farther object occlusion mask of the farther object intersection region of the two object masks √ X Fully cover?

Occlusion: bottleneck in our system – Missed detection – Erroneous estimation of local properties – Less effective depth ordering constraint

Generalized Hough voting: better at handle occlusions K. Rematas et al. CORP 2011 B. Leibe et al. IJCV 2008

Occlusion-and-geometry-aware Hough voting

√ √ √ √ X Local 3D context Global 3D context Best hypothesis

So far we have treated the entire region labeled as "vertical" as a whole

Decompose vertical region into surface segments Occlusion boundary recovery (Hoiem et al. IJCV’11) Vanishing line sweeping (Lee et al. CVPR’09)

ground plane inverse gravity √ vertical surface candidate 1 vertical surface candidate 2

ground plane vertical surface candidate 1 inverse gravity vertical surface candidate 2 X

ground plane vertical surface candidate inverse gravity object candidate √

ground plane vertical surface candidate inverse gravity X

Given object layout, erect surfaces one by one “Interpretation by synthesis” (Gupta et al. ECCV’10)

supporting plane 1

supporting plane 2

ground plane

w l β

Spring 2013 (ICCV’13 submission) – Improved depth ordering constraint – Using more prior knowledge of objects – Multiple supporting planes Fall 2013 (CVPR’14 submission) – Local geometric constraints involving vertical surfaces – Utilizing semantic categories of surface regions During Spring Semester of 2014 – Thesis writing

Expected Contributions Systematically model the relationships among global and local geometric variables Develop a RANSAC-CRF scheme to handle non-linear, non-deterministic, and possibly invalid relationships Occlusion-and-geometry-aware object detection for finer depth order reasoning Joint reasoning among global geometries, surface segments, and objects

Thank you!