1. Coherent Scene Understanding with 3D Geometric Reasoning Jiyan Pan 12/3/2012

3. Task Detect objects Identify surface regions Estimate ground plane Infer gravity direction Geometrically coherent in the 3D world 3D geometric context

4. xbxb dbdb dtdt γ nvnv θ xtxt npnp hphp ngng α H f ground plane image plane (inverse) gravity ground plane orientation ground plane height object vertical orientation real world height object depth camera center focal length object pitch and roll angles object landmarks Coordinate system Deterministic relationships Variables of global 3D geometries: n g, n p, h p

5. xbxb dbdb dtdt γ nvnv θ xtxt npnp hphp ngng α H f ground plane image plane (inverse) gravity ground plane orientation ground plane height object vertical orientation real world height object depth camera center focal length object pitch and roll angles object landmarks Coordinate system Probabilistic relationships Derived from appearance Prior knowledge

6. Can we solve them all for a coherent solution? Non-linear Non-deterministic Even invalid equations from false detections

7. √ √ √ √ X Global 3D context Local 3D context

8. √ √ √ √ X “Chicken and egg” problem:  Local entities could be validated by global 3D context  Global 3D context is induced from local entities Global 3D context Local 3D context ?

9. Possible solution (All in PGM) Put both global 3D geometries and local entities in a PGM [1] – Precision issue: Have to quantize continuous variables – Complexity issue: Pairwise potential would contain up to ~1e6 entries [1] D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008 Ground o1o1 o2o2 okok Gravity 100(pitch) × 100 (roll) × 100 (height)

10. Possible solution (Fixed global geometries as hypotheses) Task much easier under a fixed hypothesis of global 3D geometries Ground o1o1 o2o2 okok Gravity × × × × ××

11. Task much easier under a fixed hypothesis of global 3D geometries Possible solution (Fixed global geometries as hypotheses) o1o1 o2o2 okok ω1ω1 ω2ω2 ω3ω3 How to generate global 3D geometry hypotheses?

12. Possible solution (Hypotheses by exhaustive search) Exhaustive search over the quantized space of global 3D geometries [2] – Computational complexity tends to limit search space [2] S. Bao et al. Toward coherent object detection and scene layout understanding. IVC, 2011

13. Possible solution (Hypotheses by Hough voting) Each local entity casts vote to the Hough voting space of the global 3D geometries and peaks are selected [3] – False detections could corrupt the votes – Would applying EM help? Not likely, if false detections overwhelm [3] M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

14. Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

15. Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

16. Our solution We take a RANSAC-like approach: Randomly mix the contributions of local entities – Compared to averaging over all local entities: More robust against outliers – Compared to directly using estimates from each single local entity: More robust against noise L1L1 L2L2 L3L3 L5L5 L4L4 L7L7 L6L6

22. 05101520253035404550 1.6 1.8 2 2.2 2.4 2.6 2.8 3 Number of random mixtures Minimum hypothesis error Gravity Direction Individual Mixture Average

23. 05101520253035404550 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 Number of random mixtures Minimum hypothesis error Ground Plane Orientation Individual Mixture Average

24. √ √ √ √ X Local 3D context Global 3D context

25. 3D geometric context ground plane orientation valid invalid (#1) ground plane #1: Common ground (global)

26. 3D geometric context #2: Gravity direction (global) (inverse) gravity ground plane orientation invalid (#2) ground plane

27. 3D geometric context #3: Depth ordering (local) (inverse) gravity ground plane orientation incompatible (#3) ground plane

28. 3D geometric context #4: Space occupancy (local) (inverse) gravity ground plane orientation incompatible (#4) ground plane

29. 2 3 45 6 1

30. 2 3 45 6 1 Global geometric compatibility for an object: Orientation: Given a global 3D geometry hypothesis

31. 2 3 45 6 1 Global geometric compatibility for an object: Orientation: Height: Given a global 3D geometry hypothesis

32. 2 3 45 6 1 Global geometric compatibility for a surface: Orientation: local estimates vs. or Location: horizontal surface region vs. ground horizon Given a global 3D geometry hypothesis

33. 2 3 45 6 1 Local geometric compatibility for two objects: Depth ordering: Space occupancy: Given a global 3D geometry hypothesis

34. 2 3 45 6 1 Objective function of the CRF: Given a global 3D geometry hypothesis

35. √ √ √ √ X Local 3D context Global 3D context Best hypothesis

36. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

37. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

38. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

39. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

40. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

41. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

42. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

43. 3D reasoning agrees with raw detector 3D reasoning recovers detection rejected by raw detector 3D reasoning rejects detection accepted by raw detector

44. 00.511.522.533.544.55 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 False Positive per Image True Positive Rate Deformable Part Model Detector Baseline Hoiem Ours 3D geometric reasoning improves object detection performance D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

45. 00.20.40.60.811.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 False Positive per Image True Positive Rate Dalal-Triggs Detector Baseline Hoiem Ours 3D geometric reasoning improves object detection performance D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

46. Improvement in AP over baseline detector Ours 10.4% Hoiem 4.8% Sun 5.1% M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008 3D geometric reasoning improves object detection performance

47. Horizon estimation median error Ours 2.05⁰ Hoiem 3.15⁰ Sun 2.41⁰ M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010 D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

48. √ √ √ √ X Local 3D context Global 3D context Best hypothesis

49. Contributions of different geometric context 00.511.522.533.544.55 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 False Positive per Image True Positive Rate Detection ROC Curve Det Det+IdvlGeo Det+PairGeo Det+FullGeo

50. Benefit is mutual Error in gravity direction Error in ground orientation Vanishing points alone 2.62⁰4.85⁰ Whole system 2.05⁰2.21⁰

51. Extensions – Improved depth ordering constraint – Local geometric constraints involving vertical surfaces – Multiple supporting planes – Using more prior knowledge of objects – Utilizing semantic categories of surface regions

53. closer object farther object closer object farther object occlusion mask of the farther object intersection region of the two object masks √ X Fully cover?

54. Occlusion: bottleneck in our system – Missed detection – Erroneous estimation of local properties – Less effective depth ordering constraint

55. Generalized Hough voting: better at handle occlusions K. Rematas et al. CORP 2011 B. Leibe et al. IJCV 2008

57. Occlusion-and-geometry-aware Hough voting

58. √ √ √ √ X Local 3D context Global 3D context Best hypothesis

59. So far we have treated the entire region labeled as "vertical" as a whole

60. Decompose vertical region into surface segments Occlusion boundary recovery (Hoiem et al. IJCV’11) Vanishing line sweeping (Lee et al. CVPR’09)

62. ground plane inverse gravity √ vertical surface candidate 1 vertical surface candidate 2

63. ground plane vertical surface candidate 1 inverse gravity vertical surface candidate 2 X

65. ground plane vertical surface candidate inverse gravity object candidate √

66. ground plane vertical surface candidate inverse gravity X

67. Given object layout, erect surfaces one by one “Interpretation by synthesis” (Gupta et al. ECCV’10)

69. supporting plane 1

70. supporting plane 2

71. ground plane

72. w l β

74. Spring 2013 (ICCV’13 submission) – Improved depth ordering constraint – Using more prior knowledge of objects – Multiple supporting planes Fall 2013 (CVPR’14 submission) – Local geometric constraints involving vertical surfaces – Utilizing semantic categories of surface regions During Spring Semester of 2014 – Thesis writing

75. Expected Contributions Systematically model the relationships among global and local geometric variables Develop a RANSAC-CRF scheme to handle non-linear, non-deterministic, and possibly invalid relationships Occlusion-and-geometry-aware object detection for finer depth order reasoning Joint reasoning among global geometries, surface segments, and objects

76. Thank you!