1. Perceptual and Sensory Augmented Computing Integrating Recognitoin and Reconstruction Integrating Recognition and Reconstruction for Cognitive Scene Interpretation Bastian Leibe, Nico Cornelis, Kurt Cornelis, Luc Van Gool Computer Vision Laboratory ETH Zurich Sicily Workshop, Syracusa, 22.09.2006 VISICS KU Leuven & CVPR’06 Video Proceedings DAGM’06

2. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 2 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Motivation Urban traffic scene analysis from a moving vehicle  Detect objects in the image  Localize them in 3D  Build up a metric scene model Applications e.g. in driver assistance systems

3. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 3 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Challenges Brightly-lit areasMotion blur Lense flaring + Intra-category variability, multiple viewpoints, partial occlusion,... Dark shadows

4. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 4 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Cognitive Loop with 3D Geometry Connect recognition and reconstruction Reconstruction pathway delivers scene geometry  Greatly improves recognition performance Recognition detects objects that disturb reconstruction  More accurate geometry estimate

5. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 5 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Outline Hardware setup Reconstruction pathway  Real-time Structure-from-Motion  Real-time dense reconstruction Recognition pathway  Local-feature based object detection  Incorporation of scene geometry  Temporal integration in world coordinate frame  Feedback to reconstruction Results and Conclusion

6. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 6 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Hardware Setup Stereo camera rig mounted on top of the vehicle Calibrated w.r.t. wheel base points Video streams captured at 25 fps, 360  288 resolution

7. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 7 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Real-Time Structure-from-Motion Basis: very fast feature matching  Simple features  Optimized for urban environment  Only computed on green channel of a single camera Rest: standard SfM pipeline [Cornelis et al., CVPR’06]

8. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 8 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Dense reconstruction on rectified images  Ruled surface assumption to speed-up dense reconstruction  Correlation measure: Sum of per-pixel SSDs along vertical lines  Line-sweep algorithm with ordering constraints (DP)  Fast computation on GPU Errors introduced by pixels not belonging to facades! Real-Time Dense Reconstruction [Cornelis et al., CVPR’06]

9. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 9 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Merge dense reconstructions using known camera poses. “Voted polygon carving” on 2D projection Real-Time Dense Reconstruction (2)

10. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 10 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Merge dense reconstructions using known camera poses. “Voted polygon carving” on 2D projection Surfaces registered on world map using GPS Real-Time Dense Reconstruction (2)

11. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 11 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Run-times  SfM + Bundle adjustment: 26-30 fps on CPU  Dense reconstruction: 26 fps on GPU Textured 3D Model Original3D Reconstruction

12. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 12 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Information Flow into Recognition For each frame, 3D reconstruction delivers  External camera calibration  Ground plane estimate  Used for improving recognition of the next frame.

13. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 13 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Appearance-Based Car Detection Bank of 5 single-view ISM detectors Each based on 3 local cues  Harris-Laplace, Hessian-Laplace, and DoG interest regions  Local Shape Context descriptors Semi-profile detectors additionally mirrored Not real-time yet… [Leibe, Mikolajczyk, Schiele,06]

14. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 14 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Implicit Shape Model - Representation Learn appearance codebook  Extract patches at interest points  Agglomerative clustering  codebook Learn spatial distributions  Match codebook to training images  Record matching positions on object training images (+reference segmentation) Appearance codebook … … … … … Spatial occurrence distributions x y s x y s x y s x y s

15. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 15 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Implicit Shape Model - Recognition Backprojected Hypotheses Interest Points Matched Codebook Entries Probabilistic Voting 3D Voting Space (continuous) x y s Backprojection of Maxima [Leibe & Schiele,04]

16. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 16 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Implicit Shape Model - Recognition Backprojected Hypotheses Interest Points Matched Codebook Entries Probabilistic Voting Segmentation 3D Voting Space (continuous) x y s Backprojection of Maxima p(figure) Probabilities [Leibe & Schiele,04]

17. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 17 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Likelihood of 3D hypothesis H given image I and 2D detections h : 2D recognition score  Expressed in terms of per-pixel p(figure) probabilities recognition score (2D)

18. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 18 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Likelihood of 3D hypothesis H given image I and 2D detections h : 3D prior  Distance prior (uniform range)  Size prior (Gaussian)  Significantly reduced search space 3D prior x s y Search corridor

19. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 19 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Likelihood of 3D hypothesis H given image I and 2D detections h : 2D/3D transfer  Two image-plane detections are consistent if they correspond to the same 3D object  Multi-viewpoint integration  Multi-camera integration 2D/3D transfer

20. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 20 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Detections Using Ground Plane Constraints left camera 1175 frames

21. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 21 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Quantitative Results Detection performance on first 600 frames  All cars annotated that were >50% visible  Ground plane constraint significantly improves precision  Performance: 0.2 fp/image at 50% recall

22. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 22 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Temporal Integration Temporal integration in world coordinate frame  Using external camera calibration from SfM.  Each detection transfers to a 3D observation H.  Find superset of 3D hypotheses.  Estimate orientation using cluster shape & detected viewpoints.  Select set of 3D hypotheses that best explain the observations.

23. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 23 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Hypothesis Selection for 3D Detections Quadratic Boolean Optimization Problem (from MDL) Individual scores (diagonal terms) Interaction costs (off-diagonal terms) temporal decay likelihood of membership to hypothesis penalty for physical overlap [Leonardis et al,95]

24. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 24 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Result of Temporal Integration

25. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 25 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Online 3D Car Location Estimates

26. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 26 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 3D Estimates After Convergence

27. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 27 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Feedback into 3D Reconstruction Feedback of detections & segmentation maps  Used to discard features on cars for SfM  Used to mask out cars in dense reconstruction  More accurate 3D estimates in the next frame.

28. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 28 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Another Application: 3D City Modeling Original3D Reconstruction Enhancing your driving experience…

29. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 29 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Conclusion System for traffic scene analysis integrating  Structure-from-Motion  Dense 3D Reconstruction  Object detection and localization in 2D and 3D  Temporal integration in world coordinate frame Cognitive Loop between 2D and 3D processing  Reconstruction delivers camera calibration, ground plane  3D context tremendously improves recognition performance  Car detection, segmentation makes 3D estimation more accurate System applied to challenging real-world task  Real-time 3D reconstruction (26-30 fps)  Accurate object detection & 3D pose estimation results

30. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 30 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Thank you very much for your attention! http://www.vision.ethz.ch/ http://www.esat.kuleuven.be/psi/visics/

31. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 31 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Merge dense reconstructions using known camera poses. “Voted polygon carving” on 2D projection Real-Time Dense Reconstruction (2)

32. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 32 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Real-Time Dense Reconstruction (3) Ruled surfaces are registered on world map using GPS. Run-times:  SfM + Bundle adjustment: 26-30 fps on CPU  Dense reconstruction: 26 fps on GPU

33. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 33 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool MDL Hypothesis Selection Savings of a hypothesis with  S area : #data points N belonging to h  S model : model cost  S error : estimate of error, according to Final form of equation Joint work with Ales Leonardis, UOL

34. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 34 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool MDL Hypothesis Selection (2) Savings of combined hypothesis Goal: Find combination that best explains the data  Quadratic Boolean Optimization problem [Leonardis et al,95]

35. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 35 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Conclusion System for cognitive scene analysis  Structure-from-Motion  Dense 3D Reconstruction  Object detection and localization  Temporal integration in world coordinate frame Cognitive Loop between 2D and 3D processing  Reconstruction delivers camera calibration + ground plane.  3D context tremendously improves recognition performance.  Car detection/segmentation makes 3D estimation more accurate. Further work  Add motion model for moving objects  Extend recognition to more categories  Optimize recognition run-time

36. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 36 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Information Flow into Recognition For each frame, 3D reconstruction delivers  External camera calibration  Ground plane estimate  Used for improving recognition of the next frame. In return, recognition feeds back  Object detections and top-down segmentation  Used to discard features on cars for SfM  Used to mask out cars in dense reconstruction  More accurate 3D estimates in the next frame.

37. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 37 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Dense reconstruction on rectified images  Sum of per-pixel SSDs along vertical lines as correlation measure  Ruled surface assumption to speed-up dense reconstruction  Fast computation on GPU Errors introduced by pixels not belonging to facades! Real-Time Dense Reconstruction

38. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 38 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Dense reconstruction on rectified images  Ruled surface assumption to speed-up dense reconstruction  Correlation measure: Sum of per-pixel SSDs along vertical lines  Plane-sweep algorithm with ordering constraints (DP)  Fast computation on GPU Real-Time Dense Reconstruction

39. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 39 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Merge dense reconstructions using known camera poses. “Voted polygon carving” on 2D projection Real-Time Dense Reconstruction (2)

40. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 40 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Merge dense reconstructions using known camera poses. “Voted polygon carving” on 2D projection Real-Time Dense Reconstruction (2)

41. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 41 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Outline Object categorization approach  Initial hypothesis generation  Category-specific figure-ground segmentation  Hypothesis verification using segmentation Extensions Discussion & Outlook  New promising directions

42. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 42 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Hypothesis generation Segmentation Segmentation: Probabilistic Formulation Segmentation information Influence on object hypothesis pixels p patches e codebook entries I

43. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 43 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Interpretation of p(figure) map  per-pixel confidence in object hypothesis  Use for hypothesis verification p(figure) p(ground) Segmentation p(figure) p(ground) Original image

44. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 44 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Formalization in MDL Framework Savings of a hypothesis Savings of hypothesis combination Goal: Find combination that best explains the image  Quadratic Boolean Optimization problem [Leonardis et al,95] (In practice often sufficient to compute greedy approximation)

45. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 45 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Relationship between 2D hypothesis h and 3D hypo H given image I : 2D recognition score  Expressed in terms of per-pixel p(figure) probabilities recognition score (2D)

46. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 46 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Relationship between 2D hypothesis h and 3D hypo H given image I : 3D prior  Distance prior (uniform range)  Size prior (Gaussian)  Significantly reduced search space 3D prior x s y Search corridor

47. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 47 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Relationship between 2D hypothesis h and 3D hypo H given image I : 2D/3D transfer  Two image-plane detections are consistent if they correspond to the same 3D object  Multi-viewpoint integration  Multi-camera integration 2D/3D transfer

48. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 48 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Knowledge Transfer Ground plane 3D reconstruction using Computer Vision Improved car detection using higher scene knowledge

49. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 49 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Knowledge Transfer Object Locations Improved reconstruction using object knowledge 3D location estimation from detections

50. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 50 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Textured 3D model

51. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 51 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Effect of the Ground Plane x s y x s y Search window

52. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 52 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool 2D/3D Interactions Relationship between 2D hypothesis h and 3D hypo H given image I : 2D/3D transfer  Two image-plane detections are consistent if they correspond to the same 3D object  Multi-viewpoint integration  Multi-camera integration 2D/3D transfer

53. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 53 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Ground Plane Use city model to reduce the amount of false positives

54. Perceptual and Sensory Augmented Computing Integrating Recognition and Reconstruction 54 B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool Cognitive Loop with 3D Geometry Cognitive Loop  “Bidirectional knowledge transfer involving a semantic level”