1. Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) An overview of multi-view stereo and other topics in 3D modeling from 2D images Currently: sabbatical at TUM (Humboldt-fellow)

2. Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

3. Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

4. Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo Other approaches and related subjects: Other 3D modeling approaches Camera and object tracking Omnidirectional vision Sensor design... Numerical optimization Current research directions

5. Before about 1995 [Mohr et al.] 3D modeling from multiple images: image matching often done by hand main focus: given the matching, how to estimate camera positions, camera calibration, 3D point positions? 1990’s: golden age of multi-view geometry Exceptions, concerning manual matching: binocular stereo, aerial imagery close-range photogrammetry, mo-cap 3D modeling from silhouettes

6. 1996 [Beardsley et al.] The first fully automatic approaches automatic matching systematic use of robust statistics going towards surface models (here, meshes)

7. 1998 [Fitzgibbon et al.] Handling longer image sequences better numerical optimization methods system integration better data: image quality, calibration

8. Around 1998 Matchmoving applications (tracking of camera motion) in AR [Microsoft]

9. Around 1998 Matchmoving applications (tracking of camera motion) in AR [2d3] VIDEO…

10. At about the same time Polyhedral models from many georeferenced images [MIT City Scanning Project]

11. From 1998 onwards Going from “hacks” to mathematically founded surface representations and evolutions: level sets (now, increased usage of meshes) [Keriven et al.]

12. About 2003 [Strecha et al.] probabilistic occlusion handling Towards truly dense 3D models: point-based surface representations

13. About 2007 [Furukawa et al.] patch-based surface representation Current state of the art

14. About 2007 [Furukawa et al.] patch-based surface representation Current state of the art

15. Multi-View Stereo [Gargallo et al.] Optimization:

16. Quick Summary surface representation: point-based, patch-based, mesh, implicit surface, voxel sets,... Ingredients of multi-view stereo: quality measure: “photo-consistency”, invariance to lighting differences,... prior assumptions (regularization): smoothness, piecewise planarity, class-specific priors (e.g. faces) optimization approach, resolution strategy [Vetter et al.]

17. Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo... Numerical optimization Current research directions Other approaches and related subjects: Other 3D modeling approaches Camera and object tracking Omnidirectional vision Sensor design

18. [Criminisi et al.] Other approaches and subjects Interactive 3D modeling based on user annotations single-image 3D modeling

19. Other approaches and subjects Interactive 3D modeling based on user annotations multi-image 3D modeling [Debevec et al.]

20. Other approaches and subjects Exploitation of other depth cues than stereo shading focus / defocus

21. Active approaches using controlled illumination: Other approaches and subjects Photometric stereo Flash photography Structured light...

22. 3D Modeling from silhouettes (“visual hull”) [Hernandez et al.] Other approaches and subjects Possible in real-time

23. Other approaches and subjects VIDEO…

24. Other approaches and subjects Real-time camera tracking: Model-based object tracking [Fua et al.]

25. Other approaches and subjects Real-time camera tracking: SLAM (simultaneous localization and mapping) Model-based object tracking [Lhuillier et al.] VIDEO…

26. Other approaches and subjects Omnidirectional vision: wide angle cameras surveillance navigation panoramic imaging

27. Other approaches and subjects Filters for HDR mosaics Mirror design [Nayar et al.] Some works on sensor design: Lens arrays

28. Other approaches and subjects Some works on sensor design: Filters for HDR mosaics Lens arrays, e.g. for computational photography [Georgiev et al.]

29. Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo Global optimization methods for geometry computations Use of community image collections Modeling surface appearance Modeling deformable / articulated objects... Numerical optimization Current research directions: Other approaches and related subjects

30. [Snavely et al.] Current research directions PhotoSynth / PhotoTourism Exploration of community image collections:

31. [Furukawa et al.] Current research directions Interior of buildings, cities Detailed 3D modeling of large scenes VIDEO…

32. Current research directions Shiny objects Modeling surface appearance: [Lensch et al.]

33. Current research directions Shiny objects Modeling surface appearance: [Birkbeck et al.]

34. Current research directions “The Light Stage” Modeling surface appearance: [Debevec et al.]

35. Current research directions Tanslucid, refractive, reflective surfaces, smoke, fire,... Modeling surface appearance:

36. Current research directions Modeling complex geometries: [Quan et al.] [Paris et al.]

37. Current research directions Modeling deformable / articulated objects: [Fua et al.]

38. Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) An overview of multi-view stereo and other topics in 3D modeling from 2D images Currently: sabbatical at TUM (Humboldt-fellow)