1. Three Dimensional Model Construction for Visualization Avideh Zakhor Video and Image Processing Lab University of California at Berkeley avz@eecs.berkeley.edu

2. Outline zGoals and objectives zPrevious work by PI zDirections for future work

3. Goals and Objectives zDevelop a framework for fast, automatic and accurate 3D model construction for objects, scenes, rooms, buildings (interior and exterior), urban areas, and cities. zModels must be easy to compute, compact to represent and suitable for high quality view synthesis and visualization zApplications: Virtual or augmented reality fly- throughs.

4. Previous Work on Scene Modeling zFull/Assisted 3-D Modeling Kanade et al.; Koch et al.; Becker & Bove; Debevec et al.; Faugeras et al.; Malik & Yu. zMosaics and Panoramas Szeliski & Kang; McMillan & Bishop; Shum & Szeliski zLayered/LDI Representations Wang & Adelson; Sawhney & Ayer; Weiss; Baker et al. zView Interpolation/IBR/Light Fields Chen & Williams; Chang & Zakhor; Laveau & Faugeras; Seitz & Dyer; Levoy & Hanrahan

5. Previous Work on Building Models zNevatia (USC): multi-sensor integration zTeller (MIT): spherical mosaics on a wheelchair sized rover, known 6DOF zVan Gool (Belgium): roof detection from aerial photographs zPeter Allen (Columbia): images and laser range finders; view/sensor planning. zFaugeras (INRIA)

6. Previous Work on City Modeling zPlanet 9: yCombines ground photographs with existing city maps manually. zUCLA Urban Simulation Team: yUses mutligen to create models from aerial photographs, together with ground video for texture mapping. zBath and London models by Univ. of Bath. yCombines aerial photgraphs with existing maps. zAll approaches are slow and labor intensive.

7. Work at VIP lab at UCB Scene modeling and reconstruction.

8. Multi-Valued Representation: MVR zLevel k has k occluding surfaces zForm multivalued array of depth and intensity

9. Observations

10. Imaging geometry (1) zPlanar translation

11. Imaging Geometry (2) zCircular/orbital motion

12. Dense Depth Estimation zEstimate camera motion zCompute depth maps to build MVRs y Low-contrast regions problematic for dense depth estimation. y Enforce spatial coherence to achieve realistic, high quality visualization.

13. Block Diagram for Dense Depth Estimation zPlanar approximation of depth for low contrast regions.

14. Oroginal Sequences “Mug” sequence (13 frames) “Teabox” sequence (102 frames)

15. Low-Contrast Regions Mug sequenceTea-box sequence zComplete tracking

16. Multiframe Depth Estimation Apply iterative estimation algorithm to enforce piecewise smoothness, without smoothing over depth discontinuities.

17. Multiframe Depth Estimation Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing Mug Tea-box

18. Multivalued Representation zProject depths to reference coordinates

19. Results (1) zMug sequence Multivalued representation for frame 4 (Level 0)

20. Results zMug sequence Multivalued representation for frame 4 (Level 1)

21. Results zMug sequence Multivalued representation for frame 4 (Combining Levels 0 and 1)

22. Results Reconstructed sequenceArbitrary flythrough zMug sequence

23. Results (2) Multivalued representation for frame 22 (Intensity, Level 0) zTeabox sequence

24. Results Multivalued representation for frame 22 (Depth, Level 0) zTeabox sequence

25. Results Multivalued representation for frame 22 (Intensity, Level 1) zTeabox sequence

26. Results Multivalued representation for frame 22 (Depth, Level 1) zTeabox sequence

27. Results Multivalued representation for frame 22 (Intensity, combining Levels 0 and 1) zTeabox sequence

28. Results Multivalued representation for frame 22 (Depth, combining Levels 0 and 1) zTeabox sequence

29. Results Multivalued representation for frame 86 (Intensity, Level 0) zTeabox sequence

30. Results Multivalued representation for frame 86 (Depth, Level 0) zTeabox sequence

31. Results Multivalued representation for frame 86 (Intensity, Level 1) zTeabox sequence

32. Results Multivalued representation for frame 86 (Depth, Level 1) zTeabox sequence

33. Results Multivalued representation for frame 86 (Intensity, combining Levels 0 and 1) zTeabox sequence

34. Results Multivalued representation for frame 86 (Depth, combining Levels 0 and 1) zTeabox sequence

35. Multiple MVRs zPerform view interpolation w/many MVRs

36. Results: multiple MVRs Reconstructed sequence from MVR86 Reconstruct sequence from MVR22 zTeabox sequence

37. Results: Multiple MVRs Reconstructed sequenceArbitrary flyaround

38. Extensions zComplex scenes with many “levels” are difficult to model with MVR; e.g. trees, leaves, etc zDifficult to ensure realistic visualization from all angles; Need to plan capture process carefully. zTradeoff between CG polygon modeling and IBR; yUse both in real visualization databases. yBuild polygon models from MVR.

39. Issues for model construction zChoice of geometry for obtaining data zChoice of imaging technology. zChoice of representation. zChoice of models. zDealing with time varying scenes.

40. Extensions: zSo far, addressed “outside in” problem: yCamera looked inward to “scan” the object. zFuture work will focus on the “Inside out” problem: yModeling a room, office. yModeling exterior or interior of a building yModeling an urban environment e.g. a city

41. Strategy zUse: yRange sensors, position sensors (GPS), Gyros(orientation), omni camera, video. yExisting datasets: 3D CAD models, digital elevation maps (DEM), DTED, city maps, architectural drawings: apriori information

42. Modeling interior of buildings z Leverage existing work in the computer graphics group at UCB: y3D model of Soda hall available from the “soda walkthrough” project. y3D model built out of architectural drawings yUse additional video, and laser range finder input to xEnhance the details of the 3D model: furniture, etc xAdd texture maps for photo-realistic walk- throughs.

43. City Modeling zDevelop a framework for modeling parts of city of San Francisco: yUse aerial photograph as provided by Space Imaging Corp; resolution 1 ft. yUse digitized city maps yUse ground data collection vehicle to collect range and intensity video from a panoramic camera, annotated with 6 DOF parameters. yDerive data fusion algorithms to process the above in speedy, automated and accurate fashion.

44. Requirements zAutomation (little or no interaction needed from human operators) zSpeed: must scale with large areas and large data sets. zAccuracy zRobustness to location of data collection. zEase of data collection. zRepresentation suitable to hierarchical visualization databases.

45. Relationship to others zUSC: accurate tracking and registration algorithms needed for model construction. zSyracuse: uncertainty processing, and data fusion for model construction. zG. Tech: How to combine CG polygonal model building with IBR models in vis. database? How can vis. databases deal with photo-realistic rendering?

46. Conclusions zFast, accurate and automatic model construction is essential to mobile augmented reality systems. zOur goal is to provide photo-realistic rendering of objects, scenes, buildings, and cities, to enable, visualization, navigation and interaction.