1. David Luebke11-17-98 Modeling and Rendering Architecture from Photographs A hybrid geometry- and image-based approach Debevec, Taylor, and Malik SIGGRAPH 96 Presented by David Luebke

2. David Luebke11-17-98 Overview The Problem and the Idea The Problem and the Idea Background Background Model Representation and Reconstruction Model Representation and Reconstruction View-dependent Texture Mapping View-dependent Texture Mapping Model-based Stereo Model-based Stereo Conclusion and Discussion Conclusion and Discussion

3. David Luebke11-17-98 The Problem Architectural walkthroughs and flybys are an important application Architectural walkthroughs and flybys are an important application Creating detailed models is hard Creating detailed models is hard –Start with blueprints (if they exist…) –Survey an existing building Resulting systems don’t look great Resulting systems don’t look great –Hard to get all the details –Hard to get realistic exteriors

4. David Luebke11-17-98 The Idea Wanted: a system to generate realistic architectural scenes Wanted: a system to generate realistic architectural scenes Idea: Model and render from photos! Idea: Model and render from photos! –Take a few widely spaced photographs –Build simple underlying model of scene –Use correspondences between photos to adjust scene parameters –Paste photos back onto simple geometry of scene for realistic façade

5. David Luebke11-17-98 Background Computer vision: recover 3D geometry from 2D images Computer vision: recover 3D geometry from 2D images Debevec uses some CV concepts: Debevec uses some CV concepts: –Camera calibration: simplify problem by finding exact pixel  ray mappings –Structure from motion and stereo correspondence: triangulating for depth –Image-based rendering: given image & depth map, re-render from other views

6. David Luebke11-17-98 Photogrammetric Modeling Extracting 3D surfaces from multiple images is hard Extracting 3D surfaces from multiple images is hard Constrain the problem: Constrain the problem: –User builds a simple notional model using blocks: primitive solid shapes Example: boxes, wedges, prisms, frusta Example: boxes, wedges, prisms, frusta –User marks correspondences between images and model –System fits model to images

7. David Luebke11-17-98 Photogrammetric Modeling Now system need only solve parameters of blocks! Now system need only solve parameters of blocks! –Height, width, translation, rotation, etc.

8. David Luebke11-17-98 Photogrammetric Modeling Even better: build in architectural constraints! Even better: build in architectural constraints! –Roof prism lies flush on building block –Stacked tower blocks share center axis

9. David Luebke11-17-98 Photogrammetric Modeling Knowns: image  block edge correspondences Knowns: image  block edge correspondences Unknowns: block parameters, camera position/orientation Unknowns: block parameters, camera position/orientation Constraints reduce # unknowns Constraints reduce # unknowns Generally, # correspondences must equal # unknowns for reconstruction Generally, # correspondences must equal # unknowns for reconstruction

10. David Luebke11-17-98 Photogrammetric Modeling Represent block parameters as instances of shared variables Represent block parameters as instances of shared variables Lots of math… Lots of math… –Tweaking model edges to correspond to recovered edges –Computing an initial estimate

11. David Luebke11-17-98 Photogrammetric Modeling Results: Results:

12. David Luebke11-17-98 Photogrammetric Modeling Results: Results:

13. David Luebke11-17-98

14. Photogrammetric Modeling

15. David Luebke11-17-98 View-Dependent Texture Mapping Given the model, treat each camera position as a “slide projector” Given the model, treat each camera position as a “slide projector” Some images overlap! Some images overlap! –Idea: pick image taken from viewpoint closest to desired rendering viewpoint –Better: use weighted average (Fig 12)

16. David Luebke11-17-98 View-Dependent Texture Mapping Best: Do view-dependent texture mapping on per-pixel basis Best: Do view-dependent texture mapping on per-pixel basis Okay: Do it on a per-face basis Okay: Do it on a per-face basis –Subdivide large faces –Use texture hardware!

17. David Luebke11-17-98 View-Dependent Texture Mapping

18. David Luebke11-17-98 Model-Based Stereo Problem: fine architectural details still not captured Problem: fine architectural details still not captured –recessed windows, friezes, cornices Stereo depth extraction can help! Stereo depth extraction can help! –Problem: when images are taken from distant viewpoints, corresponding pixel neighborhoods can look very different

19. David Luebke11-17-98 Model-Based Stereo Key observation: Key observation: –Even though two images of the same scene may look very different, they look similar after being projected onto the approximate model. –Idea: Warp offset image by projecting onto the approximate model and re- rendering –Use McMillian warp to render image- with-depth from novel viewpoints

20. David Luebke11-17-98 Conclusion and Discussion Results speak for themselves Results speak for themselves What problems do you see? What problems do you see?