David Luebke Modeling and Rendering Architecture from Photographs A hybrid geometry- and image-based approach Debevec, Taylor, and Malik SIGGRAPH 96 Presented by David Luebke

David Luebke Overview The Problem and the Idea The Problem and the Idea Background Background Model Representation and Reconstruction Model Representation and Reconstruction View-dependent Texture Mapping View-dependent Texture Mapping Model-based Stereo Model-based Stereo Conclusion and Discussion Conclusion and Discussion

David Luebke The Problem Architectural walkthroughs and flybys are an important application Architectural walkthroughs and flybys are an important application Creating detailed models is hard Creating detailed models is hard –Start with blueprints (if they exist…) –Survey an existing building Resulting systems don’t look great Resulting systems don’t look great –Hard to get all the details –Hard to get realistic exteriors

David Luebke The Idea Wanted: a system to generate realistic architectural scenes Wanted: a system to generate realistic architectural scenes Idea: Model and render from photos! Idea: Model and render from photos! –Take a few widely spaced photographs –Build simple underlying model of scene –Use correspondences between photos to adjust scene parameters –Paste photos back onto simple geometry of scene for realistic façade

David Luebke Background Computer vision: recover 3D geometry from 2D images Computer vision: recover 3D geometry from 2D images Debevec uses some CV concepts: Debevec uses some CV concepts: –Camera calibration: simplify problem by finding exact pixel  ray mappings –Structure from motion and stereo correspondence: triangulating for depth –Image-based rendering: given image & depth map, re-render from other views

David Luebke Photogrammetric Modeling Extracting 3D surfaces from multiple images is hard Extracting 3D surfaces from multiple images is hard Constrain the problem: Constrain the problem: –User builds a simple notional model using blocks: primitive solid shapes Example: boxes, wedges, prisms, frusta Example: boxes, wedges, prisms, frusta –User marks correspondences between images and model –System fits model to images

David Luebke Photogrammetric Modeling Now system need only solve parameters of blocks! Now system need only solve parameters of blocks! –Height, width, translation, rotation, etc.

David Luebke Photogrammetric Modeling Even better: build in architectural constraints! Even better: build in architectural constraints! –Roof prism lies flush on building block –Stacked tower blocks share center axis

David Luebke Photogrammetric Modeling Knowns: image  block edge correspondences Knowns: image  block edge correspondences Unknowns: block parameters, camera position/orientation Unknowns: block parameters, camera position/orientation Constraints reduce # unknowns Constraints reduce # unknowns Generally, # correspondences must equal # unknowns for reconstruction Generally, # correspondences must equal # unknowns for reconstruction

David Luebke Photogrammetric Modeling Represent block parameters as instances of shared variables Represent block parameters as instances of shared variables Lots of math… Lots of math… –Tweaking model edges to correspond to recovered edges –Computing an initial estimate

David Luebke Photogrammetric Modeling Results: Results:

David Luebke Photogrammetric Modeling Results: Results:

David Luebke

Photogrammetric Modeling

David Luebke View-Dependent Texture Mapping Given the model, treat each camera position as a “slide projector” Given the model, treat each camera position as a “slide projector” Some images overlap! Some images overlap! –Idea: pick image taken from viewpoint closest to desired rendering viewpoint –Better: use weighted average (Fig 12)

David Luebke View-Dependent Texture Mapping Best: Do view-dependent texture mapping on per-pixel basis Best: Do view-dependent texture mapping on per-pixel basis Okay: Do it on a per-face basis Okay: Do it on a per-face basis –Subdivide large faces –Use texture hardware!

David Luebke View-Dependent Texture Mapping

David Luebke Model-Based Stereo Problem: fine architectural details still not captured Problem: fine architectural details still not captured –recessed windows, friezes, cornices Stereo depth extraction can help! Stereo depth extraction can help! –Problem: when images are taken from distant viewpoints, corresponding pixel neighborhoods can look very different

David Luebke Model-Based Stereo Key observation: Key observation: –Even though two images of the same scene may look very different, they look similar after being projected onto the approximate model. –Idea: Warp offset image by projecting onto the approximate model and re- rendering –Use McMillian warp to render image- with-depth from novel viewpoints

David Luebke Conclusion and Discussion Results speak for themselves Results speak for themselves What problems do you see? What problems do you see?