CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai

Image-based modeling Estimating 3D structure Estimating motion, e.g., camera motion Estimating lighting Estimating surface model

Traditional modeling and rendering User input Texture map survey data Geometry Reflectance Light source Camera model Images modeling rendering For photorealism: - Modeling is hard - Rendering is slow

Can we model and render this? What do we want to do for this model?

Image based modeling and rendering Images user input range scans ModelImages Image-based modeling Image-based rendering

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape What are some possible applications? knowncameraviewpoints

3D modeling From one stereo pair to a 3D head model [Frederic Deverney, INRIA]Frederic Deverney

3D modeling The Digital Michelangelo Project, Levoy et al.

Optical mocap Vicon mocap system

Z-keying: mix live and synthetic Takeo Kanade, CMU (Stereo Machine)Stereo Machine

Virtualized Reality TM [Takeo Kanade et al., CMU] collect video from 50+ stream reconstruct 3D model sequences steerable version used for SuperBowl XXV “eye vision”eye vision

View interpolation inputdepth image novel view [Szeliski & Kang ‘95]

View morphing Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96]

Image warping

Video view interpolation

Performance Interface Microsoft Natal projectNatal project

Additional applications? Real-time people tracking (systems from Pt. Gray Research and SRI) “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93] Other ideas?

Stereo matching Given two or more images of the same scene or object, compute a representation of its shape What are some possible representations for shapes? depth maps volumetric models 3D surface models planar (or offset) layers

Outline Stereo matching - Traditional stereo - Multi-baseline stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

Stereo matching Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.A taxonomy and evaluation of dense two-frame stereo correspondence algorithms Visual-hull reconstruction Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp Matusik, Buehler, Raskar, McMillan, and Gortler, “Image-Based Visual Hulls”, Proc. SIGGRAPH 2000, pp Photo-hull reconstruction Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl. Journal of Computer Vision (IJCV), 1999, 35(2), pp Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of Computer Vision, 2000, 38(3), pp Papers

Stereo scene point optical center image plane

Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires >calibration >point correspondence

Camera calibration From world coordinate to image coordinate u0u0 v0v s y 0 sxsx au v 1 Perspective projection View transformation Viewport projection Camera parameters3D points 2D projections

Stereo correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: epipolar line epipolar plane

Stereo image rectification

reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation two homographies (3x3 transform), one for each input image reprojection  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Computing Rectifying Homographies for Stereo Vision

Rectification Original image pairs Rectified image pairs

Stereo matching algorithms Match Pixels in Conjugate Epipolar Lines Assume brightness constancy This is a tough problem Numerous approaches >A good survey and evaluation:

Your basic stereo algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum matching cost Improvement: match windows This should look familiar.. Can use Lukas-Kanade or discrete search (latter more common)

Window size Smaller window + - Larger window + - W = 3W = 20 Effect of window size

Stereo results Ground truthScene Data from University of Tsukuba Similar results on other images without ground truth

Results with window search Window-based matching (best window size) Ground truth

Better methods exist... State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September Ground truth

Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth

Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

Outline Stereo matching - Traditional stereo - Multi-baseline stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

Depth from disparity f xx’ baseline z CC’ X f input image (1 of 2) [Szeliski & Kang ‘95] disparity map 3D rendering

width of a pixel Choosing the stereo baseline What’s the optimal baseline? Too small: large depth error Too large: difficult search problem Large Baseline Small Baseline all of these points project to the same pair of pixels

The effect of baseline on depth estimation

1/z width of a pixel width of a pixel 1/z pixel matching score

Multi-baseline stereo Basic Approach Choose a reference view Use your favorite stereo algorithm BUT >replace two-view SSD with SSD over all baselines Limitations Must choose a reference view (bad) Visibility! CMU’s 3D Room Video

Outline Stereo matching - Traditional stereo - Multi-baseline stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

Active stereo with structured light Project “structured” light patterns onto the object simplifies the correspondence problem camera 2 camera 1 projector camera 1 projector Li Zhang’s one-shot stereo

Active stereo with structured light

Laser scanning Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Digital Michelangelo Project

Laser scanned models The Digital Michelangelo Project, Levoy et al.

Laser scanned models The Digital Michelangelo Project, Levoy et al.

Desktop scanner Convenient to use Good quality Relatively low-cost - next engine (about 2k)next engine