CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai

Image-based modeling Estimating 3D structure Estimating motion, e.g., camera motion Estimating lighting Estimating surface model

Traditional modeling and rendering User input Texture map survey data Geometry Reflectance Light source Camera model Images modeling rendering For photorealism: - Modeling is hard - Rendering is slow

Can we model and render this? What do we want to do for this model?

Image based modeling and rendering Images user input range scans ModelImages Image-based modeling Image-based rendering

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape How can we estimate camera parameters? knowncameraviewpoints

How can we estimate the camera parameters?

Camera calibration Augmented pin-hole camera - focal point, orientation - focal length, aspect ratio, center, lens distortion Known 3D Classical calibration - 3D 2D - correspondence Camera calibration online resources

Camera and calibration target

Classical camera calibration Known 3D coordinates and 2D coordinates - known 3D points on calibration targets - find corresponding 2D points in image using feature detection algorithm

Camera parameters u0u0 v0v s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords

Camera parameters u0u0 v0v s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords Intrinsic camera parameters (5 parameters) extrinsic camera parameters (6 parameters)

Camera matrix Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into a camera matrix M = K [R | t ] (put 1 in lower r.h. corner for 11 d.o.f.)

Camera matrix calibration Directly estimate 11 unknowns in the M matrix using known 3D points (X i,Y i,Z i ) and measured feature positions (u i,v i )

Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How?

Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How? Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

Nonlinear camera calibration Perspective projection:

Nonlinear camera calibration Perspective projection: KRTP

Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem:

Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem: Solve the optimization using nonlinear optimization techniques: - Gauss-newton - Levenberg-MarquardtLevenberg-Marquardt

Nonlinear approach Advantages: can solve for more than one camera pose at a time fewer degrees of freedom than linear approach Standard technique in photogrammetry, computer vision, computer graphics - [Tsai 87] also estimates lens distortions CMU) Disadvantages: more complex update rules need a good initialization (recover K [R | t] from M)

Camera Calibration Public calibration toolbox

How can we estimate the camera parameters?

Application: camera calibration for sports video [Farin et. Al] imagesCourt model

Stereo matching Given two or more images of the same scene or object as well as their camera parameters, how to compute a representation of its shape? What are some possible representations for shapes? depth maps volumetric models 3D surface models planar (or offset) layers

Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

Stereo matching 11.1, 11.2,.11.3,11.5 in Sezliski book D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7- 42, April-June 2002.A taxonomy and evaluation of dense two-frame stereo correspondence algorithms Readings

Stereo scene point optical center image plane

Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires >calibration >point correspondence

Stereo correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: epipolar line epipolar plane

Stereo image rectification

reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation two homographies (3x3 transform), one for each input image reprojection  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Computing Rectifying Homographies for Stereo Vision

Rectification Original image pairs Rectified image pairs

Stereo matching algorithms Match Pixels in Conjugate Epipolar Lines Assume brightness constancy This is a tough problem Numerous approaches >A good survey and evaluation:

Your basic stereo algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum matching cost Improvement: match windows This should look familiar.. (normalized cross correlation or SSD) Can use Lukas-Kanade or discrete search (latter more common)

Window size Smaller window + - Larger window + - W = 3W = 20 Effect of window size

More constraints? We can enforce more constraints to reduce matching ambiguity - smoothness constraints: computed disparity at a pixel should be consistent with neighbors in a surrounding window. - uniqueness constraints: the matching needs to be bijective - ordering constraints: e.g., computed disparity at a pixel should not be larger than the disparity of its right neighbor pixel by more than one pixel.

Stereo results Ground truthScene Data from University of Tsukuba Similar results on other images without ground truth

Results with window search Window-based matching (best window size) Ground truth

Better methods exist... A better method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September Ground truth

More recent development High-Quality Single-Shot Capture of Facial Geometry [siggraph 2010, project website]project website - capture high-fidelity facial geometry from multiple cameras - pairwise stereo reconstruction between neighboring cameras - hallucinate facial details

More recent development High Resolution Passive Facial Performance Capture [siggraph 2010, project website]project website - capture dynamic facial geometry from multiple video cameras - spatial stereo reconstruction for every frame - building temporal correspondences across the entire sequence

Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth

Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

Active stereo with structured light Project “structured” light patterns onto the object simplifies the correspondence problem camera 2 camera 1 projector camera 1 projector Li Zhang’s one-shot stereo

Active stereo with structured light

Laser scanning Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Digital Michelangelo Project

Laser scanned models The Digital Michelangelo Project, Levoy et al.

Laser scanned models The Digital Michelangelo Project, Levoy et al.

RGBD Sensors Lower size/cost with better accuracy

Kinect Sensor