1. CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai

2. Image-based modeling Estimating 3D structure Estimating motion, e.g., camera motion Estimating lighting Estimating surface model

3. Traditional modeling and rendering User input Texture map survey data Geometry Reflectance Light source Camera model Images modeling rendering For photorealism: - Modeling is hard - Rendering is slow

4. Can we model and render this? What do we want to do for this model?

5. Image based modeling and rendering Images user input range scans ModelImages Image-based modeling Image-based rendering

6. Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

7. Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

8. Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

9. Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape How can we estimate camera parameters? knowncameraviewpoints

10. How can we estimate the camera parameters?

11. Camera calibration Augmented pin-hole camera - focal point, orientation - focal length, aspect ratio, center, lens distortion Known 3D Classical calibration - 3D 2D - correspondence Camera calibration online resources

12. Camera and calibration target

13. Classical camera calibration Known 3D coordinates and 2D coordinates - known 3D points on calibration targets - find corresponding 2D points in image using feature detection algorithm

14. Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords

15. Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords Intrinsic camera parameters (5 parameters) extrinsic camera parameters (6 parameters)

16. Camera matrix Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into a camera matrix M = K [R | t ] (put 1 in lower r.h. corner for 11 d.o.f.)

17. Camera matrix calibration Directly estimate 11 unknowns in the M matrix using known 3D points (X i,Y i,Z i ) and measured feature positions (u i,v i )

18. Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How?

19. Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How? Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

20. Nonlinear camera calibration Perspective projection:

21. Nonlinear camera calibration Perspective projection: KRTP

22. Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

23. Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

24. Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem:

25. Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem: Solve the optimization using nonlinear optimization techniques: - Gauss-newton - Levenberg-MarquardtLevenberg-Marquardt

26. Nonlinear approach Advantages: can solve for more than one camera pose at a time fewer degrees of freedom than linear approach Standard technique in photogrammetry, computer vision, computer graphics - [Tsai 87] also estimates lens distortions (freeware @ CMU) http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html Disadvantages: more complex update rules need a good initialization (recover K [R | t] from M)

27. Camera Calibration Public calibration toolbox - http://research.microsoft.com/en-us/um/people/zhang/Calib/http://research.microsoft.com/en-us/um/people/zhang/Calib/ - http://www.vision.caltech.edu/bouguetj/calib_doc/http://www.vision.caltech.edu/bouguetj/calib_doc/

28. How can we estimate the camera parameters?

29. Application: camera calibration for sports video [Farin et. Al] imagesCourt model

30. Stereo matching Given two or more images of the same scene or object as well as their camera parameters, how to compute a representation of its shape? What are some possible representations for shapes? depth maps volumetric models 3D surface models planar (or offset) layers

31. Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

32. Stereo matching 11.1, 11.2,.11.3,11.5 in Sezliski book D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7- 42, April-June 2002.A taxonomy and evaluation of dense two-frame stereo correspondence algorithms Readings

33. Stereo scene point optical center image plane

34. Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires >calibration >point correspondence

35. Stereo correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html epipolar line epipolar plane

36. Stereo image rectification

37. reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation two homographies (3x3 transform), one for each input image reprojection  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Computing Rectifying Homographies for Stereo Vision

38. Rectification Original image pairs Rectified image pairs

39. Stereo matching algorithms Match Pixels in Conjugate Epipolar Lines Assume brightness constancy This is a tough problem Numerous approaches >A good survey and evaluation: http://www.middlebury.edu/stereo/ http://www.middlebury.edu/stereo/

40. Your basic stereo algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum matching cost Improvement: match windows This should look familiar.. (normalized cross correlation or SSD) Can use Lukas-Kanade or discrete search (latter more common)

41. Window size Smaller window + - Larger window + - W = 3W = 20 Effect of window size

42. More constraints? We can enforce more constraints to reduce matching ambiguity - smoothness constraints: computed disparity at a pixel should be consistent with neighbors in a surrounding window. - uniqueness constraints: the matching needs to be bijective - ordering constraints: e.g., computed disparity at a pixel should not be larger than the disparity of its right neighbor pixel by more than one pixel.

43. Stereo results Ground truthScene Data from University of Tsukuba Similar results on other images without ground truth

44. Results with window search Window-based matching (best window size) Ground truth

45. Better methods exist... A better method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September 1999. Ground truth

46. More recent development High-Quality Single-Shot Capture of Facial Geometry [siggraph 2010, project website]project website - capture high-fidelity facial geometry from multiple cameras - pairwise stereo reconstruction between neighboring cameras - hallucinate facial details

47. More recent development High Resolution Passive Facial Performance Capture [siggraph 2010, project website]project website - capture dynamic facial geometry from multiple video cameras - spatial stereo reconstruction for every frame - building temporal correspondences across the entire sequence

48. Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth

49. Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

50. Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

51. Active stereo with structured light Project “structured” light patterns onto the object simplifies the correspondence problem camera 2 camera 1 projector camera 1 projector Li Zhang’s one-shot stereo

52. Active stereo with structured light

53. Laser scanning Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/

54. Laser scanned models The Digital Michelangelo Project, Levoy et al.

55. Laser scanned models The Digital Michelangelo Project, Levoy et al.

56. RGBD Sensors 2008 2010 2013 Lower size/cost with better accuracy

57. Kinect Sensor