What Does the Scene Look Like From a Scene Point? Donald Tanguay August 7, 2002 M. Irani, T. Hassner, and P. Anandan ECCV 2002

Overview Categorization of novel view synthesis Outline of approach Planar parallax formulation Synthesizing the virtual view Practical simplification Results Assessment

Novel View Synthesis Breakdown into 3 categories: 3D reconstruction View transfer Sampling methods

3D Reconstruction Fully reconstruct scene, then render view Geometric error criteria do not translate well to errors in novel view Reconstruction and rendering occur in different coordinate systems Problems amplified with novel viewpoints significantly different from real cameras

“View Transfer” For example: 2 images, dense correspondence, and trifocal tensor Avoids reconstruction Errors in correspondence Synthesis uses forward warping step, which results in “hole-filling” at surface discontinuities Problems amplified by severe changes in viewpoint

Sampling Methods E.g., lightfield and lumigraph Avoid reconstruction and correspondence Require very large sampling of view-space Data acquisition is problematic Space-time costs are impractical

Features of Their Method Avoids reconstruction, correspondence Backward (“inverse”) warping avoids holes Optimizes errors in coordinate system of novel view Handles significantly different viewpoints Small number of input images (~10)

Typical Scenario Choose a scene point V from which to look.

Imaging Geometry Black point in each image is the virtual epipole (image of selected COP).

Color Consistency Test However: Only one correspondence is known – the virtual epipole All other correspondences are warped because the lines are in different coordinate systems. If projections were aligned, matching determines the correct color:

Overview of Approach Choose virtual viewpoint V (a scene point) For each pixel in the virtual image: –Calculate the line of sight L –Map images of L into a common coordinate system –Stack the colorings of L for comparison –Select the first consistent color as the color of the pixel

Mapping onto a Common Coordinate System One camera is selected as the reference camera R. Projections of L in all other cameras will be mapped into R’s image.

Imaging Camera 1 Transform the line in C 1 into R by the homography induced by the ground plane.

Imaging Camera 3 Geometrically, the homography displaces each pixel in C i as though the corresponding 3D point was on the ground plane. The “piercing point” always maps to the same point in R.

Pencil of Lines in Reference Camera After plane alignment, the lines in the reference camera fan from the imaged piercing point to the virtual epipoles.

Projective Geometry Review Homography: –A.k.a. Collineation, projective transformation –In P 3 : 3x4 matrix with 11 degrees of freedom Points and lines: –Point x lies on line l  x†  l  0 –Intersection of lines l and m is point p  l  m –Line joining points p and q is l  p  q

Line Configuration In R’s image plane, what is the relationship between blue and red lines?

Line Alignment redgreen Given real epipoles e i and virtual epipoles v i : for any axis point p V, M i is the projective transformation that brings each line l i into alignment with l R.

Virtual View H syn is the homography between the synthesized view and the reference view R. Position is fixed by the virtual epipoles Free parameters (can be specified in H syn ): –Orientation (look direction) –Intrinsic parameters (e.g., zoom)

Virtual Epipoles In an uncalibrated setting, the position of the virtual camera can be specified in several ways: Manually pin-point same scene point in all cameras. Pin-point in two images and geometrically infer in others using “trifocal constraints.” Pin-point in one image and use correlation techniques to find correspondence in others

Relating Virtual to Reference

Algorithm Outline For each pixel p in the synthesized image: Find the imaged piercing point p v = H syn ·p. Align all imaged lines of sight using the line-to-line transformations M i. Find the first color-consistent column. Assign pixel this color.

Color Consistency Assume Lambertian objects. A is a (n+1)  3 matrix of the column of colors in YIQ color space. is the maximal eigenvalue of the covariance matrix of A. Select first column with under a threshold. Paint with the median color of that column.

Important Details Local “smoothing”: They prefer color consistent columns whose 3D position is spatially consistent with that of neighboring pixels. Uniform regions: They flag used pixels in source images to prevent their repeated matching. Pixel scanning order: They evaluate for physical points closer to ground plane first; then farther. Ground subtraction: Except for piercing point, remove ground plane from coloring stack

Practical Simplification Cameras are coplanar Real epipoles lie on a line in R Rectifying R into the “nadir view” makes the line of epipoles go to infinity The M i line-to-line transformations become affine – simple linear stretching of the lines

Synthetic Scene Extreme change in viewpoint Objects seen through gate, while source images have only floor seen through gate

Color Analysis

Folder Scene Off-the-shelf digital camera, constant-height tripod Triangle occludes distant folder 11 images used for (e)

Puppet Scene Green smear on lower left of (e): “This floor region was not visible in any of the input images.” 9 images used for (e)

Assessment +Interesting use of projective, epipolar geometry +Needs only weak calibration -Needs failure analysis -How to define H syn ? -Explicit notion of visibility could help -Manual selection among source images? -Observation: no occlusions in source imagery – hmm…