Computer Vision No. 7: Stereo part 2

Today’s topics u Projective geometry u Camera Projection Matrix u F Matrix u Multi View Shape Reconstruction

Elementary Projective Geometry

Topics 1.Projective Plane 2. Projective Space 3. Affine Space 4. Euclidean Space

Projective Plane (2D projective space) In the Euclidean plane In the Projective plane Simply add a third coordinate of 1 at the end Homogeneous coordinate ! Scaling is NOT important. Therefore the point (0, 0, 0) is disallowed in the projective plane. : Ideal points or Points at infinity

Description in Projective Plane A line in the projective plane is described as… A line through the point (x, y)… A line through a point

Duality in Projective Plane A point and a line The line through point and If point, and are on the same line The cross point of line and If line, and are passing through the same point

Conics and Dual Conics For symmetric 3 x 3 matrix … All points which satisfy …. Conic Dual Conic Therefore, for the matrix All lines which satisfy ….

Transformation in Projective Plane In the Projective plane, by 3 x 3 non-singular matrix T A point is transformed into A line is transformed into A conic is… Similarly a dual conic is…

In the Euclidean space In the Projective space Homogeneous coordinate ! : Ideal points, Points at infinity A plane in the projective space is described as… A plane through a point Projective Space (3D Projective space)

Quadrics and Dual Quadrics For symmetric 4 x 4 matrix … All points which satisfy …. Quadric Dual Quadric Therefore, for the matrix All planes which satisfy …. Similarly a dual quadric is… A quadric is transformed by T into

N-Dimensional Affine Space Among transformations T in the N-d projective space Affine Transformation In N-D affine space… : point at infinity The group of the points at infinity : plane at infinity By an affine transformation a point at infinity

N-Dimensional Euclidean Space Among N-d affine transformations Euclidean Transformation a group of points at infinity, which satisfy By an Euclidean transformation N=3: Absolute conic N=2: Absolute point

3-Dimensional Euclidean Space By an Euclidean transformation Absolute Dual Quadric The absolute dual quadric is invariant under Euclidean transformation!

Camera Projection

Topics 1.Pinhole Camera Model 2. Camera Parameters 3. Lens Distortion

Pinhole Camera

Pinhole Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) Perspective projection View point (Optical center) (sX, sY, sZ)

Pinhole Camera Model The projected point on the image plane (z = -1) The Rotation matrix of camera pose The camera position In Projective Geometry

Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v) the Principal point skew

Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v) the Principal point skew f : Focal Length a : aspect ratio

Camera Projection Matrix Transformation of a point in the 3D world into a 2D picture… Camera Matrix (3 x 4) Camera calibration 11 parameters: rotation 3 + translation 3 + intrinsic 5 Estimate of these 11 parameters (= camera matrix)

Lens distortion Short focal length Long focal length

Removal of lens distortion

Lens Distortion Z.Zhang (‘00) PAMI R. Sagawa (‘05) IROS Non-parametric

F Matrix

Topics 1.Epipolar Geometry 2.E & F Matrix 3.Homography 4.Multi View Geometry

Correspondence Problem Given two images, we want to solve a problem…… For a point in image 1, decide which point in image 2 it corresponds to. ? plane 1 plane 2 Eye 1 Eye 2

Epipolar Geometry C1C1 C2C2 3D point Epipolar plane Epipole Epipolar line Baseline

Essential Matrix C1C1 C2C2 Essential matrix : E

Essential & Fundamental Matrix Image planes Pictures Fundamental matrix : F Image 1 Image 2

Fundamental Matrix image 1 image 2 constraint Depends on the camera configuration!!

Fundamental Matrix are on the same plane! Three vectors

Epipolar Constraint

Property of F Matrix at most Rank 2 There are some points which satisfy

Null Space of F Matrix (epipole) for All epipolar lines in image 2 pass through the point

Null Space of F Matrix (epipole) Similarly,for : Epipoles All epipolar lines in image 1 pass through the point

Epipoles Epipoles: intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction C1C1 C2C2 e1e1 e2e2

Examples

Computing F Matrix (8-point algorithm)

Computing F Matrix Epipolar pencil by linear solution (due to noise and error)

Computing F Matrix Singular value decomposition (SVD) Without noise, σ 3 must be 0 modification

Computing F Matrix

Homography P In Camera 1 Camera 1 Camera 2 The distance between camera 1 and the plane In Camera 2

Homography and F Matrix P Camera 1 Camera 2

Planer homography mosaicing

camera Image plane’

Image rectification Camera 1Camera 2

Rectification Fusiello (MVA 2000) Pollefeys

Constraints of 3 images LLine – line – line 3 x 3 matrix

Constraints of 3 images L Line – line – line Tensor description L Point – line – line

Constraints of 3 images Point – point – line L Point – point – point

2-Views Geometry X M : (3 x 2) x (4 + 2) matrix

2-Views Geometry X M : (3 x 2) x (4 + 2) matrix In tensor form

3-Views Geometry X M : (3 x 3) x (4 + 3) matrix

4-Views Geometry X

Camera Calibration

Topics 1.3D Ref. 2.2D Ref. 3. 1D Ref. 4. Self Calibration (Kruppa equations)

The meaning of camera calibration Absolute Dual Quadric Camera Calibration = to estimate the image of the absolute quadric

Camera Calibration (3D Ref.) known

Homography （ 2D-2D mapping ） (x, y) (u, v) Homography (3 x 3) the origin

Camera Calibration (2D Ref.) Without loss of generality… Z.Zhang (‘00) PAMI

Camera Calibration (2D Ref.) known

Camera Calibration (2D Ref. Results)

Camera Calibration (1D Ref.) Z.Zhang (‘04) PAMI On the line in 3D :const.:fix

Camera Calibration (1D Ref. Results)

Self Calibration In most cases, there are NOT any reference objects… be able to calibrate the camera by unknown scenes

Homography at infinity P Camera 1 Camera 2 Considering P on the infinity plane … and on the absolute quadric… In camera 2 coordinate system

The image of the absolute quadric P Camera 1 Camera 2 The image of the absolute quadric at the infinity plane

The Kruppa equations P Camera 1 Camera 2 Given the F Matrix…

References u Multiple View Geometry Hartley and Zisserman, Cambridge u The Geometry of Multiple Images Faugeras and Luong, MIT u Three-Dimensional Computer Vision Faugeras, MIT u Algebraic Projective Geometry Semple and Kneebone, Oxford

Multiple View Shape Reconstruction

Topics Basic Principle Surface Approach  Multi-baseline stereo Volumetric Approach  Volume intersection method Combinatorial Approach  Photo hull Extended Algorithm Level Set Approach Graph-cut Approach

Basic Principle #1 Surface Approach

u Try to reconstruct a surface of an object –Find depth d that minimizes some matching cost function F (d) defined on N views Camera 1 Camera 2 Camera N

Matching Cost Function (b) SSSD (sum of sum. of squared difference) (a) SSAD (sum. of sum of absolute difference) (c) SNCC (sum of normalized cross correlation) Camera 1 (reference) Camera 2 Camera N Epipolar line

Why Multiple Views? u Binocular stereo is sometimes ambiguous e.x. F(d) = SAD (sum. of absolute difference) d Camera 1Camera 2 d Which is the right match? b

Multi-baseline Stereo T. Kanade, M. Okutomi, and T. Nakahara, "A Multiple-Baseline Stereo Method," Proc. ARPA Image Understanding Workshop, pp , Camera 1 (reference) Camera 2 d Camera 3Camera N d SAD 2 d SSAD d SAD 3 d SAD N Unique minima

Video-rate Stereo Vision u Hardware implementation is easy S. Kimura, T. Kanade, H. Kano, A. Yoshida, E. Kawamura, K. Oda, “CMU video-rate stereo machine", Proc. of Mobile Mapping Symposium, 1995 Image capture Apply LoG filter SSAD computation Disparity computation Block matching LUT instead of rectification SAD computation

Real-time Robot Vision 9 cameras stereo

Real-time Robot Vision u Model-based 3D Object Tracking

Virtualized Reality ™ u 51 video cameras surrounding the scene –Recorded to VCR –Off-line 3D modeling T. Kanade, P. Rander, P. J. Narayanan, “Virtualized Reality: Constructing Virtual Worlds from Real Scenes”, IEEE MultiMedia, vol.4, no.1, pp.34-47, 1997 Large scene from overlapped dense stereo images Input images 3D video

Basic Principles #2 Volumetric Approach

u Try to reconstruct a volume that contains an object –Detect silhouette in each image –Backproject each silhouette –Intersect backprojected volumes Binary Images

Volume Intersection u Reconstruction contains the true scene –But is generally not the same

Visual Hull u Minimum convex hull of an object ⇒ A volume obtained from infinite number of surrounding cameras Visual hull True shape Reconstructed volume Concave area is never recovered Bound of the reconstruction

Voxel Algorithm for Volume Intersection u Find voxel that is projected inside silhouette in all the images –O(NM 3 ), for N images, M 3 voxels –Don ’ t have to search 2 M 3 possible scenes! Octree representation

Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 1

Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 2

Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 3

Results u Reconstruction from 4 synthetic images Silhouette images 16x16x16 voxel space 32x32x32 voxel space64x64x64 voxel space

How do we get silhouette images? u Dependent on the purpose of the application –Interactive Foreground Separation »Building static 3d model from small number of images –Statistical Background Subtraction »Motion capture system in controlled environment from large number of image sequence (multiple video streams)

Interactive Foreground Separation u Use of an interactive segmentation program Y. Li, J. Sun, C.K. Tang, H.Y. Shum, “Lazy snapping,” ACM Trans. on Graphics Vol.23,No. 3, pp.303—308, Foreground regionBackground region SeparationWrong segmentation Background region Separation See “Example of Graph-cut” in the lecture 6

Building Static 3d Model u Volume intersection from 8 images –No concavities

Brightness component Chromaticity component Statistical Background Subtraction Background imageImage of interest ifthen pixel Ii is foreground else if then pixel Ii is shadow else pixel Ii is shadow

Choice of Threshold u Learn statistics of brightness and chromaticity component for each pixel Brightness distributionChromaticity distribution P % P is around 99%

Motion Capture System in Controlled Environment Input image Foreground & shadow Silhouette image

Basic Principles #3 Combinatorial Approach

1. Start with an initial volume uArbitrary rectangular shape uResult of volume intersection 2. The volume is iteratively carved to be getting closer to the true shape uConsidering photo-consictency u2 approaches: Voxel coloring and Space-carving Carve an inconsistent surface voxel

Photo-consistency u Color variance of the imaged voxel among the cameras If color variance > threshold the voxel is carved If color variance < threshold the voxel is preserved Voxel of interest True surface

Voxel Coloring Approach Input Images (Calibrated) Goal: Assign RGBA values to voxels in V photo-consistent with images u Discrete formulation

1. Choose voxel 2. Project and correlate 3. Color if photo-consistent Visibility Problem In what order do we have to choose a voxel? u In which images is each voxel visible?

The Global Visibility Problem u Inverse Visibility u known images Unknown Scene u Which points are visible in which images? Known Scene u Forward Visibility u Unknown images

Layers Solution: Depth Ordering SceneTraversal Condition: voxel space and group of cameras must be completely separated voxel space and group of cameras must be completely separated u Visit occluders first

Space-carving Algorithm u General case Image 1 Image N …... –Initialize to a volume V containing the true scene –Repeat until convergence –Choose a voxel on the current surface –Carve if not photo-consistent –Project to visible input images

Output of Space-carving: Photo hull u The Photo Hull is the UNION of all photo-consistent scenes in V –Tightest possible bound on the true scene True Scene V Photo Hull V

u Algorithm finds a surface consistent with the input images, but not necessarily the actual surface Typical Error in Photo hull actual surface reconstructed surface Cusping artifact Floating voxel artifact

Space-carving Implementation u Multi-pass plane sweep –Efficient, easy to implement Sweep in –z direction Sweep in -x directionSweep in -y direction Repeat

Results Details are recovered Volume intersection Space carving with textures Excessively carved Input image Using 16 images 300k polygons

Extended Algorithm #1 Level Set Approach

u Define problem in 1 higher dimension –An additional dimension, time t, is added φ(p(t),t). [Osher98] »P(t) is a point in the space »Φ(p(t),t) is a signed distance function –Surface, zero level set φ=0, moves along its normal direction –Front propagation is guided by Partial Differential Equation (PDE). –Cross-correlation is used as a constraint in PDE. [Faugeras98] True shape

How to move the level set surface u Define a velocity field, F, that specifies surface points move in time –In the case of stereo, F is a function of the matching cost on p(t) in multi-baseline stereo –F decreases to 0 around the true surface u Build an initial value for the level set function, φ(p(0),0), based on the initial surface u Adjust φ over time –Surface at time t is defined by φ(p(t),t)=0

Level Set Formulation u Constraint: level set value of a point p(t) on the surface must always 0 u By the chain rule u Since F supplies the speed in the outward normal direction u Hence evaluation equation for φ is Essentially a gradient-descent local optimization problem

Front Propagation by PDE u Front Propagation can be efficiently calculated in level set framework –Fast marching method –Narrow band method

Results u Can handle topology change

Extended Algorithm #2 Graph-cut Approach

Goal Calibrated images of Lambertian scene 3D model of scene

Volumetric Graph-cuts u Energy function to minimize Outer surface (Visual hull) Inner surface (at constant offset from outer surface) Volume between outer surface and S (to avoid over-smoothing) Reconstructed surface (S) Integral of photo-consistency over S (S should be photo-consistent) Photo-consistency

Designing a Graph 1. Assign a node for each voxel 2. Connect neighboring nodes –Weight is set as ρ(x) 3. Connect outer nodes to source –Weight is infinite 4. Connect inner nodes to sink –Weight is infinite Outer surface Inner surface Sink Source

Graph-cut algorithm minimizes the energy Designing a Graph 1. Connect all the nodes to source –Weight is λ Outer surface Inner surface Sink Source A cut that separates source and sink is equal to Cut

Results Volume intersection Volumetric Graph-cut

Summary u Multi-baseline stereo for accurate depth map –Good for real-time applications, dense depth map is obtained directly u Voxel-based reconstruction –Volume intersection for conservative estimate –2 background subtraction methods for different applications –Concavities can be recovered by considering Photo-consistency u Extended algorithm –2 approaches to add smoothness constraint –Level set approach solves local optimization problem »can handle topology change –Graph-cut approach solves global optimization problem »cannot handle topology change

u Volume intersection –Martin & Aggarwal, “ Volumetric description of objects from multiple views ”, Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp –Szeliski, “ Rapid Octree Construction from Image Sequences ”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp u Voxel coloring and Space-carving –Seitz & Dyer, “ Photorealistic Scene Reconstruction by Voxel Coloring ”, Proc. Computer Vision and Pattern Recognition (CVPR), 1997, pp –Kutulakos & Seitz, “ A Theory of Shape by Space Carving ”, Proc. ICCV, 1998, pp u Extended algorithm –Faugeras & Keriven, “ Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp –G. Vogiatzis and P. H. S. Torr and R. Cipolla, “Multi-View Stereo via Volumetric Graph-Cuts”, Proc. CVPR, 2005, pp References

Next Week u 6/7 Motion understanding –Prof. Ikeuchi