Vision-based Registration for AR Presented by Diem Vu Nov 20, 2003

Markerless Tracking using Planar Structure in the Scene. G. Simon, A.W. Fitzgibbon and A. Zisserman, Calibration-Free Augmented Reality. K.N Kutulakos and J.R. Vallino, 1998.

Planar-surface tracking. Camera position can be recovered from planar homography. Planar structure is common in almost all scenarios.

y x z HwHw World to image homography Image to image homography

World to image homography Consider our tracking plane is the plane Z=0 y x z HwHw

Projection matrix

y x z P

y x z P

If K and H w are known, then r 1, r 2 and t can be recovered, thus P. Question: How to compute H w ? Direct. Indirect.

Direct measurement of H w Select 4 points {x k } on a rectangle in the scene. Compute H which maps the unit square to {x k }. (0,0) (0,1)(1,1) () (1,0)

Direct measurement of H w Select 4 points {x k } on a rectangle in the scene. Compute H which maps the unit square to {x k }. Compute H w =Hdiag(1, 1 / s,1) (0,0) (0,s)(1,s) () (1,0)

Indirect measurement of H w y x z

y x z

Algorithm summary Compute (direct measure). For each frame i, compute frame to frame homography (RANSAC) Compute by:

Other … Using only 2 points in direct method ?? Matching the frame i with frame 0 in order to reduce error. Estimate intrinsic parameters K Hand-off mechanism.

Possible problems? Homography is only up-to-scale? Plain surface (no texture) or moving objects in the foreground ? Depth order, occlusion ? Speed ?

Affine virtual object representation Represent virtual objects so that their projection can be computed as a linear combination of the projection of the fiducial points.

Project a point from its affine coordinates

Compute affine coordinates from projection along two viewing direction

Algorithm Setup the affine basis

Algorithm Setup the affine basis Locate the object in 2 frames.

Algorithm Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates for each point.

Algorithm Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates for each point. Compute projection of the object and render the object in each frame.

Camera viewing direction  and  are the first and second row of  2x3. The camera viewing direction expressed in the coordinate frame of the affine basis points:  =   

Depth order w is the z-value of point p (x,y,z).

Advantages No need any metric information. Able to use with the existing hardware to accelerate graphics operations. Can be used to improve tracking.

Limitation Affine constraints. Lost of metric information.