Planar Matchmove Using Invariant Image Features Andrew Kaufman

I will talk about Matchmove Problem “Augmented Reality Using Un-calibrated Video Sequences” – Cornelis et al. My project

Matchmove Goal: Insert virtual objects into real video sequences Applications  Film  Advertising  Sports  Augmented Reality Systems

AR from Video Problems with inserting virtual objects  Rigid registration  Occlusion  Proper illumination

AR from Video Problems with inserting virtual objects  Rigid registration  Occlusion  Proper illumination

Rigid Registration Main Difficulty  Jittering 3D Reconstruction Scheme  Extract motion and structure

Motion and Structure Recovery (MSR) Based on feature tracking  Points, lines, curves, regions  Harris Corner Detector points / corners with  Significant orthogonal intensity change

Corner Matching Initially  Using small search region  NCC along epipolar line RANSAC algorithm  Treats small movements as outliers Levenberg-Marquardt optimization

Initializing MSR m k ~ P k M P k projects 3D point onto image k  To scale factor Camera for two initial images  P 1 – world frame  P 2 – based on epipolar geometry Crude 3D reconstruction of matched corners

Finishing MSR For all other images  Image i has P i Retrieve m i -M corner matches  Image j Compute m i -m j corner matches Obtain m j -M matches  Determine P j

Attach Graphics Set virtual camera using P Roughly place graphics into the crude 3D environment Render and overlay

Review Initialize with two images Take 1 processed and 1 new image  Find new 2D-3D matches  Calculate new P  Construct new 3D points For each remaining frame  Find new 2D-3D matches  Calculate new P Place virtual object LOOP

Examples Static virtual objects Moving virtual objects Moving real objects deos/ deos/

Future Work Occlusions Illumination Reduce jittering  Smooth camera path  Lock to real object

My Project Planar Matchmove Track object through video  Matching SIFT features Solve for affine transformation between reference image and video frames Create graphics UI to attach graphics to reference image Transform graphics using affine parameters Render final video

Graphic Goals Boarder around object Billboard over object Attach special FX

Current Progress Objects tracked Affine parameters found Boarders drawn

Still to do Build User Interface Create graphics Render final videos

Questions ???