3D-2D registration Kazunori Umeda Chuo Univ., Japan CRV2010 Tutorial May 30, 2010

Registration of range image and color image Necessary for texture mapping

Range image (3D model)

Color image

When 3D-2D registration is given,

Texture mapping

Parameters to obtain for 3D-2D registration Color camera Image plane Object (range image) Range image sensor Sensor coordinate system Projection of range intensity image Intensity image Intrinsic parameters Extrinsic parameters (Distortion parameters) X Y Z

Extrinsic parameters Object’ rotation R and translation t or camera’s orientation R c and position t c Color camera Range image sensor Sensor coordinate system X Y Z R, t (R c, t c ) Parameters to obtain for 3D-2D registration

Intrinsic parameters (X,Y,Z)(X,Y,Z)(u,v)(u,v) 3D spaceImage plane camera coordinate system  u,  v : focal length/pixel size s : skew, u 0, v 0 : principal point coordinates Color camera Image plane u v Parameters to obtain for 3D-2D registration

Homogenous coordinates Parameters to obtain for 3D-2D registration P: 3  4 matrix 11 unknown parameters (6 extrinsic + 5 intrinsic) 2 constraints

When correspondences between range image and color image are given, …It is hard to obtain correspondences even manually. Parameters can be calculated. Equivalent to camera calibration problem. For extrinsic parameter estimation, Equivalent to PnP (Perspective n-Point) problem 3D2D

Range image

Range intensity image (reflectance image)

By using range intensity image, obtaining correspondences becomes easier! e.g., corners, edges, SIFT [Böhm 2007]

Our approach: gradient-based method (not explicitly using correspondences) Two 2D images are matched Gradient-based method Update camera parameters Produce a 2D image from a range image Initial camera parameters End Yes No

u v Projection of range intensity image Intensity image Optical flow constraint I t : difference between intensity image and projected range intensity image Tailor expansion

(1) Constraints for extrinsic parameters When intrinsic parameters are constant, Substituting for the optical flow constraint

Camera motion: v 0,  Digital camera u v

Linear equation for 6 motion parameters v 0,  v 0,  can be solved with 6 or more points by linear least square method. Motion parameters are supposed to be small  Iteration is necessary v 0,   R(3  3 rotation matrix) and t (3D translation vector) [Yamamoto 1985] [Horn IJCV1988] cf.

(2) Constraints for intrinsic parameters When intrinsic parameters are also variables, Substituting for the optical flow constraint

a,b,c: same as previous equation Linear equation for 6 motion parameters v 0,  and 5 intrinsic parameters v 0,  and intrinsic parameters can be solved with 11 or more points by linear least square method.

(3) Constraints for distortion Distortion model (the simplest) Distortion

Linear equation for 6 motion parameters v 0, , 5 intrinsic parameters and a distortion parameter The parameters can be solved with 12 or more points by linear least square method.

Implementation Differential images So as to absorb the differences between a range intensity image and an intensity image 2 images: horizontal, vertical. Prewitt operator Coarse to fine Control of resolution and  of Gaussian Extrinsic only  v 0 +Intrinsic  all [Irani ICCV1998]

Experimental results Range image sensor: ShapeGrabber PLM300 （ Slit laser, triangulation ， wavelength 670nm) R-channel ， RAW format 2560  1920  1280  960 at registration Digital camera: Nikon COOLPIX 5000 (5M, 2560  1920 pixels, 2/3” CCD, pixel dimension 3.4  m?, f= mm)

Measurement of a range image

points

Summary ３ D-2D registration (for texture mapping, etc.) Projective geometry Obtaining camera’s extrinsic and intrinsic parameters Range intensity (reflectance) image is useful With correspondences Equivalent to {camera calibration / PnP} problems Using optical flow constraint Explicit correspondences are not necessary Linear equation for motion parameters