1. 3D-2D registration Kazunori Umeda Chuo Univ., Japan umeda@mech.chuo-u.ac.jp http://www.mech.chuo-u.ac.jp/umedalab/ CRV2010 Tutorial May 30, 2010

2. Registration of range image and color image Necessary for texture mapping

3. Range image (3D model)

4. Color image

5. When 3D-2D registration is given,

6. Texture mapping

7. Parameters to obtain for 3D-2D registration Color camera Image plane Object (range image) Range image sensor Sensor coordinate system Projection of range intensity image Intensity image Intrinsic parameters Extrinsic parameters (Distortion parameters) X Y Z

8. Extrinsic parameters Object’ rotation R and translation t or camera’s orientation R c and position t c Color camera Range image sensor Sensor coordinate system X Y Z R, t (R c, t c ) Parameters to obtain for 3D-2D registration

9. Intrinsic parameters (X,Y,Z)(X,Y,Z)(u,v)(u,v) 3D spaceImage plane camera coordinate system  u,  v : focal length/pixel size s : skew, u 0, v 0 : principal point coordinates Color camera Image plane u v Parameters to obtain for 3D-2D registration

10. Homogenous coordinates Parameters to obtain for 3D-2D registration P: 3  4 matrix 11 unknown parameters (6 extrinsic + 5 intrinsic) 2 constraints

11. When correspondences between range image and color image are given, …It is hard to obtain correspondences even manually. Parameters can be calculated. Equivalent to camera calibration problem. For extrinsic parameter estimation, Equivalent to PnP (Perspective n-Point) problem 3D2D

12. Range image

13. Range intensity image (reflectance image)

14. By using range intensity image, obtaining correspondences becomes easier! e.g., corners, edges, SIFT [Böhm 2007]

15. Our approach: gradient-based method (not explicitly using correspondences) Two 2D images are matched Gradient-based method Update camera parameters Produce a 2D image from a range image Initial camera parameters End Yes No

16. u v Projection of range intensity image Intensity image Optical flow constraint I t : difference between intensity image and projected range intensity image Tailor expansion

17. (1) Constraints for extrinsic parameters When intrinsic parameters are constant, Substituting for the optical flow constraint

18. Camera motion: v 0,  Digital camera u v

19. Linear equation for 6 motion parameters v 0,  v 0,  can be solved with 6 or more points by linear least square method. Motion parameters are supposed to be small  Iteration is necessary v 0,   R(3  3 rotation matrix) and t (3D translation vector) [Yamamoto 1985] [Horn IJCV1988] cf.

20. (2) Constraints for intrinsic parameters When intrinsic parameters are also variables, Substituting for the optical flow constraint

21. a,b,c: same as previous equation Linear equation for 6 motion parameters v 0,  and 5 intrinsic parameters v 0,  and intrinsic parameters can be solved with 11 or more points by linear least square method.

22. (3) Constraints for distortion Distortion model (the simplest) Distortion

23. Linear equation for 6 motion parameters v 0, , 5 intrinsic parameters and a distortion parameter The parameters can be solved with 12 or more points by linear least square method.

24. Implementation Differential images So as to absorb the differences between a range intensity image and an intensity image 2 images: horizontal, vertical. Prewitt operator. 1 1 1 0 0 0 0 1 0 1 0 1 Coarse to fine Control of resolution and  of Gaussian Extrinsic only  v 0 +Intrinsic  all [Irani ICCV1998]

25. Experimental results Range image sensor: ShapeGrabber PLM300 （ Slit laser, triangulation ， wavelength 670nm) R-channel ， RAW format 2560  1920  1280  960 at registration Digital camera: Nikon COOLPIX 5000 (5M, 2560  1920 pixels, 2/3” CCD, pixel dimension 3.4  m?, f=7.1-21.4mm)

26. Measurement of a range image

27. 312730 points

42. Summary ３ D-2D registration (for texture mapping, etc.) Projective geometry Obtaining camera’s extrinsic and intrinsic parameters Range intensity (reflectance) image is useful With correspondences Equivalent to {camera calibration / PnP} problems Using optical flow constraint Explicit correspondences are not necessary Linear equation for motion parameters