1. Jan. 12, 1998 CS260 Winter 1999-Wittenbrink, lect. 3 1 CS 260 Computer Graphics Craig M. Wittenbrink Lecture 3: Camera Calibration and Views

2. CS260 Winter 1999-Wittenbrink, lect. 3 2 Overview Tracking Applications Tracking and Calibration (videos) Camera Models Tsai Camera Calibration Project 1 Discussion Conclusions

3. CS260 Winter 1999-Wittenbrink, lect. 3 3 Tracking Applications Virtual Reality Augmented Reality Stereo/Crystal eyes displays Multiuser Virtual Workbench 3D Mouse Animation through motion capture Gait analysis Image Based Rendering (copyright Polhemus 1998) (copyright Bouguet/CS Caltech 1998) (copyright Derby Gait Lab1998)

4. CS260 Winter 1999-Wittenbrink, lect. 3 4 Applications for Tracking: Scanning Polhemus (www.polhemus.com) Handheld Laser Scanner (HLS) Scan by time of flight light sensing to create 3D model Magnetic tracking of position and orientation of wand Handheld Laser Scanner VRML Surface Model Copyright Polhemus 1998 Copyright Polhemus 1998

5. CS260 Winter 1999-Wittenbrink, lect. 3 5 Applications for Tracking: Camera Calibration Jean-Yves Bouguet “3D Photography with weak structured lighting” presentation given to Intel Santa Clara, Dec. 5, 1997. –www.vision.caltech.edu/bougetj/ Use corners of squares to solve for camera intrinsic and extrinsic parameters Can then remove camera distortion in images (copyright Bouguet/CS Caltech 1998)

6. CS260 Winter 1999-Wittenbrink, lect. 3 6 Applications for tracking: Gait analysis Motion capture used for medical analysis, sports training, and natural looking animation –www.gait.com Derby Gait Analysis Laboratory Use 3D Elite System from BTS Milan Infrared tracking of highly reflective markers Use multiple cameras Diagnose and plan treatment (copyright Derby Gait Lab 1998)

7. CS260 Winter 1999-Wittenbrink, lect. 3 7 Applications for tracking: Virtual Studio InterSense Inertial tracker –www.isense.com Use optical gyros and accelerometers Similar to technology in flight guidance control systems Couple with magnetic, ultrasonic, or infrared tracking, IS 900CT Use Kalman filtering to fuse sensor input (copyright InterSense 1998)

8. CS260 Winter 1999-Wittenbrink, lect. 3 8 InterSense IS-900CT System Solid State inertial measurement –Angular rate of rotation and linear acceleration in 3 axes Drift removed by Ultrasonic time-of-flight distance measurements Infrared triggers beacons (copyright InterSense 1998) (copyright InterSense 1998)

9. CS260 Winter 1999-Wittenbrink, lect. 3 9 Tracking Videos

10. CS260 Winter 1999-Wittenbrink, lect. 3 10 Camera Models Watt and Watt OpenGL Tsai

11. CS260 Winter 1999-Wittenbrink, lect. 3 11 Camera Models Watt and Watt page 7-8 (Figure 1.4) Center of projection, C View direction, N View up vector, V Positive X axis, U World Coordinates C N V U Camera Coordinates

12. CS260 Winter 1999-Wittenbrink, lect. 3 12 OpenGL Camera Model Where camera points by default in world coordinates Points down negative z axis, from origin Specified through model, perspective, and viewport matrices World Coordinates Camera View direction

13. CS260 Winter 1999-Wittenbrink, lect. 3 13 Tsai Camera Model Distortion due to non pinhole camera –Fig. 1, page 326 from paper: Roger Y. Tsai, “A Versatile Camera Calibration …”, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, Aug, 1987, pages 323-344. image plane World Coordinates Camera Coordinates or Effective focal length

14. CS260 Winter 1999-Wittenbrink, lect. 3 14 Tsai Camera Model Looks up positive Z –OpenGL looks down negative Z (or is left handed) OpenGL camera model Tsai camera model

15. CS260 Winter 1999-Wittenbrink, lect. 3 15 Camera Calibration Two good sources: Tsai, R.Y. “A Versatile Camera Calibration Technique for High- Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses”, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, pages 323-344. Code available: www.cs.cmu.edu/People/rgw/TsaiCode.html Heikkila, Janne and Olli Silven, “A Four-step Camera Calibration Procedure with Implicit Image Correction”, In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’97), San Juan, Puerto Rico, pages 1106-1112, June 1997. Matlab tool box available: www.ee.oulu.fi/~jth/calibr/Calibration.html

16. CS260 Winter 1999-Wittenbrink, lect. 3 16 Camera Calibration (cont.) Basic idea, use known 3D world coordinate points to solve for the camera’s transformation and distortion Solve for two types of parameters: –intrinsic parameters-internal camera optical characteristic –extrinsic parameters-3D position and orientation of camera

17. CS260 Winter 1999-Wittenbrink, lect. 3 17 Camera Calibration-Motivation Infer 3D location from 2D images Detect features such as laser spots, corners of equipment, structured light Determine Ray in 3D space that object point must lie on Applicable to many applications (as seen previously) Tsai’s approach: 1) autonomous 2) accurate 3) reasonably efficient

18. CS260 Winter 1999-Wittenbrink, lect. 3 18 Tsai’s review of techniques 1. Full-scale nonlinear optimization –advantage: accuracy comparable to Tsai –disadvantage: not automatic, compute intensive –2. Computing perspective transformation matrix first using linear equation solving –adv: linear optimization used –dis: lens distortion not included, error prone –3. Two plane method –adv: linear optimization dis: more unknowns –4. Geometric technique –adv: linear optimization dis: must provide focal length

19. CS260 Winter 1999-Wittenbrink, lect. 3 19 Tsai Experimental Setup Use a CCD camera Aquire a grayscale image, Threshold Link edge points to extract boundary edges. Compute across edges to get true edge fit straight lines, and solve for corners Letraset sheet positioned on steel block Find corners

20. CS260 Winter 1999-Wittenbrink, lect. 3 20 Tsai Transformations Four transformations All encapsulated in the following parameters –RotationRx, Ry, Rz –TranslationTx, Ty, Tz –focal lengthf –image centerCx, Cy –scale factor sx –radial distortionkappa1

21. CS260 Winter 1999-Wittenbrink, lect. 3 21 Tsai’s approach, Camera Model Four steps in transformation (Fig 2 from paper) Rigid body transformation Projection Matrix Radial lens distortion Viewport transform 3D World Coordinates 3D Camera coordinates Ideal undistorted image coordinate Distorted image coordinate Computer image coordinate

22. CS260 Winter 1999-Wittenbrink, lect. 3 22 Tsai’s approach, Camera Model Four steps in transformation World Coordinates Camera Coordinates or

23. CS260 Winter 1999-Wittenbrink, lect. 3 23 Step 1: Rigid Body Transformation 3D World Coordinate to 3D camera coordinate system Rotation followed by translation Similar to OpenGL Model Transformation

24. CS260 Winter 1999-Wittenbrink, lect. 3 24 Step 2: Perspective Projection 3D camera coordinates to Ideal undistorted image coordinate, involves focal length, and perspective foreshortening

25. CS260 Winter 1999-Wittenbrink, lect. 3 25 Step 3: Radial Distortion is the distorted or true image coordinate on the image plane, and The distortion is calculated by

26. CS260 Winter 1999-Wittenbrink, lect. 3 26 Step 4: Viewport calculation Calculate actual discrete pixel address location within image, with given centering

27. CS260 Winter 1999-Wittenbrink, lect. 3 27 Tsai transformation operations World space coordinates to computer image space coordinates Step 1: Rigid body Step2: Projection Step 3: distortion Distorted image coordinates Undistorted image coordinates camera coordinates

28. CS260 Winter 1999-Wittenbrink, lect. 3 28 Tsai’s versus OpenGL Four steps in transformation (Fig 2 from paper) Rigid body transformation Projection Matrix Radial lens distortion Viewport transform 3D World Coordinates 3D Camera coordinates Ideal undistorted image coordinate Distorted image coordinate Computer image coordinate Model-View Matrix Projection Matrix Perspective Division Viewport Transformation Object coordinates eye coordinates clip coordinates Normalized device coordinates window coordinates

29. CS260 Winter 1999-Wittenbrink, lect. 3 29 OpenGL Camera Model glTranslatef(0.0, 0.0, -5.0) Can see either camera moving, or object moving, it only modifies the Model-view matrix -5.0

30. CS260 Winter 1999-Wittenbrink, lect. 3 30 OpenGL transformation operations Normalized device coordinates from object coordinates Model-View Matrix Projection Matrix Perspective division Normalized device coordinates Clip coordinates Eye coordinates

31. CS260 Winter 1999-Wittenbrink, lect. 3 31 OpenGL transformation Operations cont. Calculation of window coordinates Window center width height factor and offset computed with zNear zFar

32. CS260 Winter 1999-Wittenbrink, lect. 3 32 OpenGL transformation Operations cont. Viewport set by glviewport(int x, int y, sizei w, sizei h); Z depth factor and offset are set by glDepthRange(campd n, clampd f)(zNear, zFar)

33. CS260 Winter 1999-Wittenbrink, lect. 3 33 How the calibration code works Approach, use the radial alignment as a constraint Function of only the relative rotation and translation between the camera and calibration points World Coordinates Camera Coordinates or Oi and Pd and Pu on same line

34. CS260 Winter 1999-Wittenbrink, lect. 3 34 Project 1, Set Tsai view in OpenGL Description of files Data set description No glyphs because of occlusion G01bsmall.gif/ppm Output from my solution

35. CS260 Winter 1999-Wittenbrink, lect. 3 35 Project 1: File definitions Pink blocks show what is to be implemented Error G01.corr G01.cpcc Image proc. Tsai View 35 mm camera Project 1 G01.ppm Output image cmp

36. CS260 Winter 1999-Wittenbrink, lect. 3 36 Project 1: File definitions cont. G01.cor - correlation file xw,yw,zw, Xu,Yu for all seeable glyph locations. Images have been warped to remove radial distortion G01.cpcc - view parameters from Tsai: Rx, Ry, Rz, Tx, Ty, Tz, f, Cx, Cy, sx, kappa1 Code is provided to read *.cor and *.cpcc Computed Errors are on the order of a pixel ~ 1

37. CS260 Winter 1999-Wittenbrink, lect. 3 37 Project 1: cont 4 file sets are given, different viewpoints Original images are very large (1536x1024 is subsampled from Kodak Photo CD resolution 4X in each dimension of that) A default view is provided with gluLookAt You could compute errors with that, and they will be very large Read README, run demo scripts RUN, RUN2

38. CS260 Winter 1999-Wittenbrink, lect. 3 38 Project 1: cont. Actual world coordinates description planes are drawn in provided code at locations of poster, and glyphs are drawn too. World Coordinates (x=587,y=588,z=-824) (x=587,y=588,z=0) (x=-825,y=-706,z=0) Delta z = 824 Delta y = 1294 Delta x = 1412

39. CS260 Winter 1999-Wittenbrink, lect. 3 39 Conclusions Tracking has wide application in 3D graphics Camera calibration is important for your first project and specifying view is essential in 3D graphics as well Tsai’s parameters were described Project 1 was reviewed –Tsai, R.Y. “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses”, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, pages 323-344. Next time: Assigned Reading Szeliski’s slides on image based rendering.