1. JASS `04 Benjamin Fingerle, Christian Wachinger1 2nd Joined Advanced Student School Calibration Benjamin Fingerle Christian Wachinger

2. JASS `04 Benjamin Fingerle, Christian Wachinger2 A Definition of Calibration “Calibration is the process of instantiating parameter values for mathematical models which map the physical environment to internal representations, so that the computer’s internal model matches the physical world.” Mihran Tuceryan

3. JASS `04 Benjamin Fingerle, Christian Wachinger3 Augmented Reality Requires Highly Precise Pose Estimation In an AR environment, reality is modelled in a virtual world by arranging digital counter parts of various objects positioned and orientated, based on data gathered by tracking technology This virtual world is then enriched with context based information and somehow projected back to the user in the physical world Hence any inaccuracy in estimating the pose of a real world object as well as imprecise projection from virtual to real world causes a loss of realism and thus usability

4. JASS `04 Benjamin Fingerle, Christian Wachinger4 Additional Requirements for Calibration in AR Environments Calibration procedures for different objects have to be As autonomous as possible –To make it a convenient process –To keep the possible number of user-related errors down Efficient –Some applications even require real-time capabilities Versatile –To make calibration procedures reusable in different AR setups

5. JASS `04 Benjamin Fingerle, Christian Wachinger5 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

6. JASS `04 Benjamin Fingerle, Christian Wachinger6 A Motivating Scenario A mobile user - Joe - is wearing an Optical See Through Head Mounted Display (OST-HMD) Joe stands in front of an apparently empty table But Joe seeing through his display gets the vision of several 3D-Objects placed on the table By using his hands Joe can move the objects on the table

7. JASS `04 Benjamin Fingerle, Christian Wachinger7 3 A Motivating Scenario II 21 1.Wrongly positioned and orientated 2.Correctly positioned but wrongly orientated 3.Correctly posed

8. JASS `04 Benjamin Fingerle, Christian Wachinger8 Different objects are to calibrate In the example following parameters have to be estimated Pose of the table relatively to the room Pose of Joe’s head relatively to the room Pose of Joe’s hands relatively to the room Parameters of Joe’s OST-HMD This is done using 3DOF - magnetic pointer based object calibration for the table 6DOF - magnetic tracking - the marker rigidly fixed at Joe’s HMD SPAAM method for calibrating the OST-HMD Stereovision based tracking of Joe’s hands To use above additional objects have to be calibrated A magnetic tracker transmitter

9. JASS `04 Benjamin Fingerle, Christian Wachinger9 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

10. JASS `04 Benjamin Fingerle, Christian Wachinger10 3DOF - Pointer Calibration p w = p m + R m p t Determination of the unknown vectors p w and p t 6 unknown parameters as p w and p t are 3D-Vectors Several Measurements have to be taken Least squares method

11. JASS `04 Benjamin Fingerle, Christian Wachinger11 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

12. JASS `04 Benjamin Fingerle, Christian Wachinger12 3DOF - Pointer Based Object Calibration Calculation of the transformation from the world coordinate system to the object coordinate system Coordinates are known in the object coordinate system p l and in the world coordinate system p w. p w = R * p l + T, R rotation, T translation =>12 unknown parameters => Several measurements => Solving the optimization problem:

13. JASS `04 Benjamin Fingerle, Christian Wachinger13 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

14. JASS `04 Benjamin Fingerle, Christian Wachinger14 Stereo Vision Camera Calibration Motivation: Joe’s hands’ poses to be tracked by a static stereo- vision camera This is done by Triangulation –Analysing the two 2D-images for known landmarks applied to Joe’s hands –Inferring a 3D ray for each landmark and each image on which the landmark is aligned –Intersecting the two rays for each landmark to get its 3D position –Inferring the orientation by analysis of the landmark positions

15. JASS `04 Benjamin Fingerle, Christian Wachinger15 Intrinsic and Extrinsic Parameters Have to Be Calibrated To be able to apply triangulation to camera images several camera specific parameter have to be known (Intrinsic Parameter) So far the hand’s poses are known relatively to the camera’s coordinate system (CCS) but they are needed to be in world coordinate system (WCS) Thus the static camera’s pose relatively to the WCS has to be determined as well (Extrinsic Parameter)

16. JASS `04 Benjamin Fingerle, Christian Wachinger16 The Basic Camera Model (Pinhole Camera) Intrinsic Parameters that have to be determined Focal length f [ 1 DOF]

17. JASS `04 Benjamin Fingerle, Christian Wachinger17 Spatial Relation of CCS to WCS has to be known Joe should be able to move the virtual objects displayed on the table by hand movements The virtual objects coordinates are known in the WCS Joe’s hands’ poses so far are known relatively to the CCS To obtain the spatial relation between his hands and the virtual objects the spatial relation between the CCS and the WCS has to be known

18. JASS `04 Benjamin Fingerle, Christian Wachinger18 Camera’s Pose relative to WCS forms Extrinsic Parameters Extrinsic Parameters that have to be estimated: Rotation R [3DOF] Translation T [3DOF]

19. JASS `04 Benjamin Fingerle, Christian Wachinger19 The Relation of 2D - Image Points to their 3D - Counterparts P c = R P w + T x u = f (x c /z c ) y u = f (y c /z c )

20. JASS `04 Benjamin Fingerle, Christian Wachinger20 Using CCDs introduces additional Intrinsic Parameters The use of CCD - Chips introduces additional intrinsic Parameters that have to be calibrated The image origin is shifted relatively to the optical centre Due to CCD-typical line-sampling imprecision a horizontal scale factor has to be introduced

21. JASS `04 Benjamin Fingerle, Christian Wachinger21 CCD Related Intrinsic Parameters x m = s x (x u /∆ x )(# xMem /# xCCD ) + t x y m = y u /∆ y + t y Additional Intrinsic Parameters shift S = (t x, t y ) of the image relatively to the optical centre [2 DOF] horizontal scale factor s x [1 DOF]

22. JASS `04 Benjamin Fingerle, Christian Wachinger22 Lens Distortion has to be Considered Efficient algorithms for determining the intrinsic parameters f, t x, t y and s x together with the extrinsic parameters R and T exist But optical tracking based on such calibrated cameras proved to be imprecise This is due to Lens Distortion from which common of the shelf-cameras suffer Lens distortion can be split into tangential - and radial lens distortion whereby the latter proved to be of special importance to optical tracking and thus camera calibration

23. JASS `04 Benjamin Fingerle, Christian Wachinger23 Radial Lens Distortion Requires Two More Parameters Modelled with infinite series x u = x d (1 + k 1 r 2 + k 2 r 4 ) y u = y d (1 + k 1 r 2 + k 2 r 4 ) r = (x d 2 + y d 2 ) 1/2 Additional Intrinsic Parameters: Distortion Coefficient k 1 [1DOF] Distortion Coefficient k 2 [1DOF]

24. JASS `04 Benjamin Fingerle, Christian Wachinger24 From WCS to Memory P w = (x w, y w, z w ) | point in WCS P c = (x c, y c, z c ) | point in CCS | R [3DOF] | T [3DOF] P u = (x u, y u ) | undistorted image | f [1DOF] P d = (x d, y d ) | distorted image | k 1 [1DOF] | k 2 [1DOF] P m = (x m, y m ) | distorted memory image | S [2DOF] | s x [1DOF] x c = r 1 x w + r 2 y w + r 3 z w + T x, y c = r 4 x w + r 5 y w + r 6 z w + T y, z c = r 7 x w + r 8 y w + r 9 z w + T z x u = fx c,y u = fy c z c z c x u = x d (1 + k 1 r 2 + k 2 r 4 ),| r = (x d 2 + y d 2 ) 1/2 y u = y d (1 + k 1 r 2 + k 2 r 4 ) x d = ∆ x # xCCD (x m - t x ),y d = ∆ y (y m - t y ) s x # xMem

25. JASS `04 Benjamin Fingerle, Christian Wachinger25 The “Tsai Calibration Method” Satisfies all Requirements Tsai’s method Takes a set of known non-coplanar calibration points in WCS Estimates both extrinsic and intrinsic parameters of a statically mounted of the shelf CCD camera And works –autonomously –Efficiently –And of provable accuracy

26. JASS `04 Benjamin Fingerle, Christian Wachinger26 Tsai’s Method Works in Two Stages Prerequisites: – # mem, # CCD, ∆ x, ∆ y from device specification –S = (t x, t y ) = (∆ x /2, ∆ y /2) –Measure non-coplanar calibration points P wi = (x wi,y wi,z wi ) in WCS –Take an image and find calibration points P mi = (x mi, y mi ) Stage 1: Compute –Transformation matrix R –x-and y-component T x, T y of Translation T –Horizontal scale factor s x Stage 2: Compute –Effective focal length f –Radial lens distortion coefficients k 1 and k 2 –z-component T z of Translation T

27. JASS `04 Benjamin Fingerle, Christian Wachinger27 Stage 1 … Based on parallelism observation: Radial distortion does not influence direction from origin to image point (0 0 f) T (x d y d f) T || (0 0 z c ) T (x c y c z c ) T Thus following holds (x d y d ) T = c (x c y c ) T x d = cx c, y d = cy c => x d y c = cx c y c = y d x c Now substitute x c and y c by their counterparts x w and y w transformed with R and translated by T x d = y d x w r 1 s x + y d y w r 2 s x + y d z w r 3 s x + y d T x s x - d x w r 4 - x d y w r 5 - x d z w r 6 T y

28. JASS `04 Benjamin Fingerle, Christian Wachinger28 Parallelism Constraint

29. JASS `04 Benjamin Fingerle, Christian Wachinger29 … Stage 1 for each calibration memory point P mi compute the interim distorted image point P di ’ while setting s x to 1 for each pair P di ’ and P wi formulate the former linear equation x di = … There are 7 free terms: ( r 1 s x /T y ), ( r 2 s x /T y ), ( r 3 s x /T y ), ( s x T x /T y ), ( r 4 /T y ), ( r 5 /T y ), ( r 6 /T y ) With more than 7 calibration points this system of linear equations is over determined and thus can be solved (with least square error method) From these 7 terms R, T x, T y and s x can be efficiently extracted by application of geometric observations

30. JASS `04 Benjamin Fingerle, Christian Wachinger30 Stage 2 Step 1: Compute an approximation of f and T z by ignoring lens distortion Step 2: Use the approximation of f and T z to compute the exact solution of f, T z, k 1 and k 2

31. JASS `04 Benjamin Fingerle, Christian Wachinger31 … Stage 2, Step 1… Ignoring lens distortion leads from f (y c /z c ) = y u = y d (1 + k 1 r 2 + k 2 r 4 ) to f (y c /z c ) = y u = y d for each calibration point i formulate linear equation f (y ci /z ci ) = y di Substituting y c, z c and y d leads to f (r 4 x wi + r 5 y wi + r 6 z wi + T y ) = ∆ y (y mi - t y ) (r 7 x wi + r 8 y wi + r 9 z wi + T z )

32. JASS `04 Benjamin Fingerle, Christian Wachinger32 … Stage 2, Step 2 We get an over determined and thus solvable system of linear equations with two free variables f and T z These approximation values are taken as initial guess for an algorithm solving the system of nonlinear equations computing exact f and T z as well as k 1 and k 2 This initial guess is good enough for efficiently solving the equation system even though it is not linear

33. JASS `04 Benjamin Fingerle, Christian Wachinger33 Conclusion:Tsai-Method solves Camera Calibration Problem INPUT: Mono view image of non-coplanar calibration points of known coordinates in WCS Device specific data (resolution of CCD, image centre in pixels, number of pixels scanned in a line) OUTPUT: Extrinsic Parameters –Camera pose relatively to WCS [6DOF] Intrinsic Parameters –Effective focal length[1DOF] –Horizontal scale factor[1DOF] –Radial lens distortion coefficients[2DOF]

34. JASS `04 Benjamin Fingerle, Christian Wachinger34 Different Variations of Tsai’s Method Exist Different circumstances let different variations of Tsai’s method seem feasible: Single view with coplanar calibration points Single view with non-coplanar calibration points (presented) Multiple view

35. JASS `04 Benjamin Fingerle, Christian Wachinger35 Tsai’s Method Also Works for Stereovision Cameras Remark Camera tracking requires stereo vision images For stereovision two cameras are rigidly aligned in parallel

36. JASS `04 Benjamin Fingerle, Christian Wachinger36 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

37. JASS `04 Benjamin Fingerle, Christian Wachinger37 Virtual Camera Calibration (Optical-See-Through) Setup:

38. JASS `04 Benjamin Fingerle, Christian Wachinger38 Virtual Camera Calibration (Optical-See-Through) Calculation of a projective matrix describing the mapping from 3D Points to 2D Points in the image plane  No explicit calculation of intrinsic camera parameters  No consideration of distortion Using a 6 DOF Tracker to get the pose of the camera  Head motion can be modelled  Simplified algorithm for virtual camera calibration

39. JASS `04 Benjamin Fingerle, Christian Wachinger39 Virtual Camera Calibration (Optical-See-Through)

40. JASS `04 Benjamin Fingerle, Christian Wachinger40 Virtual Camera Calibration (Optical-See-Through) Calculation of matrix A: Using the relationship A = GF F: 4 x 4 transformation matrix G: 3 x 4 projection matrix F is determined by the tracker G has to be calculated

41. JASS `04 Benjamin Fingerle, Christian Wachinger41 Virtual Camera Calibration (Optical-See-Through) Calculation of matrix G: Choosing a single point with known coordinates p w Calculating the coordinates in the marker coordinate system p m ; p m = F p w Getting the point coordiante in the image plane p i by aligning the cross-hair with the real point p i = G p m 12 unknown parameters  At least 6 “calibration” points  A single “real” point is enough

42. JASS `04 Benjamin Fingerle, Christian Wachinger42 Virtual Camera Calibration (Optical-See-Through) Similar algorithm for stereoscopic displays Instead of using a cross-hair a 3D object is used

43. JASS `04 Benjamin Fingerle, Christian Wachinger43 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

44. JASS `04 Benjamin Fingerle, Christian Wachinger44 Image Calibration Calculation of distortion parameters for scan converter and frame grabber M p v = L p d

45. JASS `04 Benjamin Fingerle, Christian Wachinger45 Image Calibration Modeling of errors through linear transformations without rotation Calculation of transformation parameters by the comparison of the coordinates of certain points

46. JASS `04 Benjamin Fingerle, Christian Wachinger46 Agenda Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration

47. JASS `04 Benjamin Fingerle, Christian Wachinger47 AR Applications Create the Desire for Auto-Calibration Tracking assumes correct calibration of ceiling- or wall- mounted components Specialised methods for getting their parameters are necessary Goal: –Calibration of AR devices without user interaction –Calibration during regular use

48. JASS `04 Benjamin Fingerle, Christian Wachinger48 AR Applications Create the Desire for Auto-Calibration Regular method: –Estimating location of mobile units based on sighting data of known fixed units locations –Sightings may contain more information than necessary for location determination => surplus data –Constraining the locations of mobile units => additional surplus data Using surplus data for self-surveying!

49. JASS `04 Benjamin Fingerle, Christian Wachinger49 AR Applications Create the Desire for Auto-Calibration Three different data gathering methods for surplus data: –People –Floor –Frame Processing self-survey data: –Simulated Annealing Finding best guess Scoring solution against gathered data –Inverting the location algorithm

50. JASS `04 Benjamin Fingerle, Christian Wachinger50 Auto-Calibration of Cameras Drawbacks of Camera Calibration –Calibration grid is not available –Change of camera parameters due to Mechanical or thermal variations Focusing and zooming Auto-Calibration –highly flexible –requires point matches from image sequences

51. JASS `04 Benjamin Fingerle, Christian Wachinger51 AR Applications Create the Desire for Auto-Calibration There are many different self-calibration techniques –number of unknown or changing parameters –type of camera movement Possible problem: –Changing focal length –Rotating scene

52. JASS `04 Benjamin Fingerle, Christian Wachinger52 Conclusion Scenario Pointer calibration Object calibration Camera calibration Virtual Camera calibration Image calibration Auto-Calibration