1. CS 395/495-25: Spring 2003 IBMR: Week 7A The Camera Matrix Continued... Jack Tumblin jet@cs.northwestern.edu

2. IBMR-Related Seminars 3D Scanning for Cultural Heritage Applications 3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson 3D Scanning for Cultural Heritage Applications Friday May 16 3:00pm, Rm 381, CS Dept. no title yet... no title yet... Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.

3. Reminders ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110 ProjB graded: Good! minor H confusions...ProjB graded: Good! minor H confusions... MidTerm graded:MidTerm graded: ProjC posted tonight; Due next Friday, May 15ProjC posted tonight; Due next Friday, May 15 Start Watson’s Late Policy: Grade -(3 n ) points; n=# of class meetings lateStart Watson’s Late Policy: Grade -(3 n ) points; n=# of class meetings late ProjD coming Thurs May 15, due Thurs May 29ProjD coming Thurs May 15, due Thurs May 29 Take-Home Final Exam: Thurs June 5, due June 11Take-Home Final Exam: Thurs June 5, due June 11

4. Basic Cameras Revisited f C ycycycyc zczczczc xcxcxcxc p Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal Axis z cPrincipal Axis z c Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’)

5. Recall Basic Camera P 0 : Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix: To translate image origin (x’,y’) away from z c axis: Shift principal point p from (0,0) to (p x,p y )To translate image origin (x’,y’) away from z c axis: Shift principal point p from (0,0) to (p x,p y ) –does NOT use ‘homogeneous’ 1 term in X –is NOT obvious: scales z c (think in P 2 z c =x3 = 1,...) x’y’z’ xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p f 0 p x 0 0 f p y 0 0 0 1 0

6. Basic Camera P 0 : P 3  P 2 ( or camera R 3 ) Basic Camera P 0 is a 3x4 matrix:Basic Camera P 0 is a 3x4 matrix: Non-square pixels? change scaling (  x,  y )Non-square pixels? change scaling (  x,  y ) Parallelogram pixels? set nonzero skew sParallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” K matrix: “(internal) camera calib. matrix” xyz xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p  x f s p x 0 0  y f p y 0 0 0 1 0 K (3x3 submatrix) [K 0] = P 0 [K 0] = P 0

7. Complete Camera Matrix P K matrix: “internal camera calib. matrix”K matrix: “internal camera calib. matrix” R·T matrix: “external camera calib. matrix”R·T matrix: “external camera calib. matrix” –T matrix: Translate world origin to cam. origin, –R matrix: then Rotate world to fit cam. axes Combine: write (P 0 ·R·T)·X = x as P·X = x Output: x P 2 Camera Image Input: X P 3 World Space zyx X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space)

8. The Pieces of Camera Matrix P Columns of P matrix = image of world-space axes: p 1,p 2,p 3 == image of x,y,z axis vanishing pointsp 1,p 2,p 3 == image of x,y,z axis vanishing points –Direction D = [1 0 0 0] T = point on P 3 ’s x 1 axis, at inifinity –PD = 1 st column of P = P 1. Repeat for y and z axes. p 4 == image of the world-space origin pt.p 4 == image of the world-space origin pt. –Proof: let D = [0 0 0 1]T = world origin –PD = 4 th column of P = image of origin pt. xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p1p1p1p1 p2p2p2p2 p3p3p3p3 p4p4p4p4 f C ycycycyc zczczczc xcxcxcxc p

9. The Pieces of Camera Matrix P Rows of P matrix: camera planes in world space row 1 = P 1T = image x-axis planerow 1 = P 1T = image x-axis plane row 2 = P 2T = image y-axis planerow 2 = P 2T = image y-axis plane row 1 = P 3T = camera’s principal planerow 1 = P 3T = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = P 1T P 2T P 3T f C ycycycyc zczczczc xcxcxcxc p

10. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane row 1 = P 3 = camera’s principal planerow 1 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

11. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane row 1 = P 3 = camera’s principal planerow 1 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

12. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane Careful! Shifting the image origin by p x, p y shifts the x,y axis planes!Careful! Shifting the image origin by p x, p y shifts the x,y axis planes! xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

13. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

14. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane – princip. plane P 3 = [p 31 p 32 p 33 p 34 ] T – its normal direction: [p 31 p 32 p 33 0 ] T –Why is it normal? It’s the world-space P 3 direction of the z c axis BOOK has yet another inconsistent notation for this..... xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p Principal plane p 3 P 1T P 2T P 3T

15. The Pieces of Camera Matrix P Principal Axis Vector (z c ) in world space:Principal Axis Vector (z c ) in world space: –Normal of principal plane: = m 3 = [p 31 p 32 p 33 ] T –P 3 Scaling  Ambiguous direction!! +/- m 3 ? –Solution: use det(M)·m 3 as front of camera Principal Point p in image space:Principal Point p in image space: –image of (infinity point on z c axis = m 3 ) M·m 3 = p = x 0 (book renames p as x 0 ) C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p4p4p4p4 M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T x0x0x0x0 f

16. 0 1 The Pieces of Camera Matrix P Where is the Camera? Camera center C is at camera origin(x c,y c,z c )=0=CCamera center C is at camera origin(x c,y c,z c )=0=C What point C will the camera P transform to C=0 ? Answer: C is the Null Space of P : P C = C = 0 (solve for C. SVD works, but here’s an easier way:)What point C will the camera P transform to C=0 ? Answer: C is the Null Space of P : P C = C = 0 (solve for C. SVD works, but here’s an easier way:) Finite Camera: C = Affine Camera: C = xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or ~ ~ -M -1 ·p 4 d (where Md=0) ~ ~ P =P =P =P = M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T f C ycycycyc zczczczc xcxcxcxc p p4p4p4p4

17. Uses for Camera Matrix P Each world space R 3 point d = (x,y,z) defines a world-space P 3 direction D = (x,y,z,0). What is the image point x d from direction D? x d = PD = [M | p 4 ] D = M d x d = PD = [M | p 4 ] D = M d (P 4 column has no effect, because of D’s zero): x d = M d or M -1 x d = d f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M xdxdxdxd ~ p4p4p4p4 zyx X (world space) d

18. Uses for Camera Matrix P Given image point x 0 and camera matrix P, Find ray X(  ) in world space through both: Slow, Obvious way: ‘Pseudo-invert’ P, apply to x 0 : Define pseudo-inverse P + as = P T (PP T ) -1 (note P·P + = I)Define pseudo-inverse P + as = P T (PP T ) -1 (note P·P + = I) Find a world-space point on ray: X 0 = P + x 0Find a world-space point on ray: X 0 = P + x 0 LIRP with world-space camera point C: X(  ) = C + (X 0 -C) LIRP with world-space camera point C: X(  ) = C + (X 0 -C)  f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M x0x0x0x0 X(  ) ~ p4p4p4p4 ~ ~ ~ ~

19. Uses for Camera Matrix P Given image point x 0 and camera matrix P, Find ray X(  ) in world space through both: Better way: ray from C to point D 0 at infinity: Point x 0 is the image of (unknown) world-space direction D 0 = (x,y,z,0) T. Define a point d 0 = (x,y,z) T.Point x 0 is the image of (unknown) world-space direction D 0 = (x,y,z,0) T. Define a point d 0 = (x,y,z) T. Recall: we can find d 0 from the image: M -1 x 0 = d 0Recall: we can find d 0 from the image: M -1 x 0 = d 0 Recall: world-space camera C = -M -1 p 4Recall: world-space camera C = -M -1 p 4 X(  ) =  M -1 x 0 + C = M -1 (  x 0 – p 4 ) 0 1 1 0 1 1 f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M x0x0x0x0 X(  ) ~ p4p4p4p4 ~ ~ ~

20. Uses for Camera Matrix P Given world-space point X 0, camera matrix P, Find camera depth z 0 : –X 0 = [x w, y w, z w, t w ] T seen thru camera P is X 0 ·P = x 0 = [x c, y c, 1] T ·w c X 0 ·P = x 0 = [x c, y c, 1] T ·w c –Then signed depth z 0 is: z 0 = w c sign(det M) t w || m 3 || f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = x0x0x0x0 X0X0X0X0 z0z0z0z0 p4p4p4p4 M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T

21. Skipped: P = [K|0]·R·T How can we separate K, R, T?P = [K|0]·R·T How can we separate K, R, T? –Answer: K is triangular; use QR decomposition Cameras at Infinity:Cameras at Infinity: –Orthographic or ‘Parallel Projection’ Cameras –Transition to Orthographic: Weak Perspective projection camerasWeak Perspective projection cameras the ‘zoom’ lens (variable f) the ‘zoom’ lens (variable f) –Moving line-scan or ‘pushbroom’ cameras Translation Scan: aerial/sattelite camerasTranslation Scan: aerial/sattelite cameras Cylindrical Scan: panoramic camerasCylindrical Scan: panoramic cameras UNC ‘HiBall Tracker’: 6 tiny self-locating line-scan camerasUNC ‘HiBall Tracker’: 6 tiny self-locating line-scan cameras

22. Chapter 6 In Just One Slide: Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Surprise! You already know how ! DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0 -vectorize, stack, solve Ah = 0 for h vector -vectorize, stack, solve Ap = 0 for p vector -Normalizing step removes origin dependence More data  better results (at least 28 point pairs)More data  better results (at least 28 point pairs) Algebraic & Geometric Error, Sampson Error…Algebraic & Geometric Error, Sampson Error…

23. END