1. CS 395/495-25: Spring 2003 IBMR: Week 6B R 3  R 2 and P 3  P 2 The Camera Matrix Jack Tumblin jet@cs.northwestern.edu

2. Reminders ProjA graded: Good Job!ProjA graded: Good Job! ProjB being graded...ProjB being graded... MidTerm Due today, Thurs May 8: Solutions!MidTerm Due today, Thurs May 8: Solutions! Watson’s Late Policy: Grade -(3 n ) points; n=# of class meetings lateWatson’s Late Policy: Grade -(3 n ) points; n=# of class meetings late ProjC posted tonight; Due next Thurs, May 15ProjC posted tonight; Due next Thurs, May 15 ProjD coming Thurs May 15, due Thurs May 29ProjD coming Thurs May 15, due Thurs May 29 Take-Home Final Exam coming June 5, dueTake-Home Final Exam coming June 5, due

3. Cameras Revisited Goal: Formalize projective 3D  2D mappingGoal: Formalize projective 3D  2D mapping Homogeneous coords handles infinities wellHomogeneous coords handles infinities well –Projective cameras ( convergent ‘eye’ rays) –Affine cameras (parallel ‘eye’ rays) –Composed, controlled as matrix product But First: Cameras as Euclidean R 3  R 2 :But First: Cameras as Euclidean R 3  R 2 : x’ = f x / z y’ = f y / z y y’ f z C

4. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) ycycycyc zczczczc xcxcxcxc

5. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f ycycycyc zczczczc xcxcxcxc

6. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc

7. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

8. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

9. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point P !?!?)Principal Plane (?!?! DOESN’T touch principle point P !?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

10. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

11. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

12. Homogeneous Coords: R 3  P 2 Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix: Tricky! X is in augmented R 3, not P 3 (yet)Tricky! X is in augmented R 3, not P 3 (yet) x is in P 2 space x is in P 2 space As shown: origin of (x’,y’) is principal point p,As shown: origin of (x’,y’) is principal point p, but pixel counting starts at corners… x’y’z’ f 0 0 0 0 f 0 0 0 0 1 0 xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p

13. Homogeneous Coords: R 3  P 2 Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix: Translate image coords (x’,y’): Shifts principal point p from (0,0) to (p x,p y )Translate image coords (x’,y’): Shifts principal point p from (0,0) to (p x,p y ) –is NOT obvious: scales z c –does NOT use ‘homogeneous’ 1 term in X x’y’z’ xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p f 0 p x 0 0 f p y 0 0 0 1 0

14. Homogeneous Coords: R 3  P 2 Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix: Less common adjustments: Non-square pixels? change scaling (  x,  y )Non-square pixels? change scaling (  x,  y ) Parallelogram pixels? set nonzero skew sParallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” K matrix: “(internal) camera calib. matrix” xyz xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p  x f s p x 0 0  y f p y 0 0 0 1 0 K (3x3 submatrix)

15. Matrix Form: Euclidean R 3  R 2 K matrix: “internal camera calib. matrix”K matrix: “internal camera calib. matrix” R·T matrix: “external camera calib. matrix”R·T matrix: “external camera calib. matrix” –T matrix: Translate world to cam. origin –R matrix: 3D rotate world to fit cam. axes Combine: write (P 0 ·R·T)·X = x as P·X = x Output: x 2D Camera Image Input: X 3D World Space zyx X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space)

16. END

17. 0 1 Uses for Camera Matrix P P’s Null Space: camera center C in world space! P C = 0 (solve for C. SVD works, but answer matches these:)P’s Null Space: camera center C in world space! P C = 0 (solve for C. SVD works, but answer matches these:) –Finite Camera: C = Affine Camera: C = Surprise! (minor) works projectively too: P 3  P 2Surprise! (minor) works projectively too: P 3  P 2 (see book pg. 152-3 for restrictions—almost none!) xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or ~~ P =P =P =P = M p4p4p4p4 -M -1 ·p 4 d (where Md=0) ~ ~

18. Uses for Camera Matrix P Columns of P matrix: Points in image-space: P 1,P 2,P 3 == image of x,y,z axis vanishing pointsP 1,P 2,P 3 == image of x,y,z axis vanishing points –Proof: let D = [1 0 0 0] T = point on x axis, at inifinity –PD = 1 st column of P. Repeat for y and z axes P 4 == image of the world-space origin pt.P 4 == image of the world-space origin pt. –Proof: let D = [0 0 0 1]T = world origin –PD = 4 th column of P = image of origin pt. xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = P1P1P1P1 P2P2P2P2 P3P3P3P3 f C ycycycyc zczczczc xcxcxcxc p P4P4P4P4

19. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1T = image x-axis planerow 1 = p 1T = image x-axis plane row 2 = p 2T = image y-axis planerow 2 = p 2T = image y-axis plane row 1 = p 3T = camera’s principal planerow 1 = p 3T = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p

20. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1 = image x-axis planerow 1 = p 1 = image x-axis plane row 2 = p 2 = image y-axis planerow 2 = p 2 = image y-axis plane row 1 = p 3 = camera’s principal planerow 1 = p 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p

21. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1 = image x-axis planerow 1 = p 1 = image x-axis plane row 2 = p 2 = image y-axis planerow 2 = p 2 = image y-axis plane row 1 = p 3 = camera’s principal planerow 1 = p 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p

22. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1 = image x-axis planerow 1 = p 1 = image x-axis plane row 2 = p 2 = image y-axis planerow 2 = p 2 = image y-axis plane Careful! Shifting image originCareful! Shifting image origin shifts the x,y axis planes! xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p

23. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1 = image x-axis planerow 1 = p 1 = image x-axis plane row 2 = p 2 = image y-axis planerow 2 = p 2 = image y-axis plane row 3 = p 3 = camera’s principal planerow 3 = p 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p

24. Uses for Camera Matrix P Rows of P matrix: planes in world space row 1 = p 1 = image x-axis planerow 1 = p 1 = image x-axis plane row 2 = p 2 = image y-axis planerow 2 = p 2 = image y-axis plane row 3 = p 3 = camera’s principal planerow 3 = p 3 = camera’s principal plane – princip. plane p 3 = [p 31 p 32 p 33 p 34 ] T – its normal direction: [p 31 p 32 p 33 0 ] T –Why is it normal? It’s the world-space endpoint of z c axis (at infinity ) xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p 1T p 2T p 3T f C ycycycyc zczczczc xcxcxcxc p Principal plane p 3

25. Chapter 6 In Just One Slide: Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Surprise! You already know how ! DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0 -vectorize, stack, solve Ah = 0 for h vector -vectorize, stack, solve Ap = 0 for p vector -Normalizing step removes origin dependence More data  better results (at least 28 point pairs)More data  better results (at least 28 point pairs) Algebraic & Geometric Error, Sampson Error…Algebraic & Geometric Error, Sampson Error…