1. CS 395/495-25: Spring 2004 IBMR: R 3  R 2 and P 3  P 2 : The Projective Camera Matrix Jack Tumblin jet@cs.northwestern.edu

2. Admin Reminders Projects Progress: Reading Week Signup…Projects Progress: Reading Week Signup… Can you explain how Debevec finds:Can you explain how Debevec finds: –Camera response curve? –One single unified floating-point image from multiple photos taken with different exposures? No? For next class, write down the steps you DO understand…No? For next class, write down the steps you DO understand…

3. Cameras Revisited Goal: Formalize projective 3D  2D mappingGoal: Formalize projective 3D  2D mapping Homogeneous coords handles infinities wellHomogeneous coords handles infinities well –Projective cameras ( convergent ‘eye’ rays) –Affine cameras (parallel ‘eye’ rays) –Composed, controlled as matrix product But First: Cameras as Euclidean R 3  R 2 :But First: Cameras as Euclidean R 3  R 2 : x’ = f x / z y’ = f y / z y y’ f z C Review Trucco & Verri: End of Chapter 2…

4. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) ycycycyc zczczczc xcxcxcxc

5. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f ycycycyc zczczczc xcxcxcxc

6. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal PointPrincipal Point Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc

7. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

8. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

9. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point P !?!?)Principal Plane (?!?! DOESN’T touch principle point P !?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

10. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C =[0,0,0] ycycycyc zczczczc xcxcxcxc p

11. Cameras Revisited Plenty of Terminology: Image Plane or Focal PlaneImage Plane or Focal Plane Focal Distance fFocal Distance f Camera Center CCamera Center C Principal Point pPrincipal Point p Principal AxisPrincipal Axis Principal Plane (?!?! DOESN’T touch principle point!?!?)Principal Plane (?!?! DOESN’T touch principle point!?!?) Camera Coords (x c,y c,z c )Camera Coords (x c,y c,z c ) Image Coords (x’,y’)Image Coords (x’,y’) f C ycycycyc zczczczc xcxcxcxc p

12. Cameras Revisited Goal: Formalize projective 3D  2D mappingGoal: Formalize projective 3D  2D mapping Homogeneous coords handles infinities well:Homogeneous coords handles infinities well: –Projective cameras ( convergent ‘eye’ rays) –Affine cameras (parallel ‘eye’ rays) –Composed, controlled as matrix product Recall Euclidian R 3  R 2 : x’ = f x / z y’ = f y / zRecall Euclidian R 3  R 2 : x’ = f x / z y’ = f y / z (Much Better: P 3  P 2 ) y y’ f z C Review Trucco & Verri: End of Chapter 2…

13. Most-Basic Camera Instead of x’ = f x / z (As in Trucco & Verri Chap. 2) y’ = f y / zInstead of x’ = f x / z (As in Trucco & Verri Chap. 2) y’ = f y / z try this 3x4 matrix:try this 3x4 matrix: Note! X is an augmented R 3 vector: ? OK for P 3 ?Note! X is an augmented R 3 vector: ? OK for P 3 ? ? is x in P 2 space? ? is x in P 2 space? As shown: image origin (x’,y’) is principal point p,As shown: image origin (x’,y’) is principal point p, but pixel counts usually start at an image corner… x’y’z’ f 0 0 0 0 f 0 0 0 0 1 0 xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p

14. Most Basic Camera P 0 Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix: To translate image origin (x’,y’) away from z c axis: Shift principal point p from (0,0) to (p x,p y )To translate image origin (x’,y’) away from z c axis: Shift principal point p from (0,0) to (p x,p y ) –does NOT use ‘homogeneous’ 1 term in X –is NOT obvious: scales z c (think in P 2 z c =x3 = 1,...) x’y’z’ xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p f 0 p x 0 0 f p y 0 0 0 1 0

15. Basic Camera P 0 : P 3  P 2 ( or camera R 3 ) Basic Camera P 0 is a 3x4 matrix:Basic Camera P 0 is a 3x4 matrix: Non-square pixels? change scaling (  x,  y )Non-square pixels? change scaling (  x,  y ) Parallelogram pixels? set nonzero skew sParallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” K matrix: “(internal) camera calib. matrix” xyz xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p  x f s p x 0 0  y f p y 0 0 0 1 0 K (3x3 submatrix) [K 0] = P 0 [K 0] = P 0

16. Complete Camera Matrix P K matrix: “internal camera calib. matrix”K matrix: “internal camera calib. matrix” R·T matrix: “external camera calib. matrix”R·T matrix: “external camera calib. matrix” –T matrix: Translate world to cam. origin –R matrix: 3D rotate world to fit cam. axes Combine: write (P 0 ·R·T)·X = x as P·X = x Output: x 2D Camera Image Input: X 3D World Space zyx X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space)

17. The Pieces of Camera Matrix P Columns of P matrix = image of world-space axes: p 1,p 2,p 3 == image of x,y,z axis vanishing pointsp 1,p 2,p 3 == image of x,y,z axis vanishing points –Direction D = [1 0 0 0] T = point on P 3 ’s x 1 axis, at inifinity –PD = 1 st column of P = P 1. Repeat for y and z axes. p 4 == image of the world-space origin pt.p 4 == image of the world-space origin pt. –Proof: let D = [0 0 0 1]T = direction to world origin –PD = 4 th column of P = image of origin pt. xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p1p1p1p1 p2p2p2p2 p3p3p3p3 p4p4p4p4 f C ycycycyc zczczczc xcxcxcxc p

18. The Pieces of Camera Matrix P Rows of P matrix: camera planes in world space row 1 = P 1T = image x-axis planerow 1 = P 1T = image x-axis plane row 2 = P 2T = image y-axis planerow 2 = P 2T = image y-axis plane row 1 = P 3T = camera’s principal planerow 1 = P 3T = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = P 1T P 2T P 3T f C ycycycyc zczczczc xcxcxcxc p

19. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = world plane whose image is x=0row 1 = P 1 = world plane whose image is x=0 row 2 = P 2 = world plane whose image is y=0row 2 = P 2 = world plane whose image is y=0 row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = P 1T P 2T P 3T Why? Recall that in P 3, point X is on plane  if and only if  T ·X = 0, so… f C ycycycyc zczczczc xcxcxcxc p

20. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = world plane whose image is x=0row 1 = P 1 = world plane whose image is x=0 row 2 = P 2 = world plane whose image is y=0row 2 = P 2 = world plane whose image is y=0 row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = P 1T P 2T P 3T f C ycycycyc zczczczc xcxcxcxc p

21. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image’s x=0 planerow 1 = P 1 = image’s x=0 plane row 2 = P 2 = image’s y=0 planerow 2 = P 2 = image’s y=0 plane Careful! Shifting the image origin by p x, p y shifts the x=0,y=0 planes!Careful! Shifting the image origin by p x, p y shifts the x=0,y=0 planes! xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

22. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image’s x=0 planerow 1 = P 1 = image’s x=0 plane row 2 = P 2 = image’s y=0 planerow 2 = P 2 = image’s y=0 plane row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p P 1T P 2T P 3T

23. The Pieces of Camera Matrix P Rows of P matrix: planes in world space row 1 = P 1 = image x-axis planerow 1 = P 1 = image x-axis plane row 2 = P 2 = image y-axis planerow 2 = P 2 = image y-axis plane row 3 = P 3 = camera’s principal planerow 3 = P 3 = camera’s principal plane – princip. plane P 3 = [p 31 p 32 p 33 p 34 ] T – its normal direction: [p 31 p 32 p 33 0 ] T –Why is it normal? It’s the world-space P 3 direction of the z c axis xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = f C ycycycyc zczczczc xcxcxcxc p Principal plane P 3 P 1T P 2T P 3T

24. The Pieces of Camera Matrix P Principal Axis Vector (z c ) in world space:Principal Axis Vector (z c ) in world space: –Normal of principal plane: = m 3 = [p 31 p 32 p 33 ] T –P 3 Scaling  Ambiguous direction!! +/- m 3 ? –Solution: use det(M)·m 3 as front of camera Principal Point p in image space:Principal Point p in image space: –image of (infinity point on z c axis = m 3 ) M·m 3 = p = x 0 (Zisserman book renames p as x 0 ) C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = p4p4p4p4 M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T x0x0x0x0 f

25. 1 The Pieces of Camera Matrix P Where is camera in world space? at C: Camera center C is at camera origin(x c,y c,z c )=0=CCamera center C is at camera origin(x c,y c,z c )=0=C Camera P transforms world-space point C to C=0Camera P transforms world-space point C to C=0 But how do we find C ? It is the Null Space of P : P C = C = 0 (solve for C. SVD works, but here’s an easier way:)But how do we find C ? It is the Null Space of P : P C = C = 0 (solve for C. SVD works, but here’s an easier way:) Camera’s Position in the World: C = xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or ~ -M -1 ·p 4 ~ ~ P =P =P =P = M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T f C ycycycyc zczczczc xcxcxcxc p p4p4p4p4 ~ ~ ~

26. Uses for Camera Matrix P Each world space R 3 point d = (x,y,z) defines a world-space P 3 direction D = (x,y,z,0). What is image point x d from direction D? x d = PD = [M | p 4 ] D = M d x d = PD = [M | p 4 ] D = M d (P 4 column has no effect, because of D’s zero): x d = M d or M -1 x d = d f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M xdxdxdxd ~ p4p4p4p4 zyx X (world space) d

27. Uses for Camera Matrix P Given image point x 0 and camera matrix P, Find ray X(  ) in world space through both: Slow, Obvious way: ‘Pseudo-invert’ P, apply to x 0 : Define pseudo-inverse P + as = P T (PP T ) -1 (note P·P + = I)Define pseudo-inverse P + as = P T (PP T ) -1 (note P·P + = I) Find a world-space point on ray: X 0 = P + x 0Find a world-space point on ray: X 0 = P + x 0 LIRP with world-space camera point C: X(  ) = C + (X 0 -C) LIRP with world-space camera point C: X(  ) = C + (X 0 -C)  f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M x0x0x0x0 X(  ) ~ p4p4p4p4 ~ ~ ~

28. Uses for Camera Matrix P Given image point x 0 and camera matrix P, Find ray X(  ) in world space through both: Better way: ray from C to point D 0 at infinity: Point x 0 is the image of (unknown) world-space direction D 0 = (x,y,z,0) T. Define a point d 0 = (x,y,z) T.Point x 0 is the image of (unknown) world-space direction D 0 = (x,y,z,0) T. Define a point d 0 = (x,y,z) T. Recall: we can find d 0 from the image: M -1 x 0 = d 0Recall: we can find d 0 from the image: M -1 x 0 = d 0 Recall: world-space camera position C = -M -1 p 4Recall: world-space camera position C = -M -1 p 4 X(  ) =  M -1 x 0 + C = M -1 (  x 0 – p 4 ) 0 1 1 0 1 1 f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = M x0x0x0x0 X(  ) ~ p4p4p4p4 ~ ~ ~

29. Uses for Camera Matrix P Given world-space point X 0, camera matrix P, Find camera depth z 0 : –X 0 = [x w, y w, z w, t w ] T seen thru camera P is X 0 ·P = x 0 = [x c, y c, 1] T ·w c X 0 ·P = x 0 = [x c, y c, 1] T ·w c –Then signed depth z 0 is: z 0 = w c sign(det M) t w || m 3 || f C ycycycyc zczczczc xcxcxcxc p xcxcycyczczcxcxcycyczczc xwxwywywzwzwtwtwxwxwywywzwzwtwtw = P P·X = x, or P =P =P =P = x0x0x0x0 X0X0X0X0 z0z0z0z0 p4p4p4p4 M m1Tm1Tm1Tm1T m2Tm2Tm2Tm2T m3Tm3Tm3Tm3T

30. Optional (Skipped): P = [K|0]·R·T = K [ R | -RC ] How can we separate K, R?P = [K|0]·R·T = K [ R | -RC ] How can we separate K, R? –Answer: K is triangular; use QR decomposition Cameras at Infinity:Cameras at Infinity: –Orthographic or ‘Parallel Projection’ Cameras –Transition to Orthographic: Weak Perspective projection camerasWeak Perspective projection cameras the ‘zoom’ lens (variable f) the ‘zoom’ lens (variable f) –Moving line-scan or ‘pushbroom’ cameras Translation Scan: aerial/sattelite camerasTranslation Scan: aerial/sattelite cameras Cylindrical Scan: panoramic camerasCylindrical Scan: panoramic cameras UNC ‘HiBall Tracker’: 6 tiny self-locating line-scan camerasUNC ‘HiBall Tracker’: 6 tiny self-locating line-scan cameras ~

31. Chapter 6 In Just One Slide: Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Given point correspondence sets (x i  X i ), How do you find camera matrix P ? (full 11 DOF) Surprise! You already know how ! DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0 -vectorize, stack, solve Ah = 0 for h vector -vectorize, stack, solve Ap = 0 for p vector More data  better results (at least 28 point pairs) (why so many? rule-of-thumb: #constraints = 5x #DOF = 55 = 27.5 point pairs)More data  better results (at least 28 point pairs) (why so many? rule-of-thumb: #constraints = 5x #DOF = 55 = 27.5 point pairs) Algebraic & Geometric Error, Sampson Error…Algebraic & Geometric Error, Sampson Error…

32. END

33. Camera Matrix P links P 3  P 2 Basic camera:Basic camera: x = P 0 X where P 0 = [K | 0] = World-space camera:World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P 0 ·R·T) X Rewrite as:Rewrite as: P = K[R | -RC] Redundant notation: P = [M p 4 ] M = RK p 4 = -K R CRedundant notation: P = [M | p 4 ] M = RK p 4 = -K R C ~ ~  x f s p x 0 0  y f p y 0 0 0 1 0 ~ Output: x 2D Camera Image Input: X 3D World Space z y x X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space) C  C ~

34. More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? 3-D Point:3-D Point: –Take any 3D world-space point X, –Coord. systems change: let X be X p = (x p,y p, 0); –Matrix P reduces to 3x3 matrix H in P 2 : x p = P·X p = THUS Any P 3 point becomes a P 2 pointTHUS Any P 3 point becomes a P 2 point xpyp0tpxpyp0tpxpyp0tpxpyp0tp xpyptpxpyptpxpyptpxpyptp h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = p 11 p 12 p 13 p 14 p 21 p 12 p 23 p 24 p 31 p 32 p 33 p 34

35. More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? 3-D Plane3-D Plane –Given any point X  on a plane  in P 3, –Coord. systems change: let X be X  = (x ,y , 0); –Matrix P reduces to 3x3 matrix H in P 2 : x  = P·X  = THUS Any P 3 plane becomes a P 2 planeTHUS Any P 3 plane becomes a P 2 plane xy0txy0txy0txy0t xytxytxytxyt h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = p 11 p 12 p 13 p 14 p 21 p 12 p 23 p 24 p 31 p 32 p 33 p 34

36. Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? 3D Directions and image Points:3D Directions and image Points: World-space P 3 direction D  image space point x d: Recall direction D = (x,y,z,0) (a point at infinity) sets a R 3 finite point d = (x,y,z). x d = PD = [M | p 4 ] D = M d x d = PD = [M | p 4 ] D = M d p 4 column has no effect, because of D’s zero; R ecall M = KR x d = M d M -1 x d = d More One-Camera Details f C ycycycyc zczczczc xcxcxcxc p xdxdxdxd ~ zyx X (world space) d D

37. More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? Forward Projection: Line / Ray in world  Line/Ray in image:Line / Ray in world  Line/Ray in image: – Ray in P 3 is: X(  ) = A +  B –Camera changes to P 2 : x(  ) = PA +  PB f C ycycycyc zczczczc xcxcxcxc p PA A BBBB

38. More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? Back Projection: Line L in image  Plane  L in world:Line L in image  Plane  L in world: –Recall: Line L in P 2 (a 3-vector): L = [x 1 x 2 x 3 ] T –Plane  L in P 3 (a 4-vector) is easy:  L = P T ·L = (Why? Look back at bits and pieces of P…)(Why? Look back at bits and pieces of P…) x1x2x3x1x2x3x1x2x3x1x2x3 p 11 p 21 p 31 p 12 p 22 p 32 p 13 p 23 p 33 p 14 p 24 p 34 f C ycycycyc zczczczc xcxcxcxc p L LLLL

39. Full 3x4 camera matrix P maps P 3 world to P 2 image ?What if the image plane moves? A) Translation: Given internal camera calibration K =Given internal camera calibration K = (x c, y c ) changes affect p x,p y and (z c ) changes affect the focal length f: let  = (f+t z )/f, then:(x c, y c ) changes affect p x,p y and (z c ) changes affect the focal length f: let  = (f+t z )/f, then: K’ = Focal length changeFocal length change Image coords changeImage coords change More One-Camera Details f c ycycycyc xcxcxcxc p zczczczc  x f s p x 0  y f p y 0 0 1  0 0 0  0 0 0 1  x f s (p x -t x ) 0  y f (p y -t y ) 0 0 1

40. Full 3x4 camera matrix P maps P 3 world to P 2 image ? What if the image plane moves? A) Translation: Given internal camera calibration K =Given internal camera calibration K = (x c, y c ) changes affect p x,p y and (z c ) changes affect the focal length f: let k = (f+t z )/f, then: K’ =(x c, y c ) changes affect p x,p y and (z c ) changes affect the focal length f: let k = (f+t z )/f, then: K’ = Define effect of K’ on image points x,x’; x = [K 0]X; x’ = [K’ | 0]XDefine effect of K’ on image points x,x’; x = [K | 0]X; x’ = [K’ | 0]X x’ = (K’ K -1 ) x More One-Camera Details f c ycycycyc xcxcxcxc p zczczczc  x f s p x 0  y f p y 0 0 1 k 0 0 0 k 0 0 0 1  x f s (p x -t x ) 0  y f (p y -t y ) 0 0 1

41. Full 3x4 camera matrix P maps P 3 world to P 2 image ?What if the image plane moves? B) Rotation: Given internal camera calibration K =Given internal camera calibration K = Rotate basic camera’s output: about its center C using 3D rotation matrix R (3x3): x = [K | 0]X; x’ = [KR | 0] X x’ = [KR(K -1 K) | 0] X = (KRK -1 ) [K|0] XRotate basic camera’s output: about its center C using 3D rotation matrix R (3x3): x = [K | 0]X; x’ = [KR | 0] X x’ = [KR(K -1 K) | 0] X = (KRK -1 ) [K|0] X Get new points x’ from old image pts x (K · R · K -1 ) x = x ’Get new points x’ from old image pts x (K · R · K -1 ) x = x ’ aka ‘conjugate rotation’; use this to construct planar panoramasaka ‘conjugate rotation’; use this to construct planar panoramas More One-Camera Details f c ycycycyc xcxcxcxc p zczczczc  x f s p x 0  y f p y 0 0 1 R

42. (K · R · K -1 ) x = x ’ (K’ K -1 ) x = x’ Full 3x4 camera matrix P maps P 3 world to P 2 image THUS: if the image plane moves: just rearranges the image points: a) Translations: b) Rotations: ALL cameras gather the same image content! Same Center Point? Same image, just rearranged! (just a planar reprojection in P 3 ) The camera center C must move to change the image content: no zooming, warping, rotation can change this! The camera center C must move to change the image content: no zooming, warping, rotation can change this! Gathering 3-D image data requires camera movement. More One-Camera Details

43. Cameras as Protractors Define world-space direction d:Define world-space direction d: –From a P 3 ideal point D = [x d y d z d 0] T define the ‘direction’ d == [x d y d z d ] D’s image point is x d = M d (recall M=KR)D’s image point is x d = M d (recall M=KR) Link direction d to image pt. x=(x c,y c,z c ): PX d = K[R | 0 ] X d = KR d = xLink direction d to image pt. x=(x c,y c,z c ): PX d = K[R | 0 ] X d = KR d = x Ray thru image pt. x has direction d = (KR) -1 xRay thru image pt. x has direction d = (KR) -1 x

44. Cameras as Protractors Angle between C and 2 image points x 1,x 2 (see book pg 199)Angle between C and 2 image points x 1,x 2 (see book pg 199) cos  = x 1 T (K -T K -1 ) x 2 (x 1 T (K -T K -1 ) x 1 )(x 2 T (K -T K -1 ) x 2 ) cos  = x 1 T (K -T K -1 ) x 2 (x 1 T (K -T K -1 ) x 1 )(x 2 T (K -T K -1 ) x 2 ) Image line L defines a plane  LImage line L defines a plane  L –(Careful! P 3 world =P 2 camera axes here!) –Plane normal direction: n = K T L C x2x2x2x2 x1x1x1x1 L  d2d2d2d2 d1d1d1d1 n

45. Cameras as Protractors Angle between C and 2 image points x 1,x 2 (see book pg 199)Angle between C and 2 image points x 1,x 2 (see book pg 199) cos  = x 1 T (K -T K -1 ) x 2 (x 1 T (K -T K -1 ) x 1 )(x 2 T (K -T K -1 ) x 2 ) cos  = x 1 T (K -T K -1 ) x 2 (x 1 T (K -T K -1 ) x 1 )(x 2 T (K -T K -1 ) x 2 ) Image line L defines a plane  LImage line L defines a plane  L –(Careful! P 3 world =P 2 camera axes here!) –Plane normal direction: n = K T L C x2x2x2x2 x1x1x1x1 L  d2d2d2d2 d1d1d1d1 n Something Special here? Yes !

46. Cameras as Protractors What is (K -T K -1 ) ? Recall P 3 Conic Weirdness:Recall P 3 Conic Weirdness: –Plane at infinity   holds all ‘horizon points’ D (‘wrapper around the universe’) –Absolute Conic   is imaginary outermost circle of   for ANY camera, Translation won’t change ‘Horizon point’ images:for ANY camera, Translation won’t change ‘Horizon point’ images: P D = x = KRd (pg200) Absolute conic is inside   ; it’s all ‘horizon points’Absolute conic is inside   ; it’s all ‘horizon points’ for ANY camera, (KR) -T   (KR) -1 =(K -T K -1 ) =  = ‘I mage of A bsolute C onic ’

47. Cameras as Protractors P (   ) = (K -T K -1 ). OK. Now what was   again? Absolute Conic   = all imaginary points found on  Absolute Conic   = all imaginary points found on   –Satisfies BOTH x 1 2 + x 2 2 + x 3 2 = 0 AND x 4 2 = 0 –Finds right angles-- if D 1  D 2, then: D 1 T ·   · D 2 = 0 (Recall D==[x,y,z,0]) Dual of absolute conic is Dual Quadric Q*  = all imaginary planes   to  , and tangent to Q* Dual of absolute conic is Dual Quadric Q*  = all imaginary planes   to  , and tangent to Q*  –Finds right angles-- if  1   2, then:  1 T · Q*  ·  2 = 0 –can write   or Q*  as the same matrix: 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 x1x1x2x2x30x30x1x1x2x2x30x30 x 1 x 2 x 3 0

48. Why do we care? P   = (K -T K -1 ) =  = ‘I mage of A bsolute C onic ’ IAC is a ‘magic tool’ for camera calibration KIAC is a ‘magic tool’ for camera calibration K Recall   let us find H from perp. lines.Recall   let us find H from perp. lines. Much better than ‘vanishing pt.’ methodsMuch better than ‘vanishing pt.’ methods With IAC, find P matrix from an image of just 3 or more (non-coplanar) squares…  Teo’s camera calibration methodWith IAC, find P matrix from an image of just 3 or more (non-coplanar) squares…  Teo’s camera calibration method

49. Summary: Cameras as Protractors World-Space Direction:D = [x c, y c, z c, 0] TWorld-Space Direction:D = [x c, y c, z c, 0] T Direction from image point x: d = (KR) -1 xDirection from image point x: d = (KR) -1 x Point C and points x 1,x 2 form angle  : (pg 199)Point C and points x 1,x 2 form angle  : (pg 199) (K -T K -1 ) =  = ‘I mage of A bsolute C onic ’ Line L makes plane  with normal n = K T L Line L makes plane  with normal n = K T L (in camera coords) cos  = x 1 T (K -T K -1 ) x 2 (x 1 T (K -T K -1 ) x 1 ) (x 2 T (K -T K -1 ) x 2 ) C x2x2x2x2 x1x1x1x1 L  D2D2D2D2 D1D1D1D1 n

50. Movement Detection? Can we do it from images only?Can we do it from images only? –2D projective transforms often LOOK like 3-D; –External cam. calib. affects all elements of P YES. Camera moved if-&-only-if Camera-ray points (C  x  X 1,X 2,…) willYES. Camera moved if-&-only-if Camera-ray points (C  x  X 1,X 2,…) will map to LINE (not a point) in the other image ‘Epipolar Line’ == l’ = image of L‘Epipolar Line’ == l’ = image of L ‘Parallax’ == x 1 ’  x 2 ’ vector‘Parallax’ == x 1 ’  x 2 ’ vector C’ C x L X1X1X1X1 X2X2X2X2 x2’x2’x2’x2’ x1’x1’x1’x1’ l’

51. OPTIONAL: Conics and Cameras

52. Chapter 7: More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? Conic C in image  Cone Quadric Q co in worldConic C in image  Cone Quadric Q co in world Q co = P T · C · P Tip of cone is camera center VTip of cone is camera center V f V ycycycyc zczczczc xcxcxcxc p C

53. Chapter 7: More One-Camera Details Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? Dual (line) Quadric Q* in world  Dual (line) Conic C* silhouette in imageDual (line) Quadric Q* in world  Dual (line) Conic C* silhouette in image C* = P T · Q* · P Works for ANY world quadric! sphere, cylinder, ellipsoid, paraboloid, hyperboloid, line, disk …Works for ANY world quadric! sphere, cylinder, ellipsoid, paraboloid, hyperboloid, line, disk … f V ycycycyc xcxcxcxc p zczczczc C* Q*

54. Full 3x4 camera matrix P maps P 3 world to P 2 image ? What does it do to basic 3D world shapes? World-space quadric Q  World-space view cone Q co, a degenerate quadric Q co = (V T QV)Q – (QV)(QV) T Chapter 7: More One-Camera Details f V ycycycyc xcxcxcxc p zczczczc Q Q co

55. END

56. Cameras as Protractors P   = (K -T K -1 ). OK. Now what was   again? Recall P 3 Conic Weirdness: (pg. 63-67) –Plane at infinity   holds all ‘horizon points’ d (‘universe wrapper’) –Absolute Conic   imaginary points in outermost circle of   Satisfies BOTH x 1 2 + x 2 2 + x 3 2 = 0 AND x 4 2 = 0Satisfies BOTH x 1 2 + x 2 2 + x 3 2 = 0 AND x 4 2 = 0 Can rewrite equations to look like a quadric (but isn’t— no x 4 )Can rewrite equations to look like a quadric (but isn’t— no x 4 ) AHA! ‘points’ on it are the (complex conjugate) directions d !AHA! ‘points’ on it are the (complex conjugate) directions d ! –Finds right angles-- if D 1  D 2, then: D 1 T ·   · D 2 = 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 = DT··D = DT··D = DT··D = DT··D x1x1x2x2x30x30x1x1x2x2x30x30 x 1 x 2 x 3 0

57. Cameras as Protractors P   = (K -T K -1 ). OK. Now what was   again? –Dual of Absolute Conic   is Dual Quadric Q*  (?!?!) –More compact notation: for imaginary planes  –Same matrix, but different use: --find a plane  for every possible direction d --  is  to  , and tangent to the quadric Q*  –   is circle in   where tangent planes  are  to   –Finds right angles-- if  1   2, then:  1 T · Q*  ·  2 = 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 =  T · Q*  ·  =  T · Q*  ·  1122343411223434  1  2  3  4 Inconsistent notation!

58. Cameras as Protractors P   = (K -T K -1 ) =  = ‘I mage of A bsolute C onic ’ Just   as has a dual Q* ,  has dual  * :Just   as has a dual Q* ,  has dual  * :  * =  -1 = K K T  * =  -1 = K K T The dual conic  * is the image of Q* , soThe dual conic  * is the image of Q* , so  * = P Q*  = P = 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 st 3 columns of P, then a column of 0

59. Cameras as Protractors P (   ) = (K -T K -1 ) =  = ‘I mage of A bsolute C onic ’ Just   as has a dual Q* ,  has dual  * :Just   as has a dual Q* ,  has dual  * :  * =  -1 = K K T  * =  -1 = K K T The dual conic  * is the image of Q* , soThe dual conic  * is the image of Q* , so  * = P (Q*  ) Vanishing points v 1,v 2 of 2  world-space lines: v 1 T  v 2 = 0 Vanishing lines L 1, L 2 of 2  world-space planes: L 1 T  * L 2 = 0 (first 3 columns of P)

60. Cameras as Protractors Clever vanishing point trick: Perpendicular lines in image?Perpendicular lines in image? Find their vanishing pts. by construction:Find their vanishing pts. by construction: Use v 1 T  v 2 = 0, stack, solve for  = (K -T K -1 )Use v 1 T  v 2 = 0, stack, solve for  = (K -T K -1 ) v1v1v1v1 v2v2v2v2 v3v3v3v3

61. Cameras as Protractors P   = (K -T K -1 ) =  = ‘I mage of A bsolute C onic ’ Just   as has a dual Q* ,  has dual  * :Just   as has a dual Q* ,  has dual  * :  * =  -1 = K K T  * =  -1 = K K T The dual conic  * is the image of Q* , soThe dual conic  * is the image of Q* , so  * = P (Q*  ) = P( ) Vanishing points v 1,v 2 of 2  world-space lines: v 1 T  v 2 = 0Vanishing points v 1,v 2 of 2  world-space lines: v 1 T  v 2 = 0 Vanishing lines L 1, L 2 of 2  world-space planes: L 1 T  * L 2 = 0Vanishing lines L 1, L 2 of 2  world-space planes: L 1 T  * L 2 = 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0

62. END