1. CS 395/495-26: Spring 2004 IBMR: Projective 2D DLT Jack Tumblin jet@cs.northwestern.edu

2. x’ Apply the 3x3 matrix H x’ = Hx Apply the 3x3 matrix H x’ = Hx Recall: Projective Transform H x3x3x3x3 x2x2x2x2 x1x1x1x1 x y x’ y’ (x/w,y/w) x (x’/w’,y’/w’) Homog. coords x’ = [x’,y’,w’] T w’=1  2D image (x’,y’) 2D image (x,y)  Homog. coords [x,y,w] T = x w=1

3. Recall: Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H defines point pairs (x,x’) (correspondences) x’ = = y’ = = = xyw Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem… x = [x 1, x 2, x 3 ] notation is confusing for multiple points! is confusing for multiple points! Instead, let’s use [x,y,w]: (so we can use subscripts to number the point pairs x,x’…) to number the point pairs x,x’…)

4. Recall: Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H defines point pairs (x,x’) (correspondences) x’ = = y’ = = = xyw Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem… let w=1

5. Recall: Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 Finding H from point pairs (correspondences) x’ = = y’ = = = xy1 Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem…

6. Naïve Method: 4 point correspondence: Book shows: Rearrange: known vector (dot) unknown vector stack, solve for null space… But this assumes x 1 =x, x 2 =y, x 3 =w=1, There is a better way (DLT)… x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [x y 1 0 0 0 -x’x -x’y -x’] = 0 [0 0 0 x y 1 -y’x -y’y -y’]

7. Direct Linear Tranform (DLT) Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 always) compare:compare: A BETTER WAY: DLT method: Solve Hx  x’ = 0A BETTER WAY: DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1 OK…) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

8. Direct Linear Tranform (DLT) Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 always) Compare to DLT:Compare to DLT: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1, x 3 ’=w’  1) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [ 0 T -w’ i x i T y’ I x i T ] [ w’ i x i T 0 T -x’ i x i T ]

9. Direct Linear Tranform (DLT) DLT method: Solve H  x = 0DLT method: Solve H  x = 0 –Accepts any scale, any point (x 3 =w  1 is OK) –‘Pure’, Compatible –uses only P 2 terms. –Much better suited to error measurements. Subtlety:Subtlety: –Won’t divide-by-zero if w=0 or h 33 =0 (it happens!) –has a 3 rd row; it is not degenerate if/when w’=0 –OK to use it… (Solve 8x12) h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 0 T -w’ i x i T y’ I x i T 0 T -w’ i x i T y’ I x i T w’ i x i T 0 T -x’ i x i T =0 w’ i x i T 0 T -x’ i x i T =0 y’ i x i T -x’ i x i T 0 T y’ i x i T -x’ i x i T 0 T

10. STOP: Learn about SVDs …

11. Matrix Math & SVDs Matrix multiply is a change of coordinatesMatrix multiply is a change of coordinates –Rows of A = coordinate axis vectors –Ax = a dot product for each axis input space  output space input space  output space SVD ‘factors’ matrix A into 3 parts:SVD ‘factors’ matrix A into 3 parts: –U columns: orthonormal axes for OUTPUT –V columns: orthonormal axes for INPUT –S: scale factors V  U svd(A) = U S V T

12. Actual ‘Robustness’ Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics… Key to DLT formulation: (Hx  x’) = 0Key to DLT formulation: (Hx  x’) = 0 Rearrange as dot product: (known)·(unknown) = 0Rearrange as dot product: (known)·(unknown) = 0 Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… (pg 75) tricky when you mix types: points, lines,… (pg 75) measured input transformed  measured output measured output = 0

13. Deceptions: false ‘Robustness’… Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences: –Use DLT to write 8x9 (or 12x9) matrix A: Ah=0 –Solve for h null space. ALWAYS gives H matrix But what if points are bad / fictional?But what if points are bad / fictional? –3 collinear input pts, crooked out: IMPOSSIBLE! –Yet we get an H solution! Why? A matrix rank is 7 or 6  rank 1 H result(s)A matrix rank is 7 or 6  rank 1 H result(s) –‘Null Space’ of A may contain >1 H solution! –‘Degenerate’ H solution of form aH 1 + bH 2 … Answer: SVD ranks A; reject bad point sets.Answer: SVD ranks A; reject bad point sets.

14. Most Important SVD Ideas SVD solves Null-Space problems Ax = 0 (‘flatten, stack, find null space’ to solve) –MANY problems can be written this way –Expandable! --more data points  tall, ‘overconstrained’ A matrix --SVD finds optimal ‘least squares’ solution. Robustness: bane of all inverse methods Accuracy vs. # of measurements (more isn’t always better! Outliers...) Data-rich images: quantity easier than quality more is easier than better) ?What links measurement errors  H errors? ?How can more measurements reduce error?

15. How else can we use DLT? I know this building consists of MANY parallel and perpendicular lines, corners, regularly spaced points.I know this building consists of MANY parallel and perpendicular lines, corners, regularly spaced points. How many different ways can we find to ‘rectify’ this image to get this one?How many different ways can we find to ‘rectify’ this image to get this one?

16. Direct Linear Tranform (DLT) Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0 –Uses constant scale (x 3 =w=1 in measurement) Compare to DLT:Compare to DLT: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 – (a different identity: x’  x’ = 0) –Accepts any scale (x 3 =w  1 OK, xx 3 ) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

17. Direct Linear Tranform (DLT) Naïve method: Solve Hx – x’ = 0Naïve method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 always) compare:compare: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1, x 3 ’=w’  1) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [ 0 T -w’ i x i T y’ I x i T ] [ w’ i x i T 0 T -x’ i x i T ]

18. Direct Linear Tranform (DLT) DLT method: Solve H  x = 0DLT method: Solve H  x = 0 –Accepts any scale, any point (x 3 =w  1 is OK) –‘Pure’ and Compatible -- P 2 terms only, no R 2 –Much better suited to error measurements. Subtlety:Subtlety: –Won’t divide-by-zero if w=0 or h 33 =0 (it happens!) –has a 3 rd row; it is not degenerate if w’=0 –OK to use it… (Solve 8x12) h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 0 T -w’ i x i T y’ I x i T 0 T -w’ i x i T y’ I x i T w’ i x i T 0 T -x’ i x i T =0 w’ i x i T 0 T -x’ i x i T =0 -y’ i x i T -x’ i x i T 0 T -y’ i x i T -x’ i x i T 0 T

19. Adding More Measurements How can we use >4 point correspondences? Easy:Easy: –Add more rows to our 8x9 matrix A: A·h = 0 –Use SVD to find Null space (Always gives an answer!) –Result: ‘Least squares’ solution; error=A·h =  minimizes || A h || 2 =  i  2 i where,  i is error for i-th pt. correspondence:minimizes || A h || 2 =  i  2 i where,  i is error for i-th pt. correspondence:  i = || Hx i  x i ’ || 2 = || (2 rows of A)*h || 2 = ‘algebraic distance’  i = || Hx i  x i ’ || 2 = || (2 rows of A)*h || 2 = ‘algebraic distance’ –‘Algebraic Distance’ ? No geometric meaning!

20. Adding More Measurements 2D‘Algebraic Distance’ ? No geometric meaning!2D‘Algebraic Distance’ ? No geometric meaning! 2D ‘Geometric Distance’ d(a,b) 2 is Better: measurable length in input or output space2D ‘Geometric Distance’ d(a,b) 2 is Better: measurable length in input or output space if a = (a 1 a 2 a 3 ) and b = (b 1 b 2 b 3 ), then define Turns out that: ( - ) 2 + ( - ) 2 d(a,b) 2 = d(a,b) 2 = a1a1a3a3a1a1a3a3 b2b2b3b3b2b2b3b3 b1b1b3b3b1b1b3b3 a2a2a3a3a2a2a3a3 d(a,b) 2 = d algebraic (a,b) d(a,b) 2 = d algebraic (a,b) a3·b3a3·b3a3·b3a3·b3 (Not very surprising)

21. Adding More Measurements Iterative Strategy: Overconstrain the answer HOverconstrain the answer H –Collect extra measurements (>4 point pairs, etc. …) –expect errors; call them ‘estimates’ x Compute a 1 st solution (probably by SVD)Compute a 1 st solution (probably by SVD) Compute error d(Hx, x’) 2, and use this to…Compute error d(Hx, x’) 2, and use this to… ‘Tweak’ answer H and estimates x‘Tweak’ answer H and estimates x Compute new answerCompute new answer Stop when error < useful thresholdStop when error < useful threshold ^ ^ ^ ^

22. Using Estimates Simplest: ‘one image’ transfer method:Simplest: ‘one image’ transfer method: –Assume inputs are a perfect test pattern: only output pts are in error –Adjust output estimates x’ until d(Hx, x’) 2  0 (note we re-compute H as x’ changes) Better: ‘Symmetric’ transfer method:Better: ‘Symmetric’ transfer method: –Assume BOTH input x and output x’ has error. –Adjust BOTH input and output estimates x x’ (note we also re-compute H as x, x’ change) –Stop when d(Hx,x’) 2 + d(H -1 x’, x) 2 is nearly 0 ^^ ^^ ^ ^ ^ ^

23. END