1. CS 395/495-26: Spring 2003 IBMR: Week 4A from Chapter 3: Point Correspondence and Direct Linear Transform (DLT) Jack Tumblin jet@cs.northwestern.edu

2. Proj_A Workaround Programming ‘Pick & Drag’ is messy!Programming ‘Pick & Drag’ is messy! KeyDown, KeyUp is easy (see ProjA_06)KeyDown, KeyUp is easy (see ProjA_06) We have only a few points to drag; –Use number keys to select points: ‘1’ for 1 st point, ‘2’ for 2 nd point, etc.‘1’ for 1 st point, ‘2’ for 2 nd point, etc. ‘SHIFT 1’ for 1 st line, ‘SHIFT 2’ for 2 nd line...‘SHIFT 1’ for 1 st line, ‘SHIFT 2’ for 2 nd line... –Use arrow keys to change point / line coords: left/right for x1 value,left/right for x1 value, up/down for x2 value,up/down for x2 value, SHIFT up/down for x3 value, etc.SHIFT up/down for x3 value, etc.

3. Most Important Ideas SVD solves Null-Space problems Ax = 0 (‘flatten, stack, find null space’ to solve) –MANY problems can be written this way –Expandable! --more data points  tall, ‘overconstrained’ A matrix --SVD finds optimal ‘least squares’ solution. Robustness: bane of all inverse methods Accuracy vs. # of measurements (more isn’t always better! Outliers...) Data-rich images: quantity easier than quality more is easier than better) ?What links measurement errors  H errors? ?How can more measurements reduce error?

4. Recall: Rectification Undo parts of H x’ = H xwhere H = H S H A H P GOAL: Put world plane x’ into view plane x –Affine Rect.; (find only H P (2DOF)); –Metric Rect.;(find H A and H P (6DOF)); –Full Rect.;(find all: H S H A H P (8DOF)); METHODS: 1.Affine: ‘Vanishing Point’, Horizon line methods 2.Metric: Conics & Circular Points (!?!?) 3.Full: 4-point correspondence

5. Finding H from Image Space –Affine Rect; (find only H P (2DOF)); 2 parallel line pairs  2 points on horizon line –Metric Rect.(find H A and H P (4DOF)); 5 perpendicular line pairs  C*  ’ –Full Rect.;(find all: H S H A H P (8DOF)); 4 (or more point pairs)  all H coeffs 2D world space 2D image space H S H A H P L M m L

6. x’ Apply the 3x3 matrix H x’ = Hx Apply the 3x3 matrix H x’ = Hx Recall: Projective Transform H x3x3x3x3 x2x2x2x2 x1x1x1x1 x y x’ y’ (x/w,y/w) x (x’/w’,y’/w’) Homog. coords x’ = [x’,y’,w’] T w’=1  2D image (x’,y’) 2D image (x,y)  Homog. coords [x,y,w] T = x w=1

7. Recall: Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H defines point pairs (x,x’) (correspondences) x’ = = y’ = = = xyw Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem… let w=1

8. Naïve Method: (pg 13-14) 4 point correspondence: Book shows: Rearrange: known vector (dot) unknown vector stack, solve for null space… But this assumes x 1 =x, x 2 =y, x 3 =w=1, There is a better way (DLT)… x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [x y 1 0 0 0 -x’x -x’y -x’] = 0 [0 0 0 x y 1 -y’x -y’y -y’]

9. Direct Linear Tranform (DLT) Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0 –Requires constant scale (x 3 =1 in measurement) compare:compare: include the w, w’ terms:include the w, w’ terms: –look what happens when w’  0! [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

10. Direct Linear Tranform (DLT) Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0 –Requires constant scale (x 3 =1 in measurement) compare:compare: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 – (a different identity: x’  x’ = 0) –Accepts any scale (x 3 =w  1 OK, xx 3 ) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

11. Direct Linear Tranform (DLT) Naïve method: Solve Hx – x’ = 0Naïve method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 always) compare:compare: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1, x 3 ’=w’  1) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [ 0 T -w’ i x i T y’ I x i T ] [ w’ i x i T 0 T -x’ i x i T ]

12. Direct Linear Tranform (DLT) DLT method: Solve H  x = 0DLT method: Solve H  x = 0 –Accepts any scale, any point (x 3 =w  1 is OK) –‘Pure’ and Compatible -- P 2 terms only, no R 2 –Much better suited to error measurements. Subtlety:Subtlety: –Won’t divide-by-zero if w=0 or h 33 =0 (it happens!) –has a 3 rd row; it is not degenerate if w’=0 –OK to use it… (Solve 8x12) h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 0 T -w’ i x i T y’ I x i T 0 T -w’ i x i T y’ I x i T w’ i x i T 0 T -x’ i x i T =0 w’ i x i T 0 T -x’ i x i T =0 -y’ i x i T -x’ i x i T 0 T -y’ i x i T -x’ i x i T 0 T

13. Deceptive ‘Robustness’ Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences: –Use DLT to write 8x9 matrix A: Ah=0 –Solve for h null space. ALWAYS gives H matrix But what if points are bad / fictional?But what if points are bad / fictional? –3 collinear input pts, crooked out: IMPOSSIBLE! –Yet we get an H solution! Why? A matrix rank is 7 or 6  rank 1 H result(s)A matrix rank is 7 or 6  rank 1 H result(s) –‘Null Space’ of A may contain >1 H solution! –‘Degenerate’ H solution of form aH 1 + bH 2 … Answer: SVD ranks A; reject bad point sets.Answer: SVD ranks A; reject bad point sets.

14. Actual ‘Robustness’ Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics… Use DLT formulation: (Hx  x’) = 0Use DLT formulation: (Hx  x’) = 0 Rearrange as dot product: (known)·(unknown) = 0Rearrange as dot product: (known)·(unknown) = 0 Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… (pg 75) tricky when you mix types: points, lines,… (pg 75) measured input transformed  measured output measured output = 0

15. Adding More Measurements How can we use >4 point correspondences? Easy:Easy: –Add more rows to our 8x9 matrix A: A·h = 0 –Use SVD to find Null space (Always gives an answer!) –Result: ‘Least squares’ solution; error=A·h =  minimizes || A h || 2 =  i  2 i where,  i is error for i-th pt. correspondence:minimizes || A h || 2 =  i  2 i where,  i is error for i-th pt. correspondence:  i = || Hx i  x i ’ || 2 = || (2 rows of A)*h || 2 = ‘algebraic distance’  i = || Hx i  x i ’ || 2 = || (2 rows of A)*h || 2 = ‘algebraic distance’ –‘Algebraic Distance’ ? No geometric meaning!

16. Adding More Measurements 2D‘Algebraic Distance’ ? No geometric meaning!2D‘Algebraic Distance’ ? No geometric meaning! 2D ‘Geometric Distance’ d(a,b) 2 is Better: measurable length in input or output space2D ‘Geometric Distance’ d(a,b) 2 is Better: measurable length in input or output space if a = (a 1 a 2 a 3 ) and b = (b 1 b 2 b 3 ), then define Turns out that: ( - ) 2 + ( - ) 2 d(a,b) 2 = d(a,b) 2 = a1a1a3a3a1a1a3a3 b2b2b3b3b2b2b3b3 b1b1b3b3b1b1b3b3 a2a2a3a3a2a2a3a3 d(a,b) 2 = d algebraic (a,b) d(a,b) 2 = d algebraic (a,b) a3·b3a3·b3a3·b3a3·b3 (Not very surprising)

17. Adding More Measurements Overall Strategy: Overconstrain the answer HOverconstrain the answer H –Collect extra measurements (>4 point pairs, etc. …) –expect errors; call them ‘estimates’ x Compute a 1 st solution (probably by SVD)Compute a 1 st solution (probably by SVD) Compute error d(Hx, x’) 2, and use this to…Compute error d(Hx, x’) 2, and use this to… ‘Tweak’ answer H and estimates x‘Tweak’ answer H and estimates x Compute new answerCompute new answer Stop when error < useful thresholdStop when error < useful threshold ^ ^ ^ ^

18. Using Estimates Simplest: ‘one image’ transfer method:Simplest: ‘one image’ transfer method: –Assume inputs are a perfect test pattern: only output pts are in error –Adjust output estimates x’ until d(Hx, x’) 2  0 (note we re-compute H as x’ changes) Better: ‘Symmetric’ transfer method:Better: ‘Symmetric’ transfer method: –Assume BOTH inputs and outputs have error. –Adjust BOTH input and output ests x x’ (note we re-compute H as x, x’ change) –Stop when d(Hx,x’) 2 + d(H -1 x’, x) 2  0 ^^ ^^ ^ ^ ^ ^

19. END

20. DLT: Four-Point Correspondence 4 point correspondence:4 point correspondence: –Book shows: –Rearrange: known vector (dot) unknown vector –stack, solve for null space… x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [x y 1 0 0 0 -x’x -x’y -x’] = 0 [0 0 0 x y 1 -y’x -y’y -y’]

21. Direct Linear Tranform (DLT) Project 2 method: Solve Hx – x’ = 0Project 2 method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 in measurement) compare:compare: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1 OK, xx 3 ) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

22. Direct Linear Tranform (DLT) Project 2 method: Solve Hx – x’ = 0Project 2 method: Solve Hx – x’ = 0 –Requires constant scale (x 3 =1 always) compare:compare: DLT method: Solve Hx  x’ = 0DLT method: Solve Hx  x’ = 0 –Accepts any scale (x 3 =w  1, x 3 ’=w’  1) [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 [ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0 [w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [ 0 T -w’ i x i T y’ I x i T ] [ w’ i x i T 0 T -x’ i x i T ]

23. Direct Linear Tranform (DLT) DLT method: Solve H  x = 0DLT method: Solve H  x = 0 –Accepts any scale, any point (x 3 =w  1 is OK) –‘Pure’, Compatible -- P 2 terms only –Much better suited to error measurements. Subtlety:Subtlety: –Won’t divide-by-zero if w=0 or h 33 =0 (it happens!) –has a 3 rd row; it is not degenerate if w’=0 –OK to use it… (Solve 8x12) h1 h1 h2 h2 h3 h3 h1 h1 h2 h2 h3 h3 0 T -w’ i x i T y’ I x i T 0 T -w’ i x i T y’ I x i T w’ i x i T 0 T -x’ i x i T =0 w’ i x i T 0 T -x’ i x i T =0 -y’ i x i T -x’ i x i T 0 T -y’ i x i T -x’ i x i T 0 T

24. Deceptive ‘Robustness’ Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences: –Use DLT to write 8x9 (or 12x9) matrix A: Ah=0 –Solve for h null space. ALWAYS gives H matrix But what if points are bad / fictional?But what if points are bad / fictional? –3 collinear input pts, crooked out: IMPOSSIBLE! –Yet we get an H solution! Why? A matrix rank is 7 or 6  rank 1 H result(s)A matrix rank is 7 or 6  rank 1 H result(s) –‘Null Space’ of A may contain >1 H solution! –‘Degenerate’ H solution of form aH 1 + bH 2 … Answer: SVD ranks A; reject bad point sets.Answer: SVD ranks A; reject bad point sets.

25. Actual ‘Robustness’ Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…Vectorizing (‘Flatten, Stack, Null Space’ method) works for almost ANY input! (Points, lines, planes, ||, , conics, quadrics, cross-ratios, vanishing points, twisted cubics… Use DLT formulation: (Hx  x’) = 0Use DLT formulation: (Hx  x’) = 0 Rearrange as dot product: (known)·(unknown) = 0Rearrange as dot product: (known)·(unknown) = 0 Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… (pg 75) tricky when you mix types: points, lines,… (pg 75) measured input transformed  measured output measured output = 0