1. Robot Vision SS 2008 Matthias Rüther 1 ROBOT VISION Lesson 6: Shape from Stereo Matthias Rüther Slides partial courtesy of Marc Pollefeys Department of Computer Science University of North Carolina, Chapel Hill

2. Robot Vision SS 2008 Matthias Rüther 2 Content  Two View Geometry –Epipolar Geometry –3D reconstruction  Computing F –Point Correspondences –Interest Points –Matching

3. Robot Vision SS 2008 Matthias Rüther 3 C, C’, x, x’ and X are coplanar Epipolar Geometry

4. Robot Vision SS 2008 Matthias Rüther 4 (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three Questions

5. Robot Vision SS 2008 Matthias Rüther 5 What if only C, C’, x are known? Epipolar Geometry

6. Robot Vision SS 2008 Matthias Rüther 6 All points on  project on l and l’ Epipolar Geometry

7. Robot Vision SS 2008 Matthias Rüther 7 Family of planes  and lines l and l’ Intersection in e and e’ Epipolar Geometry

8. Robot Vision SS 2008 Matthias Rüther 8 epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) Epipolar Geometry

9. Robot Vision SS 2008 Matthias Rüther 9 Example: Converging Cameras

10. Robot Vision SS 2008 Matthias Rüther 10 Example: motion parallel with image plane

11. Robot Vision SS 2008 Matthias Rüther 11 e e’ Example: forward motion

12. Robot Vision SS 2008 Matthias Rüther 12 algebraic representation of epipolar geometry we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F The Fundamental Matrix (F)

13. Robot Vision SS 2008 Matthias Rüther 13 geometric derivation mapping from 2-D to 1-D family (rank 2) The Fundamental Matrix (F)

14. Robot Vision SS 2008 Matthias Rüther 14 algebraic derivation (note: doesn’t work for C=C’  F=0) The Fundamental Matrix (F)

15. Robot Vision SS 2008 Matthias Rüther 15 correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x ’ in the two images The Fundamental Matrix (F)

16. Robot Vision SS 2008 Matthias Rüther 16 F is the unique 3x3 rank 2 matrix that satisfies x’ T Fx=0 for all x↔x’ (i)Transpose: if F is fundamental matrix for (P,P’), then F T is fundamental matrix for (P’,P) (ii)Epipolar lines: l’=Fx & l=F T x’ (iii)Epipoles: on all epipolar lines, thus e’ T Fx=0,  x  e’ T F=0, similarly Fe=0 (iv)F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v)F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible) The Fundamental Matrix (F)

17. Robot Vision SS 2008 Matthias Rüther 17 l,l’ epipolar lines, k line not through e  l’=F[k] x l and symmetrically l=F T [k’] x l’ (pick k=e, since e T e≠0) Epipolar Line Geometry

18. Robot Vision SS 2008 Matthias Rüther 18 Derivation based purely on projective concepts F invariant to transformations of projective 3-space unique not unique canonical form Invariance under projective transformation

19. Robot Vision SS 2008 Matthias Rüther 19 F matrix corresponds to P,P’ iff P’ T FP is skew-symmetric Possible choice: Canonical representation: Canonical cameras given F

20. Robot Vision SS 2008 Matthias Rüther 20 ~fundamental matrix for calibrated cameras (remove K) 5 d.o.f. (3 for R; 2 for t up to scale) E is essential matrix if and only if two singular values are equal (and third=0) The Essential Matrix

21. Robot Vision SS 2008 Matthias Rüther 21 (only one solution where points is in front of both cameras) Reconstruction from E: four possible solutions

22. Robot Vision SS 2008 Matthias Rüther 22 C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1 Canonical representation: Epipolar Geometry

23. Robot Vision SS 2008 Matthias Rüther 23 given x i ↔x‘ i, compute P,P‘ and X i reconstruction problem: for all i without additional information possible up to projective ambiguity 3D reconstruction of cameras and structure

24. Robot Vision SS 2008 Matthias Rüther 24 (i)Compute F from correspondences (ii)Compute camera matrices from F (iii)Compute 3D point for each pair of corresponding points computation of F use x‘ i Fx i =0 equations, linear in coeff. F 8 points (linear), 7 points (non-linear), 8+ (least-squares) computation of camera matrices use triangulation compute intersection of two backprojected rays Outline of reconstruction

25. Robot Vision SS 2008 Matthias Rüther 25 Reconstruction ambiguity: similarity

26. Robot Vision SS 2008 Matthias Rüther 26 Reconstruction ambiguity: projective

27. Robot Vision SS 2008 Matthias Rüther 27 x i ↔x‘ i Original scene X i Projective, affine, similarity reconstruction = reconstruction that is identical to original up to projective, affine, similarity transformation Literature: Metric and Euclidean reconstruction = similarity reconstruction Terminology

28. Robot Vision SS 2008 Matthias Rüther 28 If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent  along same ray of P 2, idem for P ‘ 2 two possibilities: X 2 i = HX 1 i, or points along baseline key result: allows reconstruction from pair of uncalibrated images The projective reconstruction theorem

29. Robot Vision SS 2008 Matthias Rüther 29

30. Robot Vision SS 2008 Matthias Rüther 30 (i)Projective reconstruction (ii)Affine reconstruction (iii)Metric reconstruction Stratified reconstruction

31. Robot Vision SS 2008 Matthias Rüther 31 use control points X E i with known coordinates to go from projective to metric (2 lin. eq. in H -1 per view, 3 for two views) Direct metric reconstruction using ground truth

32. Robot Vision SS 2008 Matthias Rüther 32 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1 Canonical representation: Epipolar Geometry: computation of F

33. Robot Vision SS 2008 Matthias Rüther 33 separate known from unknown (data) (unknowns) (linear) Computation of F: basic equation

34. Robot Vision SS 2008 Matthias Rüther 34 SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation Imposing the singularity constraint

35. Robot Vision SS 2008 Matthias Rüther 35

36. Robot Vision SS 2008 Matthias Rüther 36 ~10000 ~100 1 ! Orders of magnitude difference Between column of data matrix  least-squares yields poor results The NOT normalized 8-point algorithm

37. Robot Vision SS 2008 Matthias Rüther 37 Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1)(-1,1) (-1,-1) Least squares yields good results (Hartley, PAMI´97) The normalized 8-point algorithm

38. Robot Vision SS 2008 Matthias Rüther 38 Maximum Likelihood Estimation (= least-squares for Gaussian noise) Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct X i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized) The Gold Standard Algorithm

39. Robot Vision SS 2008 Matthias Rüther 39 Examples

40. Robot Vision SS 2008 Matthias Rüther 40 Examples

41. Robot Vision SS 2008 Matthias Rüther 41 Examples

42. Robot Vision SS 2008 Matthias Rüther 42 1.Do not use unnormalized algorithms 2.Quick and easy to implement: 8-point normalized 3.Better: enforce rank-2 constraint during minimization 4.Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation) Recommendations

43. Robot Vision SS 2008 Matthias Rüther 43  Extract feature points to relate images  Required properties: –Well-defined (i.e. neigboring points should all be different) –Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints) The correspondence problem: feature points

44. Robot Vision SS 2008 Matthias Rüther 44 homogeneous edge corner M should have large eigenvalues (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

45. Robot Vision SS 2008 Matthias Rüther 45 Select strongest features (e.g. 1000/image) Feature points

46. Robot Vision SS 2008 Matthias Rüther 46 Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ? Feature matching

47. Robot Vision SS 2008 Matthias Rüther 47 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Similarity Example

48. Robot Vision SS 2008 Matthias Rüther 48 Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 8 matches) Step 3.2 compute F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) RANSAC

49. Robot Vision SS 2008 Matthias Rüther 49 restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

50. Robot Vision SS 2008 Matthias Rüther 50  Degenerate cases –Planar scene –Pure rotation  No unique solution –Remaining DOF filled by noise –Use simpler model (e.g. homography)  Model selection (Torr et al., ICCV´98, Kanatani, Akaike) –Compare H and F according to expected residual error (compensate for model complexity) Degenerate Cases

51. Robot Vision SS 2008 Matthias Rüther 51  Absence of sufficient features (no texture)  Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition More problems