1. 3D Reconstruction Jeff Boody

2. Goals ● Reconstruct 3D models from a sequence of at least two images ● No prior knowledge of the camera or scene ● Use the resulting 3D depth map as an input into the Biomimetic vision system

3. Overview (1)Feature Extraction/ Matching (2)Relating Images (3)Projective Reconstruction (4)Self-Calibration (5)Dense Matching (6)3D Model Building

4. Feature Extraction ● Harris corner detector – Create gradient images Ix, Iy using convolution mask [-2, -1, 0, 1, 2] – Compute Ixx, Ixy, Iyy (smoothing optional) – The corner function: R(x,y) = det(x,y) - k trace(x,y)^2 – det(x,y) = Ixx(x,y) Iyy(x, y) – Ixy(x,y) Ixy(x,y) – trace(x,y) = Ixx(x,y) + Iyy(x,y) – Select local maxima over a threshold (vxl – adaptive) – Compute the sub-pixel location of corners

5. Feature Matching ● Correlation – Compare features in I to features in I' that are within a search widow (approx 1/8 th the image) – C = ∑∑ (I(x-i,y-j)-Imean)(I'(x-i,y-j)-I'mean) – The correlation score is summed over a region i=(-N,N), j=(-N,N), where N = 3 ● Zhang's robust matching – Correlation, strength/unambiguity of matches, iterative relaxation – Eliminates ambiguity caused by multiple matches

6. Relating Images(1) ● Fundamental matrix – u' T Fu = 0 – uu'F11 + uv'F21 + uF31 + vu'F12 + vv'F22 + vF32 + u'F13 + v'F23 + F33 = 0 – Af = 0, (uu', uv', u, vu', vv', v, u', v', 1) – ||f|| = 1, (F is only defined up to an unknown scale) – det(F) = 0 (singularity constraint, rank 2) – The epipoles are the left and right null spaces of F

7. Relating Images(2) ● 8-Point Algorithm – Normalize points, u = T1 u, u' = T2 u' – Points are translated to their centroid – Points are scaled so their average distance from the origin is √2 – Solve Af = 0, using SVD (A = UDV T, F is V 9 ) – Take the SVD of F (F = UDV T ) and set the smallest eigenvalue to 0, corresponding to the closest singular matrix under the Frobenius norm (||F|| = 1) – De-normalize F, F = T2 T F T1

8. Relating Images(3) ● 7-Point Algorithm – Same as the 8-point algorithm, except det(F) = 0 is enforced differently, and only 7 point correspondences are required – F = aF1 + (1-a)F2, where F1 and F2 are V 9 and V 8 respectively – Enforce the singularity constraint by solving det(aF1 + (1-a)F2) = 0 for a – This leads to a cubic equation in a that has one or three real solutions

9. Relating Images(4) ● In practice we have many more matches then 8, some of which can be noisy (є). ● Ransac or Least Median Squares (to name a few) – N is the number of matches – m is the number of samples chosen – p is the sample size (i.e. 7 or 8) – є is the percentage of outliers

10. Relating Images(5) ● Ransac ● while(1 - (1 - (1 - є)^p)^m < 0.95) – Select random sample (bucketing) – Solve for FM – Determine inliers (over all matches) ● Matches that are within a given threshold (typically 1 or 2 pixels) are chosen as inliers ● Keep the FM which has the largest number of inliers

11. Relating Images(6) ● Least Median Squares ● The same loop as Ransac can be used ● Inliers are chosen as follows – Calculate the residuals for every match, and compute the median – б = 1.4826(1 + 5 / (N-p))√M J – inlier if r i 2 ≤ (2.5 б ) 2, outlier otherwise – Select the FM for which the inliers minimize ∑r i 2

12. Relating Images(7) ● Distance metric: Euclidean distance from the point u' to its epipolar line Fu – Fu ~ l', FTu' ~ l – d(u', Fu) = |u' T Fu| / √((Fu) 2 1 + (Fu) 2 2 ) ● Residual – r i 2 = d 2 (u' i, Fu i ) + d 2 (u i, FTu' i )

13. Relating Images(8) ● Non-linear minimization (Levenberg- Marquardt) – Often, the fundamental matrix that was found using the robust method is still not good enough – Noise in the input data, outliers might not be detected, errors in setting the singularity constraint,... – Parameterize F = – Solve min d 2 (u' i, Fu i ) + d 2 (u i, FTu' i ), for inlier matches – The solution requires taking the derivative of the distance metric and possibly using up to 36 different parameterizations of F (i.e. epipoles at infiniti)

14. Projective Reconstruction(1) ● Projective Transform – x = PX – P 1 M = K[I 3 | 0 3 ] – P 2 M = K[R T | -R T t] – K =

15. Projective Reconstruction(2) ● Initial projection matrices (first method) – Normalize images: m = K *-1 m – P 1 = [I 3 | 0 3 ] – P 2 = [[e 2 ]xF + e 2 pi T | sigma e 2 ] – Choose pi such that: [e 2 ]xF + e 2 pi T = R * = I – sigma can be arbitrarily set to 1, cx = Image.width/2, cy = Image.height/2, s = 0, aspect ratio = 1 – For the focal length, several guesses must be tried, keeping the one with the most points reconstructed

16. Projective Reconstruction(3) ● Initial structure (first method) – u = PX where u = w(u,v,1) T, X = (x, y, z, w) T – wu = p 1 T X, wv = p 2 T X, w = p 3 T X ● Each point in each view results in – [up 3 T -p 1 T ]X = 0, [vp 3 T -p 2 T ]X = 0 ● Resulting in this set of linear equations: AX = 0 – M is number of points, N is number of views – A is 2N x 4M, X is 4M x 1

17. Projective Reconstruction(4) ● Perspective Factorization (second method)

18. Projective Reconstruction(5) ● Perspective Factorization (second method)

19. Projective Reconstruction(6) ● Other algorithms in projective reconstruction – Triangulation: a more robust method for reconstructing the initial structure – Iterated Extended Kalman Filters: a method to update the structure when more than two views are available – Bundle Adjustment: another non-linear minimization of the structure which requires the solution to take advantage of the sparse structure of the problem matrix. Otherwise, for a typical image sequence, the solution must be minimized over more than 6000 variables

20. Self-calibration(1) ● Self-calibration is the process that upgrades a projective reconstruction to a metric reconstruction

21. Self-calibration(2) ● The image of the absolute conic

22. Self-calibration(3) ● Constraints on the DIAC ● n x (#known) + (n - 1) x (#fixed) >= 8

23. Self-calibration(4) ● Linear algorithm

24. Self-calibration(5) ● Linear algorithm

25. Self-calibration(6) ● Example critical motion sequences – Pure translation: Scaling of optical axis (1 DOF) – Pure rotation: Arbitrary position of PI (3 DOF) – Orbital motion: ? – Planar motion: ?

26. Results(1) ● Feature extraction (first and third images)

27. Results(2) ● Feature matching

28. Results(3) ● Inliers after Ransac

29. Results(4) ● Inliers after tracking

30. Results(5) ● Epipolar lines

31. Results(6) ● Metric reconstruction using 3 virtual cameras

32. Future Work ● Finish projective reconstruction and self- calibration algorithms ● Non-linear minimization for FM, projective reconstruction, self-calibration ● Dense matching (image rectification followed by correlation along epipolar lines) ● Model building (Delunay triangulation) ● Critical motion sequences