1. 3D Urban Modeling Marc Pollefeys Jan-Michael Frahm, Philippos Mordohai Fall 2006 / Comp 790-089 Tue & Thu 14:00-15:15

2. 3D Urban Modeling Basics: sensor, techniques, … Video, LIDAR, GPS, IMU, … SfM, Stereo, … Robot Mapping, GIS, … Review state-of-the-art Video and LIDAR-based systems Projects to experiment Virtual UNC-Chapel Hill DARPA Urban Challenge …

3. 3D Urban Modeling course schedule (tentative) Sep. 5, 7Introduction Video-based Urban 3D Capture Sep. 12, 14Cameras Sep. 19, 21 Sep. 26, 28Project proposals Oct. 3, 5 Oct. 10, 12 Oct. 17, 19(fall break) Oct. 24, 26Project Status Update Oct. 31, Nov. 2 Nov. 7, 9 Nov. 14, 16 Nov. 21, 23(Thanksgiving) Nov. 28, 30 Dec. 5Project demonstrations(classes ended) Note: Dec. 3 is CVPR deadline

4. Course material Slides, notes, papers and references see course webpage/wiki (later) On-line “shape-from-video” tutorial: http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/

5. Geometry primer Incidence of point and line Joins Conics and dual conics Incidence of point and plane Joins Conics and dual conics

6. Geometry primer Transformation for lines Transformation for conics Transformation for dual conics Transformation for points Transformation for planes Transformation for quadrics Transformation for dual quadrics Transformation for points

7. Geometry primer Hierarchy of 2D transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l ∞ Ratios of lengths, angles. The circular points I,J lengths, areas. invariants transformed squares

8. Geometry primer Hierarchy of 3D transformations Projective 15dof Affine 12dof Similarity 7dof Euclidean 6dof Intersection and tangency Parallellism of planes, Volume ratios, centroids, The plane at infinity π ∞ Angles, ratios of length The absolute conic Ω ∞ Volume

9. Camera model Relation between pixels and rays in space ?

10. Projector model Relation between pixels and rays in space (dual of camera) (main geometric difference is vertical principal point offset to reduce keystone effect) ?

11. Affine cameras

12. Pinhole camera model linear projection in homogeneous coordinates! homogeneous coordinates non-homogeneous coordinates

13. Pinhole camera model

14. Principal point offset principal point

15. Principal point offset calibration matrix

16. Camera rotation and translation ~

17. Object motion

18. Camera motion

19. CCD camera

20. General projective camera non-singular 11 dof (5+3+3) intrinsic camera parameters extrinsic camera parameters

21. Camera matrix decomposition Finding the camera center (use SVD to find null-space) Finding the camera orientation and internal parameters (use RQ decomposition ~QR) Q R =( ) -1 = -1 -1 Q R (if only QR, invert) (for all X and λ  C must be camera center)

22. Affine cameras

23. Radial distortion Due to spherical lenses (cheap) Model: R R http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg straight lines are not straight anymore pincushion dist. barrel dist.

24. Radial distortion example

25. Meydenbauer camera vertical lens shift to allow direct ortho-photographs

26. Action of projective camera on points and lines forward projection of line back-projection of line projection of point

27. Action of projective camera on conics and quadrics back-projection to cone projection of quadric

28. Resectioning

29. Direct Linear Transform (DLT) rank-2 matrix

30. Direct Linear Transform (DLT) Minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint use SVD

31. Singular Value Decomposition Homogeneous least-squares

32. Degenerate configurations (i)Points lie on plane or single line passing through projection center (ii)Camera and points on a twisted cubic

33. Scale data to values of order 1 1.move center of mass to origin 2.scale to yield order 1 values Data normalization

34. Line correspondences Extend DLT to lines (back-project line) (2 independent eq.)

35. Geometric error

36. Gold Standard algorithm Objective Given n≥6 2D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelyhood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

37. Calibration example (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (H&Z rule of thumb: 5 n constraints for n unknowns)

38. Errors in the world Errors in the image and in the world Errors in the image (standard case)

39. Restricted camera estimation Minimize geometric error  impose constraint through parametrization Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)||

40. Restricted camera estimation Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights

42. Image of absolute conic

43. A simple calibration device (i)compute H for each square (corners  (0,0),(1,0),(0,1),(1,1)) (ii)compute the imaged circular points H(1,±i,0) T (iii)fit a conic to 6 circular points (iv)compute K from  through cholesky factorization (≈ Zhang’s calibration method)

44. Some typical calibration algorithms Tsai calibration Zhangs calibration http://research.microsoft.com/~zhang/calib/ Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999. http://www.vision.caltech.edu/bouguetj/calib_doc/

45. ** ** projection constraints Self-calibration using absolute conic Absolute conic projection: Translate constraints on K through projection equation to constraints on  * Euclidean projection matrix: Upgrade from projective to metric Transform structure and motion so that  *  diag(1,1,1,0) some constraints, e.g. constant, no skew,... (Faugeras ECCV’92; Triggs CVPR’97; Pollefeys et al. ICCV’98; etc.)

46. Practical linear self-calibration (Pollefeys et al., ECCV‘02) (only rough aproximation, but still usefull to avoid degenerate configurations) (relatively accurate for most cameras) Don’t treat all constraints equal after normalization! when fixating point at image-center not only absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs., but also diag(1,1,1,a), i.e. real or imaginary spheres!