1. Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y

2. Camera calibration

3. Estimate both intrinsic and extrinsic parameters Mainly, two categories: 1.Using objects with known geometry as reference 2.Self calibration (structure from motion)

4. One app of camera pose application Virtual gaming –http://www.livestream.com/emtech/video?clipId= pla_74103098-95fb-4704-99f0-d07339dc16a1

5. Camera calibration approaches Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

6. Linear regression

8. Solve for Projection Matrix M using least- square techniques

9. Normal equation (Geometric Interpretation) Given an overdetermined system the normal equation is that which minimizes the sum of the square differences between left and right sides

10. Normal equation (Differential Interpretation) n x m, n equations, m variables

11. Normal equation Carl Friedrich Gauss

12. Any issues with the method?

13. Nonlinear optimization A probabilistic view of least square Feature measurement equations Likelihood of M given {(u i,v i )}

14. Optimal estimation Log likelihood of M given {(u i,v i )} It is a least square problem (but not necessarily linear least square) How do we minimize C?

15. Nonlinear least square methods

16. Least square fitting number of data points number of parameters

17. Nonlinear least square fitting

18. Function minimization It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum. Least square is related to function minimization.

19. Function minimization

20. Quadratic functions Approximate the function with a quadratic function within a small neighborhood

21. Function minimization

22. Computing gradient and Hessian Gradient Hessian

23. Computing gradient and Hessian Gradient Hessian

24. Computing gradient and Hessian Gradient Hessian

25. Computing gradient and Hessian Gradient Hessian

26. Computing gradient and Hessian Gradient Hessian

27. Searching for update h GradientHessian Idea 1: Steepest Descent

28. Steepest descent method isocontourgradient

29. Steepest descent method It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

30. Searching for update h GradientHessian Idea 2: minimizing the quadric directly Converge faster but needs to solve the linear system

31. Recap: Calibration Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi) Camera Model:

32. Recap: Calibration Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi) Linear Approach:

33. Recap: Calibration Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi) NonLinear Approach:

34. Practical Issue is hard to make and the 3D feature positions are difficult to measure!

35. A popular calibration tool

36. Multi-plane calibration Images courtesy Jean-Yves Bouguet, Intel Corp. Advantage Only requires a plane Don’t have to know positions/orientations Good code available online! –Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/ http://www.intel.com/research/mrl/research/opencv/ –Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html http://www.vision.caltech.edu/bouguetj/calib_doc/index.html –Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/ http://research.microsoft.com/~zhang/Calib/

37. Step 1: data acquisition

38. Step 2: specify corner order

39. Step 3: corner extraction

41. Step 4: minimize projection error

42. Step 4: camera calibration

44. Step 5: refinement

45. Next Image Mosaics and Panorama Today’s Readings –Szeliski and Shum paper http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski/ http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski/ Full screen panoramas (cubic): http://www.panoramas.dk/http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.htmlhttp://www.panoramas.dk/fullscreen3/f2_mars97.html 2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.htmlhttp://www.panoramas.dk/fullscreen3/f1.html

46. Why Mosaic? Are you getting the whole picture? –Compact Camera FOV = 50 x 35° Slide from Brown & Lowe

47. Why Mosaic? Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° Slide from Brown & Lowe

48. Why Mosaic? Are you getting the whole picture? –Compact Camera FOV = 50 x 35° –Human FOV = 200 x 135° –Panoramic Mosaic = 360 x 180° Slide from Brown & Lowe

49. Mosaics: stitching images together Creating virtual wide-angle camera

50. Auto Stitch: the State of Art Method Demo Project 2 is a striped-down AutoStitch

51. How to do it? Basic Procedure –Take a sequence of images from the same position Rotate the camera about its optical center –Compute transformation between second image and first –Transform the second image to overlap with the first –Blend the two together to create a mosaic –If there are more images, repeat

52. Geometric Interpretation of Mosaics Image 1 Image 2 If we capture all the 360º rays in different images, we can assemble them into a panorama. The basic operation is projecting an image from one plane to another The projective transformation is scene-INDEPENDENT Optical Center

53. What is the transformation? Translations are not enough to align the images left on topright on top