1. Advanced Computer Vision Feature-based Alignment Lecturer: Lu Yi & Prof. Fuh lynn.luyi@gmail.com CSIE NTU

2. Content Pre-requisites: mainly about camera model Alignment Algorithm RANdom Sample Consensus (RANSAC) Pose Estimation

3. Sources EGGN 512 – Computer Vision course by Prof. Willliam Hoff http://inside.mines.edu/~whoff/courses/EENG512/ Playlist@Youtube: http://www.youtube.com/playlist?list=PL4B3F8D4A5CAD8DA3 Digital Visual Effects course by Prof. Yung-Yu Chuang http://www.csie.ntu.edu.tw/~cyy/courses/vfx/14spring/news/ Both are highly recommended

4. Taught Way Cover the main parts of textbook chapter 6 Lecture mostly depended on EGGN 512 course videos

5. Frames of Reference Image frames are 2D; other are 3D The “pose” (position and orientation) of a 3D rigid body has 6 degrees of freedom

6. Pre-requisites Image Formation Geometric Transformations (2D-2D, 3D-3D, 3D-2D) Camera Model (Camera Matrix = Intrinsic + Extrinsic camera matrix) All related contents are in textbook chapter 2: Image formation

7. Pre-requisites Review 1.Projective Transformation 2.Camera Matrix EGGN 512 – Lecture 5-1 3D-2D Transforms http://www.youtube.com/watch?v=DFNjOUMuecU&index=11&list=P L4B3F8D4A5CAD8DA3 http://www.youtube.com/watch?v=DFNjOUMuecU&index=11&list=P L4B3F8D4A5CAD8DA3 EGGN 512 – Lecture 5-2 3D-2D Transforms http://www.youtube.com/watch?v=5gesrLgNuQo&list=PL4B3F8D4A5 CAD8DA3 http://www.youtube.com/watch?v=5gesrLgNuQo&list=PL4B3F8D4A5 CAD8DA3

8. Alignment Algorithm EGGN 512 – Lecture 14-1 Alignment http://www.youtube.com/watch?v=UcU4814hvR8&list=PL4B3F8D4A5CAD 8DA3 http://www.youtube.com/watch?v=UcU4814hvR8&list=PL4B3F8D4A5CAD 8DA3 EGGN 512 – Lecture 14-2 Alignment http://www.youtube.com/watch?v=XxEKMecNZk0&list=PL4B3F8D4A5CAD 8DA3 http://www.youtube.com/watch?v=XxEKMecNZk0&list=PL4B3F8D4A5CAD 8DA3 EGGN 512 – Lecture 14-3 Alignment http://www.youtube.com/watch?v=lSXfv4baMwk&list=PL4B3F8D4A5CAD8 DA3 http://www.youtube.com/watch?v=lSXfv4baMwk&list=PL4B3F8D4A5CAD8 DA3 Slides http://inside.mines.edu/~whoff/courses/EENG512/lectures/15- AlignmentNonlinear.pdf http://inside.mines.edu/~whoff/courses/EENG512/lectures/15- AlignmentNonlinear.pdf

9. Alignment Algorithm Feature-based methods: only use feature points to estimate parameters Features: SIFT, SURF, Harris.etc.

10. Determine pairwise alignment p’=Mp, where M is a transformation matrix, p and p’ are feature matches It is possible to use more complicated models such as affine or perspective For example, assume M is a 2x2 matrix Find M with the least square error

11. Determine pairwise alignment Overdetermined system

12. Normal Equation Given an overdetermined system the normal equation is that which minimizes the sum of the square differences between left and right sides Why?

13. Normal Equation n x m, n equations, m variables

14. Normal Equation

15. →

18. Determine pairwise alignment p’=Mp, where M is a transformation matrix, p and p’ are feature matches For translation model, it is easier. What if the match is false? Avoid impact of outliers. (Ans: RANSAC)

19. Application: Image Stitching Stitching = alignment + blending geometrical registration photometric registration

20. Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35°

21. Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35° Human FOV = 200 x 135°

22. Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35° Human FOV = 200 x 135° Panoramic Mosaic = 360 x 180°

23. Application: Image Stitching Example: http://graphics.stanford.edu/courses/cs178/applets/projection.html

24. Application: Face Alignment Face alignment is the key preprocessing step for face recognition

25. Application: Face Alignment Examples from LFW dataset :

26. RANSAC Algorithm RANSAC = RANdom Sample Consensus An iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers [Wiki] Compare to robust statistics Given N data points x i, assume that majority of them are generated from a model with parameters , try to recover . Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395.

27. RANSAC Algorithm Run k times: (1) draw n samples randomly (2) fit parameters  with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output  with the largest c How many times? How big? Smaller is better How to define? Depends on the problem.

28. RANSAC Algorithm (How to determine K) p : probability of real inliers P : probability of success after k trials n samples are all inliers a failure failure after k trials npk 30.535 60.697 60.5293 for P=0.99

29. RANSAC Algorithm (Example: line fitting)

30. n=2

31. RANSAC Algorithm (Example: line fitting)

33. c=3

34. RANSAC Algorithm (Example: line fitting) c=3

35. RANSAC Algorithm (Example: line fitting) c=15

36. RANSAC Algorithm EGGN 512 – Lecture 27-1 RANSAC http://www.youtube.com/watch?v=NKxXGsZdDp8&list=PL4B3F8D4A 5CAD8DA3&index=54 http://www.youtube.com/watch?v=NKxXGsZdDp8&list=PL4B3F8D4A 5CAD8DA3&index=54

37. RANSAC Algorithm (Example: Homography)

40. Pose Estimation EGGN 512 – Lecture 16-1 Pose Estimation http://www.youtube.com/watch?v=kq3c6QpcAGc&list=PL4B3F8D4A5 CAD8DA3 http://www.youtube.com/watch?v=kq3c6QpcAGc&list=PL4B3F8D4A5 CAD8DA3 EGGN 512 – Lecture 17-1 Pose from Lines http://www.youtube.com/watch?v=D_4eUoqgWdc&list=PL4B3F8D4A 5CAD8DA3 http://www.youtube.com/watch?v=D_4eUoqgWdc&list=PL4B3F8D4A 5CAD8DA3

41. Pose Estimation EGGN 512 – Lecture 18-1 SVD http://www.youtube.com/watch?v=C852P1JrHXI&list=PL4B3F8D4A5C AD8DA3 http://www.youtube.com/watch?v=C852P1JrHXI&list=PL4B3F8D4A5C AD8DA3 EGGN 512 – Lecture 18-2 SVD http://www.youtube.com/watch?v=aIkzK4CdYes&list=PL4B3F8D4A5C AD8DA3 http://www.youtube.com/watch?v=aIkzK4CdYes&list=PL4B3F8D4A5C AD8DA3

42. Pose Estimation EGGN 512 – Lecture 19-1 Linear Pose Estimation http://www.youtube.com/watch?v=HojSSrxsB4Q&list=PL4B3F8D4A5 CAD8DA3 http://www.youtube.com/watch?v=HojSSrxsB4Q&list=PL4B3F8D4A5 CAD8DA3 EGGN 512 – Lecture 19-2 Linear Pose Estimation http://www.youtube.com/watch?v=ik8dFybnyPY&list=PL4B3F8D4A5C AD8DA3 http://www.youtube.com/watch?v=ik8dFybnyPY&list=PL4B3F8D4A5C AD8DA3 EGGN 512 – Lecture 19-3 Linear Pose Estimation http://www.youtube.com/watch?v=OS5b- 3Xfn1M&list=PL4B3F8D4A5CAD8DA3 http://www.youtube.com/watch?v=OS5b- 3Xfn1M&list=PL4B3F8D4A5CAD8DA3

43. END