Advanced Computer Vision Feature-based Alignment Lecturer: Lu Yi & Prof. Fuh CSIE NTU

Content Pre-requisites: mainly about camera model Alignment Algorithm RANdom Sample Consensus (RANSAC) Pose Estimation

Sources EGGN 512 – Computer Vision course by Prof. Willliam Hoff Digital Visual Effects course by Prof. Yung-Yu Chuang Both are highly recommended

Taught Way Cover the main parts of textbook chapter 6 Lecture mostly depended on EGGN 512 course videos

Frames of Reference Image frames are 2D; other are 3D The “pose” (position and orientation) of a 3D rigid body has 6 degrees of freedom

Pre-requisites Image Formation Geometric Transformations (2D-2D, 3D-3D, 3D-2D) Camera Model (Camera Matrix = Intrinsic + Extrinsic camera matrix) All related contents are in textbook chapter 2: Image formation

Pre-requisites Review 1.Projective Transformation 2.Camera Matrix EGGN 512 – Lecture 5-1 3D-2D Transforms L4B3F8D4A5CAD8DA3 L4B3F8D4A5CAD8DA3 EGGN 512 – Lecture 5-2 3D-2D Transforms CAD8DA3 CAD8DA3

Alignment Algorithm EGGN 512 – Lecture 14-1 Alignment 8DA3 8DA3 EGGN 512 – Lecture 14-2 Alignment 8DA3 8DA3 EGGN 512 – Lecture 14-3 Alignment DA3 DA3 Slides AlignmentNonlinear.pdf AlignmentNonlinear.pdf

Alignment Algorithm Feature-based methods: only use feature points to estimate parameters Features: SIFT, SURF, Harris.etc.

Determine pairwise alignment p’=Mp, where M is a transformation matrix, p and p’ are feature matches It is possible to use more complicated models such as affine or perspective For example, assume M is a 2x2 matrix Find M with the least square error

Determine pairwise alignment Overdetermined system

Normal Equation Given an overdetermined system the normal equation is that which minimizes the sum of the square differences between left and right sides Why?

Normal Equation n x m, n equations, m variables

Normal Equation

→

Determine pairwise alignment p’=Mp, where M is a transformation matrix, p and p’ are feature matches For translation model, it is easier. What if the match is false? Avoid impact of outliers. (Ans: RANSAC)

Application: Image Stitching Stitching = alignment + blending geometrical registration photometric registration

Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35°

Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35° Human FOV = 200 x 135°

Application: Image Stitching Panorama – the whole picture Compact Camera FOV = 50 x 35° Human FOV = 200 x 135° Panoramic Mosaic = 360 x 180°

Application: Image Stitching Example:

Application: Face Alignment Face alignment is the key preprocessing step for face recognition

Application: Face Alignment Examples from LFW dataset :

RANSAC Algorithm RANSAC = RANdom Sample Consensus An iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers [Wiki] Compare to robust statistics Given N data points x i, assume that majority of them are generated from a model with parameters , try to recover . Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6),

RANSAC Algorithm Run k times: (1) draw n samples randomly (2) fit parameters  with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output  with the largest c How many times? How big? Smaller is better How to define? Depends on the problem.

RANSAC Algorithm (How to determine K) p : probability of real inliers P : probability of success after k trials n samples are all inliers a failure failure after k trials npk for P=0.99

RANSAC Algorithm (Example: line fitting)

n=2

RANSAC Algorithm (Example: line fitting)

c=3

RANSAC Algorithm (Example: line fitting) c=3

RANSAC Algorithm (Example: line fitting) c=15

RANSAC Algorithm EGGN 512 – Lecture 27-1 RANSAC 5CAD8DA3&index=54 5CAD8DA3&index=54

RANSAC Algorithm (Example: Homography)

Pose Estimation EGGN 512 – Lecture 16-1 Pose Estimation CAD8DA3 CAD8DA3 EGGN 512 – Lecture 17-1 Pose from Lines 5CAD8DA3 5CAD8DA3

Pose Estimation EGGN 512 – Lecture 18-1 SVD AD8DA3 AD8DA3 EGGN 512 – Lecture 18-2 SVD AD8DA3 AD8DA3

Pose Estimation EGGN 512 – Lecture 19-1 Linear Pose Estimation CAD8DA3 CAD8DA3 EGGN 512 – Lecture 19-2 Linear Pose Estimation AD8DA3 AD8DA3 EGGN 512 – Lecture 19-3 Linear Pose Estimation 3Xfn1M&list=PL4B3F8D4A5CAD8DA3 3Xfn1M&list=PL4B3F8D4A5CAD8DA3

END