Computing transformations Prof. Noah Snavely CS1114

Slides:



Advertisements
Similar presentations
Summary of Friday A homography transforms one 3d plane to another 3d plane, under perspective projections. Those planes can be camera imaging planes or.
Advertisements

Least squares CS1114
Announcements Final Exam May 13th, 8 am (not my idea).
Image alignment Image from
776 Computer Vision Jared Heinly Spring 2014 (slides borrowed from Jan-Michael Frahm, Svetlana Lazebnik, and others)
Robust and large-scale alignment Image from
Mosaics con’t CSE 455, Winter 2010 February 10, 2010.
Announcements Final Exam May 16 th, 8 am (not my idea). Practice quiz handout 5/8. Review session: think about good times. PS5: For challenge problems,
Affine Linear Transformations Prof. Graeme Bailey (notes modified from Noah Snavely, Spring 2009)
Geometric Optimization Problems in Computer Vision.
Lecture 11: Structure from motion CS6670: Computer Vision Noah Snavely.
Lecture 9: Image alignment CS4670: Computer Vision Noah Snavely
Lecture 10: Cameras CS4670: Computer Vision Noah Snavely Source: S. Lazebnik.
Feature-based object recognition Prof. Noah Snavely CS1114
Fitting a Model to Data Reading: 15.1,
Lecture 8: Image Alignment and RANSAC
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and.
Lecture 10: Robust fitting CS4670: Computer Vision Noah Snavely.
Lecture 12: Structure from motion CS6670: Computer Vision Noah Snavely.
Feature Matching and RANSAC : Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and Rick Szeliski.
כמה מהתעשייה? מבנה הקורס השתנה Computer vision.
Robust fitting Prof. Noah Snavely CS1114
Image Stitching Ali Farhadi CSE 455
CSE 185 Introduction to Computer Vision
CSC 589 Lecture 22 Image Alignment and least square methods Bei Xiao American University April 13.
Computer Vision - Fitting and Alignment
Lecture 12: Image alignment CS4670/5760: Computer Vision Kavita Bala
Advanced Computer Vision Feature-based Alignment Lecturer: Lu Yi & Prof. Fuh CSIE NTU.
IMAGE MOSAICING Summer School on Document Image Processing
Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac.
Example: line fitting. n=2 Model fitting Measure distances.
CSCE 643 Computer Vision: Structure from Motion
Computer Vision : CISC 4/689 Going Back a little Cameras.ppt.
EECS 274 Computer Vision Model Fitting. Fitting Choose a parametric object/some objects to represent a set of points Three main questions: –what object.
Geometric Transformations
Computer Vision - Fitting and Alignment (Slides borrowed from various presentations)
Fitting image transformations Prof. Noah Snavely CS1114
776 Computer Vision Jan-Michael Frahm Spring 2012.
Robust Estimation Course web page: vision.cis.udel.edu/~cv April 23, 2003  Lecture 25.
Optimization and least squares Prof. Noah Snavely CS1114
CSCI 631 – Foundations of Computer Vision March 15, 2016 Ashwini Imran Image Stitching.
Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging.
SIFT.
Lecture 10: Image alignment CS4670/5760: Computer Vision Noah Snavely
Recognizing objects Prof. Noah Snavely CS1114
776 Computer Vision Jan-Michael Frahm Spring 2012.
SIFT Scale-Invariant Feature Transform David Lowe
University of Ioannina
SIFT paper.
Lecture 7: Image alignment
Image transformations
Image transformations
RANSAC and mosaic wrap-up
A special case of calibration
Image Stitching Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi.
Features Readings All is Vanity, by C. Allan Gilbert,
Object recognition Prof. Graeme Bailey
Noah Snavely.
SIFT.
Lecture 8: Image alignment
Lecture 8: Image alignment
Computational Photography
Calibration and homographies
Back to equations of geometric transformations
CS5760: Computer Vision Lecture 9: RANSAC Noah Snavely
CS5760: Computer Vision Lecture 9: RANSAC Noah Snavely
Image Stitching Linda Shapiro ECE/CSE 576.
Lecture 11: Image alignment, Part 2
Image Stitching Linda Shapiro ECE P 596.
Presentation transcript:

Computing transformations Prof. Noah Snavely CS1114

Administrivia  A5 Part 1 due on Friday, A5 Part 2 out soon  Prelim 2 next week, 4/7 (in class) –Covers everything since Prelim 1 –Review session next Monday (time TBA) 2

Bilinear interpolation 3 Q 11 Q 12 Q 21 Q 22 R2R2 R1R1 P y1y1 y2y2 x1x1 x2x2 x y (y, x)

Trilinear interpolation  How do we find the value of the function at C? 4

Invariant local features  Find features that are invariant to transformations –geometric invariance: translation, rotation, scale –photometric invariance: brightness, exposure, … Feature Descriptors (Slides courtesy Steve Seitz)

Object matching in three steps 1.Detect features in the template and search images 2.Match features: find “similar-looking” features in the two images 3.Find a transformation T that explains the movement of the matched features 6 sift

Step 1: Detecting SIFT features  SIFT gives us a set of feature frames and descriptors for an image 7 img = imread(‘futurama.png’); [frames, descs] = sift(img);

Step 2: Matching SIFT features  Answer: for each feature in image 1, find the feature with the closest descriptor in image 2  Called nearest neighbor matching 8

Matching SIFT features  Output of the matching step: Pairs of matching points [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ] [ x 2 y 2 ]  [ x 2 ’ y 2 ’ ] [ x 3 y 3 ]  [ x 3 ’ y 3 ’ ] … [ x k y k ]  [ x k ’ y k ’ ] 9

Step 3: Find the transformation  How do we draw a box around the template image in the search image?  Key idea: there is a transformation that maps template  search image! 10

Image transformations  2D affine transformation 11

Solving for image transformations  Given a set of matching points between image 1 and image 2… … can we solve for an affine transformation T mapping 1 to 2? 12

Solving for image transformations  T maps points in image 1 to the corresponding point in image 2 13 (1,1,1)

How do we find T ?  We already have a bunch of point matches [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ] [ x 2 y 2 ]  [ x 2 ’ y 2 ’ ] [ x 3 y 3 ]  [ x 3 ’ y 3 ’ ] … [ x k y k ]  [ x k ’ y k ’ ]  Solution: Find the T that best agrees with these known matches  This problem is a form of (linear) regression 14

An Algorithm: Take 1 1.To find T, randomly guess a, b, c, d, e, f, check how well T matches the data 2.If it matches well, return T 3.Otherwise, go to step 1  The “snailsort” method  We can do much better 15

Linear regression  Simplest case: fitting a line 16

Linear regression  But what happens here? 17 What does this remind you of ?

Linear regression  Simplest case: just 2 points 18

Linear regression  Simplest: just 2 points  Want to find a line y = mx + b  x 1  y 1, x 2  y 2  This forms a linear system: y 1 = mx 1 + b y 2 = mx 2 + b  x’s, y’s are knowns  m, b are unknown  Very easy to solve 19

Multi-variable linear regression  What about 2D affine transformations? –maps a 2D point to another 2D point  We have a set of matches [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ] [ x 2 y 2 ]  [ x 2 ’ y 2 ’ ] [ x 3 y 3 ]  [ x 3 ’ y 3 ’ ] … [ x 4 y 4 ]  [ x 4 ’ y 4 ’ ] 20

 Consider just one match [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ] ax 1 + by 1 + c = x 1 ’ dx 1 + ey 1 + f = y 1 ’  2 equations, 6 unknowns  we need 3 matches 21 Multi-variable linear regression

Finding an affine transform  This is just a bigger linear system, still (relatively) easy to solve  Really just two linear systems with 3 equations each (one for a,b,c, the other for d,e,f)  We’ll figure out how to solve this in a minute 22

An Algorithm: Take 2  We have many more than three matches  Some are correct, many are wrong  Idea 2: select three matches at random, compute T 23

An Algorithm: Take 2  Better then randomly guessing a,b,c,d,e,f  What could go wrong? 24

Robustness  Suppose 1/3 of the matches are wrong  We select three at random  The probability of at least one selected match being wrong is ?  If we get just one match wrong, the transformation could be wildly off  (The Arnold Schwarzenegger problem)  How do we fix this? 25

Fixing the problem  First observation:  There is a way to test how good the transformation we get is (how?) 26

Testing goodness  A good transformation will agree with most of the matches  A bad transformation will disagree with most of the matches  How can we tell if a match agrees with the transformation T? [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ]  Compute the distance between and 27

Testing goodness  Find the distance between and  If the distance is small, we call this match an inlier to T  If the distance is large, it’s an outlier to T  For a correct match and transformation, this distance will be close to (but not exactly) zero  For an incorrect match or transformation, this distance will probably be large 28

Testing goodness 29 inlier outlier

Testing goodness % define a threshold thresh = 5.0; % 5 pixels num_agree = 0; diff = T * [x1 y1 1]’ – [x1p y1p 1]’; if norm(diff) < thresh num_agree = num_agree + 1; 30

Finding T, third attempt 1.Select three points at random 2.Solve for the affine transformation T 3.Count the number of inlier matches to T 4.If T is has the highest number of inliers so far, save it 5.Repeat for N rounds, return the best T 31

Testing goodness  This algorithm is called RANSAC (RANdom SAmple Consensus)  Used in an amazing number of computer vision algorithms  Requires two parameters: –The agreement threshold –The number of rounds (how many do we need?) 32

How do we solve for T?  Given three matches, we have a linear system with six equations: ax 1 + by 1 + c = x 1 ’ dx 1 + ey 1 + f = y 1 ’ ax 2 + by 2 + c = x 2 ’ dx 2 + ey 2 + f = y 2 ’ ax 3 + by 3 + c = x 3 ’ dx 3 + ey 3 + f = y 3 ’ 33 [ x 1 y 1 ]  [ x 1 ’ y 1 ’ ] [ x 2 y 2 ]  [ x 2 ’ y 2 ’ ] [ x 3 y 3 ]  [ x 3 ’ y 3 ’ ]

Two 3x3 linear systems ax 1 + by 1 + c = x 1 ’ ax 2 + by 2 + c = x 2 ’ ax 3 + by 3 + c = x 3 ’ dx 1 + ey 1 + f = y 1 ’ dx 2 + ey 2 + f = y 2 ’ dx 3 + ey 3 + f = y 3 ’ 34

Solving a 3x3 system ax 1 + by 1 + c = x 1 ’ ax 2 + by 2 + c = x 2 ’ ax 3 + by 3 + c = x 3 ’  We can write this in matrix form:  Now what? 35

Putting it all together 1.Select three points at random 2.Solve for the affine transformation T 3.Count the number of inlier matches to T 4.If T is has the highest number of inliers so far, save it 5.Repeat for N rounds, return the best T 36

Finding the object boundary 37