Lecture 4 FEATURE DETECTION Computer Vision Lecture 4 FEATURE DETECTION

Today’s Topics Feature Detection Feature Description Feature Matching

Questions of Interest How to find camera parameters? Where is the camera, where is it directed at? What is the movement of the camera? Where are the objects located in 3D? What are the dimensions of objects in 3D? What is the 3D structure of a scene? How to process stereo video? How to detect and match image features? How to stitch images?

Introduction to Feature Detection

Image Mozaics

Object Recognition

Image Matching

Image Matching using Feature Points NASA Mars Rover images

Feature Points are Used for … Image alignment (e.g., mosaics) 3D reconstruction Motion tracking Object recognition Indexing and database retrieval Robot navigation …

Advantages of Local Features Locality features are local, so robust to occlusion and clutter Distinctiveness can differentiate a large database of objects Quantity hundreds or thousands in a single image Efficiency real-time performance achievable Generality exploit different types of features in different situations

Point vs. Line Features

Processing Stages Feature detection/extraction (look for stable/accurate features) Feature description (aim for invariance to transformations) Feature matching (for photo collection) or tracking (for video)

Feature Detection

Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters Features Descriptors

Choosing interest points Where would you tell your friend to meet you? Corners!

Choosing interest points Where would you tell your friend to meet you? Blobs!

Stable (Good) Features: High Accuracy in Localization

Stable (Good) Features Aperture problem Good feature “flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions (uniqueness)

Autocorrelation: Indicator of “Good”ness

Autocorrelation Calculation Autocorrelation can be approximated by sum-squared-difference (SSD):

Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: I(x, y) E(u, v) E(3,2) w(x, y)

Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: I(x, y) E(u, v) E(0,0) w(x, y)

Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: Intensity Window function Shifted intensity or Window function w(x,y) = Gaussian 1 in window, 0 outside Source: R. Szeliski

Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: We want to find out how this function behaves for small shifts E(u, v)

Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: We want to find out how this function behaves for small shifts Local quadratic approximation of E(u,v) in the neighborhood of (0,0) is given by the second-order Taylor expansion:

Taylor Expansion

Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

Corner Detection: Mathematics The quadratic approximation simplifies to where M is a second moment matrix computed from image derivatives: M

Corners as distinctive interest points 2 x 2 matrix of image derivatives (averaged in neighborhood of a point). Notation:

Interpreting the second moment matrix The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape.

Interpreting the second moment matrix First, consider the axis-aligned case (gradients are either horizontal or vertical) If either λ is close to 0, then this is not a corner, so look for locations where both are large.

Interpreting the second moment matrix Consider a horizontal “slice” of E(u, v): This is the equation of an ellipse.

Interpreting the second moment matrix Consider a horizontal “slice” of E(u, v): This is the equation of an ellipse. Diagonalization of M: The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R direction of the slowest change direction of the fastest change (max)-1/2 (min)-1/2

Interpreting the eigenvalues Classification of image points using eigenvalues of M: 2 “Edge” 2 >> 1 “Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Flat” region 1

Corner response function α: constant (0.04 to 0.06) “Edge” R < 0 “Corner” R > 0 |R| small “Edge” R < 0 “Flat” region

Harris corner detector Compute M matrix for each image window to get their cornerness scores. Find points whose surrounding window gave large corner response (f> threshold) Take the points of local maxima, i.e., perform non-maximum suppression C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.  38

Harris Detector [Harris88] 1. Image derivatives Ix Iy Second moment matrix (optionally, blur first) Ix2 Iy2 IxIy 2. Square of derivatives g(IxIy) 3. Gaussian filter g(sI) g(Ix2) g(Iy2) 4. Cornerness function – both eigenvalues are strong 5. Non-maxima suppression har

Use Sobel Operator for Gradient Computation Gaussian derivative of Gaussian -1 1 -2 2 1 2 -1 -2 Horizontal derivative Vertical derivative

Harris Detector Example

f value (red = high, blue = low)

Threshold (f > value)

Find Local Maxima of f

Harris Features (in red)

Many Existing Detectors Available Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG [Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others… K. Grauman, B. Leibe

Feature Description

Feature Descriptor How can we extract feature descriptors that are invariant to variations (transformations) and yet still discriminative enough to establish correct correspondences? ?

Transformation Invariance Find features that are invariant to transformations geometric invariance: translation, rotation, scale photometric invariance: brightness, exposure

Transformation Invariance We would like to find the same features regardless of the transformation between images. This is called transformational invariance. Most feature methods are designed to be invariant to Translation, 2D rotation, scale They can usually also handle Limited affine transformations (some are fully affine invariant) Limited 3D rotations (SIFT works up to about 60 degrees) Limited illumination/contrast changes n차원 공간이 1차식으로 나타내어지는 점대응(點對應) (x,y)→(x', y')을 말한다. 일반적으로 평면 위의 임의의 직선좌표계에 관하여         x'＝a₁x＋b₁y＋c₁         y'＝a₂x＋b₂y＋c₂ (a₁b₂－a₂b₁≠0) 라는 꼴의 1차식으로 나타내어진다. 의사변환(擬似變換)이라고도 한다. 일반적으로 n차원 공간에서 점대응 (x₁,x₂,…,xn) → (y₁,y₂…,yn) 이 1차식 yi = ai1x1 + ai2x2 + … + ainxn 인 꼴로 나타내어질 때, 이 대응을 아핀변환이라고 한다. In geometry, an affine transformation , affine map or an affinity (from the Latin, affinis , "connected with") is a function between affine spaces which preserves points, straight lines and planes. Also, sets of parallel lines remain parallel after an affine transformation. An affine transformation does not necessarily preserve angles between lines...

Types of invariance Illumination

Types of invariance Illumination Scale

Types of invariance Illumination Scale Rotation

Types of invariance Illumination Scale Rotation Affine

Types of invariance Illumination Scale Rotation Affine Full Perspective

How to Achieve Invariance Make sure the detector is invariant Harris is invariant to translation and rotation Scale is trickier A common approach is to detect features at many scales using a Gaussian pyramid (e.g., MOPS) More sophisticated methods find “the best scale” to represent each feature (e.g., SIFT) Design an invariant feature descriptor A descriptor captures the information in a region around the detected feature point The simplest descriptor: a square window of pixels What’s this invariant to? Let’s look at some better approaches…

Rotational Invariance for Feature Descriptors Find dominant orientation of the image patch. This is given by e+, the eigenvector of M corresponding to + Rotate the patch to horizontal direction Intensity normalize the window by subtracting the mean, dividing by the standard deviation in the window 40 pixels 8 pixels CSE 576: Computer Visio Adapted from slide by Matthew Brown

Scale Invariance Detecting features at the finest stable scale possible may not always be appropriate e.g., when matching images with little high frequency detail (e.g., clouds), fine-scale features may not exist. One solution to the problem is to extract features at a variety of scales e.g., by performing the same operations at multiple resolutions in a pyramid and then matching features at the same level. This kind of approach is suitable when the images being matched do not undergo large scale changes e.g., when matching successive aerial images taken from an airplane or stitching panoramas taken with a fixed-focal-length camera.

Scale Invariance for Feature Descriptors

Scale Invariant Detection Key idea: find scale that gives local maximum of f f is a local maximum in both position and scale

Scale Invariant Feature Transform (SIFT) Features In order to detect the local maxima and minima of f, each sample point is compared to its eight neighbors in the current image and nine neighbors in the scale above and below It is selected only if it is larger than all of these neighbors or smaller than all of them. Instead of extracting features at many different scales and then matching all of them, it is more efficient to extract features that are stable in both location and scale.

Scale Invariant Feature Transform Take 16x16 square window around detected feature Compute edge orientation (angle of the gradient - 90) for each pixel Throw out weak edges (threshold gradient magnitude) Create histogram of surviving edge orientations 2 angle histogram Adapted from slide by David Lowe

SIFT Descriptor Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) Compute an orientation histogram for each cell 16 cells * 8 orientations = 128 dimensional descriptor Adapted from slide by David Lowe

Properties of SIFT Extraordinarily robust matching technique Can handle changes in viewpoint Up to about 60 degree out of plane rotation Can handle significant changes in illumination Sometimes even day vs. night (below) Fast and efficient—can run in real time Lots of code available http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

Feature Matching

Feature Matching Given a feature in image I1, how to find the best match in image I2? Define a distance function that compares two descriptors Test all the features in I2, find the one with minimum distance

Feature Distance f1 f2 f2' I2 I1 SSD is commonly used to measure the difference between two features f1 and f2 Use ratio-of-distance = SSD(f1, f2) / SSD(f1, f2’) as a measure of reliability of the feature match: f2 is best SSD match to f1 in I2 f2’ is 2nd best SSD match to f1 in I2 Ratio-of-distance is large for ambiguous matches f1 f2 f2' I2 I1

Evaluating Feature Matching Performance

Evaluating the Results How can we measure the performance of a feature matcher? 50 75 200 feature distance

True/false positives The distance threshold affects performance 50 75 True positives = # of detected matches that are correct Suppose we want to maximize these—how to choose threshold? False positives = # of detected matches that are incorrect Suppose we want to minimize these—how to choose threshold? 50 75 200 feature distance false match true match

Terminology True positives = # of detected matching features False negatives= # of missed matching features False positives = # of detected not matching features True negatives = # of missed not matching features Detected to have a match Matching features Detected to have a match

Recall, Precision, and F-Number Fraction of detected true positives among all positives fraction of relevant instances that are retrieved Fraction of true positives among all detected as positives fraction of retrieved instances that are relevant Approaches to 1 only when both recall and precision approach to 1

True and False Positive Rates Fraction of detected true positives among all positives Fraction of detected false positives among all negatives

Evaluating the Performance of a Feature Matcher true positive rate 0.7 1 0.1 false positive rate

Receiver Operating Characteristic (ROC) Curve 0.7 1 false positive rate true positive rate 0.1 Generated by counting # current/incorrect matches, for different thresholds Want to maximize area under the curve (AUC) Useful for comparing different feature matching methods For more info: http://en.wikipedia.org/wiki/Receiver_operating_characteristic