1. Computational Photography
Prof. Feng Liu
Spring 2015
04/22/2015
Computer Graphics

2. Last Time
Re-lighting
HDR

3. Today Panorama Mid-term project presentation Overview
Feature detection
Mid-term project presentation
Not real mid-term
4 minutes presentation
Schedule
May 6
With slides by Prof. C. Dyer and K. Grauman

4. Panorama Building: History
Along the River During Ching Ming Festival by Z.D Zhang ( )
San Francisco from Rincon Hill, 1851, by Martin Behrmanx

5. Panorama Building: A Concise History
The state of the art and practice is good at assembling images into panoramas
Mid 90s –Commercial Players (e.g. QuicktimeVR)
Late 90s –Robust stitchers(in research)
Early 00s –Consumer stitching common
Mid 00s –Automation

6. Stitching Recipe Align pairs of images Align all to a common frame
Adjust (Global) & Blend

7. Stitching Images Together

8. When do two images “stitch”?

9. Images can be transformed to match

10. Images are related by Homographies

11. Compute Homographies

12. Automatic Feature Points Matching
Match local neighborhoods around points
Use descriptors to efficiently compare SIFT
[Lowe 04] most common choice

13. Stitching Recipe Align pairs of images Align all to a common frame
Adjust (Global) & Blend

14. Wide Baseline Matching
Images taken by cameras that are far apart make the correspondence problem very difficult
Feature-based approach: Detect and match feature points in pairs of images
Credit: C. Dyer
Credit: C. Dyer

15. Matching with Features
Detect feature points
Find corresponding pairs
Slide credit: C. Dyer
Credit: C. Dyer

16. Matching with Features
Problem 1:
Detect the same point independently in both images
no chance to match!
We need a repeatable detector
Credit: C. Dyer

17. Matching with Features
Problem 2:
For each point correctly recognize the corresponding point ?
We need a reliable and distinctive descriptor
Credit: C. Dyer

18. Properties of an Ideal Feature
Local: features are local, so robust to occlusion and clutter (no prior segmentation)
Invariant (or covariant) to many kinds of geometric and photometric transformations
Robust: noise, blur, discretization, compression, etc. do not have a big impact on the feature
Distinctive: individual features can be matched to a large database of objects
Quantity: many features can be generated for even small objects
Accurate: precise localization
Efficient: close to real-time performance
Credit: C. Dyer

19. Problem 1: Detecting Good Feature Points
[Image from T. Tuytelaars ECCV 2006 tutorial]
Credit: C. Dyer

20. Feature Detectors Hessian Harris Lowe: SIFT (DoG)
Mikolajczyk & Schmid: Hessian/Harris-Laplacian/Affine
Tuytelaars & Van Gool: EBR and IBR
Matas: MSER
Kadir & Brady: Salient Regions
Others
Credit: C. Dyer

21. Harris Corner Point Detector
C. Harris, M. Stephens, “A Combined Corner and Edge Detector,” 1988
Credit: C. Dyer

22. Harris Detector: Basic Idea
We should recognize the point by looking through a small window
Shifting a window in any direction should give a large change in response
Credit: C. Dyer

23. Harris Detector: Basic Idea
“flat” region: no change in all directions
“edge”: no change along the edge direction
“corner”: significant change in all directions
Credit: C. Dyer

24. Harris Detector: Derivation
Change of intensity for a (small) shift by [u,v] in image I:
Intensity
Weighting function
Shifted intensity or
Weighting function w(x,y) =
Gaussian
1 in window, 0 outside
Credit: R. Szeliski

25. Harris Detector Apply 2nd order Taylor series expansion:
Credit: R. Szeliski

26. Harris Corner Detector
Expanding E(u,v) in a 2nd order Taylor series, we have, for small shifts, [u,v], a bilinear approximation:
where M is a 2  2 matrix computed from image derivatives:
Note: Sum computed over small neighborhood around given pixel
Credit: R. Szeliski

27. Harris Corner Detector
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of M
direction of the fastest change
Ellipse E(u,v) = const
direction of the slowest change
(max)-1/2
(min)-1/2
Credit: R. Szeliski

28. Selecting Good Features
Image patch
SSD surface
l1 and l2 both large
Credit: C. Dyer

29. Selecting Good Features
SSD surface
large l1, small l2
Credit: C. Dyer

30. Selecting Good Features
SSD surface
small l1, small l2
Credit: C. Dyer

31. Harris Corner Detector2
“Edge” 2 >> 1
“Corner” 1 and 2 both large, 1 ~ 2; E increases in all directions
Classification of image points using eigenvalues of M:
1 and 2 are small; E is almost constant in all directions
“Edge” 1 >> 2
“Flat” region 1
Credit: C. Dyer

32. Harris Corner Detector
Measure of corner response:
k is an empirically-determined constant; e.g., k = 0.05
Credit: C. Dyer

33. Harris Corner Detector2
“Edge”
“Corner”
R depends only on eigenvalues of M
R is large for a corner
R is negative with large magnitude for an edge
|R| is small for a flat region
R < 0
R > 0
“Flat”
“Edge”
|R| small
R < 0 1
Credit: C. Dyer

34. Harris Corner Detector: Algorithm
Find points with large corner response function R
(i.e., R > threshold)
Take the points of local maxima of R (for localization) by non-maximum suppression
Credit: C. Dyer

35. Harris Detector: Example
Credit: C. Dyer

36. Harris Detector: Example
Compute corner response R = 12 – k(1 + 2)2
Credit: C. Dyer

37. Harris Detector: Example
Find points with large corner response: R > threshold
Credit: C. Dyer

38. Harris Detector: Example
Take only the points of local maxima of R
Credit: C. Dyer

39. Harris Detector: Example
Credit: C. Dyer

40. Harris Detector: Example
Interest points extracted with Harris (~ 500 points)
Credit: C. Dyer

41. Harris Detector: Example
Credit: C. Dyer

42. Harris Detector: Summary
Average intensity change in direction [u,v] can be expressed in bilinear form:
Describe a point in terms of eigenvalues of M: measure of corner response:
A good (corner) point should have a large intensity change in all directions, i.e., R should be a large positive value
Credit: C. Dyer

43. Harris Detector Properties
Rotation invariance
Ellipse rotates but its shape (i.e., eigenvalues) remains the same
Corner response R is invariant to image rotation
Credit: C. Dyer

44. Harris Detector Properties
But not invariant to image scale
Fine scale: All points will be classified as edges
Coarse scale: Corner
Credit: C. Dyer

45. Harris Detector Properties
Quality of Harris detector for different scale changes
Repeatability rate:
# correct correspondences # possible correspondences
C. Schmid et al., “Evaluation of Interest Point Detectors,” IJCV 2000
Credit: C. Dyer

46. Invariant Local Features
Goal: Detect the same interest points regardless of image changes due to translation, rotation, scale, viewpoint

47. Models of Image Change Geometry Photometry Rotation
Similarity (rotation + uniform scale)
Affine (scale dependent on direction) valid for: orthographic camera, locally planar object
Photometry
Affine intensity change (I  a I + b)
Credit: C. Dyer

48. SIFT Detector [Lowe ’04]
Difference-of-Gaussian (DoG) is an approximation of the Laplacian-of-Gaussian (LoG)  =
Lowe, D. G., “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, 60, 2, pp , 2004
Credit: C. Dyer

49. SIFT Detector
Credit: C. Dyer

50. SIFT Detector
Credit: C. Dyer

51. SIFT Detector Algorithm Summary
Detect local maxima in position and scale of squared values of difference-of-Gaussian
Fit a quadratic to surrounding values for sub-pixel and sub-scale interpolation
Output = list of (x, y, ) points
Credit: C. Dyer

52. References on Feature Descriptors
A performance evaluation of local descriptors, K. Mikolajczyk and C. Schmid, IEEE Trans. PAMI 27(10), 2005
Evaluation of features detectors and descriptors based on 3D objects, P. Moreels and P. Perona, Int. J. Computer Vision 73(3), 2007
Credit: C. Dyer

53. Next Time
Panorama
Feature matching