1. Vision Review: Image Processing
September 17, 2010

2. Announcements
Homework and paper presentation guidelines are up on web page
Readings for next Tuesday: Chapters 6, 11.1, and 18
For next Thursday: “Stochastic Road Shape Estimation”

3. Computer Vision Review Outline
Image formation
Image processing
Motion & Estimation
Classification
sort of in order from light being emitted to the brain understanding

4. Outline Images Binary operators Filtering Modeling, matching
Smoothing
Edge, corner detection
Modeling, matching
Scale space

5. Images An image is a matrix of pixels Note: Matlab uses Resolution
Digital cameras: 1600 X 1200 at a minimum
Video cameras: ~640 X 480
Grayscale: generally 8 bits per pixel  Intensities in range [0…255]
RGB color: 3 8-bit color planes

6. Image Conversion
RGB  Grayscale: Mean color value, or weight by perceptual importance (Matlab: rgb2gray)
Grayscale  Binary: Choose threshold based on histogram of image intensities (Matlab: imhist)
right image is center image thresholded at 128

7. Color Representation RGB, HSV (hue, saturation, value), YUV, etc.
Luminance: Perceived intensity
Chrominance: Perceived color
HS(V), (Y)UV, etc.
Normalized RGB removes some illumination dependence:

8. Binary Operations Dilation, erosion (Matlab: imdilate, imerode)
Dilation: All 0’s next to a 1  1 (Enlarge foreground)
Erosion: All 1’s next to a 0  0 (Enlarge background)
Connected components
Uniquely label each n-connected region in binary image
4- and 8-connectedness
Matlab: bwfill, bwselect
Moments: Region statistics
Zeroth-order: Size
First-order: Position (centroid)
Second-order: Orientation

9. Image Transformations
Geometric: Compute new pixel locations
Rotate
Scale
Undistort (e.g., radial distortion from lens)
Photometric: How to compute new pixel values when non-integral
Nearest neighbor: Value of closest pixel
Bilinear interpolation (2 x 2 neighborhood)
Bicubic interpolation (4 x 4)

10. Bilinear Interpolation
Idea: Blend four pixel values surrounding source, weighted by nearness
Vertical blend
Horizontal blend

11. Image Comparison: SSD
Given a template image and an image , how to quantify the similarity between them for a given alignment?
Sum of squared differences (SSD)
image similarity not the same as scene similarity, but we’d like it to be

12. Cross-Correlation for Template Matching
Note that SSD formula can be written:
First two terms fixed  last term measures mismatch—the cross-correlation:
In practice, normalize by image magnitude when shifting template to search for matches

13. Filtering
Idea: Analyze neighborhood around some point in image with filter function ; put result in new image at corresponding location
System properties
Shift invariance: Same inputs give same outputs, regardless of location
Superposition: Output on sum of images =
Sum of outputs on separate images
Scaling: Output on scaled image = Scaled output on image
Linear shift invariance  Convolution

14. Convolution Definition: Shorter notation: Properties
Commutative
Associative
Fourier theorem: Convolution in spatial domain = Multiplication in frequency domain
More on Fourier transforms on Thursday

15. Discrete Filtering 1 -1 2
Linear filter: Weighted sum of pixels over rectangular neighborhood—kernel defines weights
Think of kernel as template being matched by correlation (Matlab: imfilter, filter2)
Convolution: Correlation with kernel rotated 180
Matlab: conv2
Dealing with image edges
Zero-padding
Border replication

16. Filtering Example 1: 1 -1 2 2 3 1
Rotate 1 -1 2

17. 1 1 1 2 2 2 3
Step 1 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 -1 3 2 1 4 2 5 -1 -2 1

18. 1 1 1 2 2 2 3
Step 2 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 3 2 1 -2 4 2 4 5 -2 -1 3

19. 1 1 1 2 2 2 3
Step 3 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 3 2 1 -2 4 3 4 5 -1 -3 3

20. 1 1 1 2 2 2 3
Step 4 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 3 2 1 -2 6 1 4 -2 5 -3 -3 1

21. 1 1 1 2 2 2 3
Step 5 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 3 2 1 2 2 4 9 -2 5 -1 4 1 -1 -2 2

22. 1 1 1 2 2 2 3
Step 6 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 2 3 2 1 2 2 6 4 9 -2 5 -2 2 3 -2 -2 1

23. Final Result 12 7 6 4 8 14 5 9 11 -2

24. Separability
Definition: 2-D kernel can be written as convolution of two 1-D kernels
Advantage: Efficiency—Convolving image with kernel requires multiplies vs for non-separable kernel

25. Smoothing (Low-Pass) Filters
Replace each pixel with average of neighbors
Benefits: Suppress noise, aliasing
Disadvantage: Sharp features blurred
Types
Mean filter (box)
Median (nonlinear)
Gaussian 1
3 x 3 box filter

26. Box Filter: Smoothing
Original image
7 x 7 kernel

27. Gaussian Kernel
Idea: Weight contributions of neighboring pixels by nearness
Matlab: fspecial(‘gaussian’,…)

28. Gaussian: Benefits
Rotational symmetry treats features of all orientations equally (isotropy)
Smooth roll-off reduces “ringing”
Efficient: Rule of thumb is kernel width  5
Separable
Cascadable: Approach to large  comes from identity

29. Gaussian: Smoothing
Original
image
7 x 7
kernel
 = 1
 = 3

30. Gradient
Think of image intensities as a function Gradient of image is a vector field as for a normal 2-D height function:
Edge: Place where gradient magnitude is high; orthogonal to gradient direction

31. Edge Causes Depth discontinuity Surface orientation discontinuity
Reflectance discontinuity (i.e., change in surface material properties)
Illumination discontinuity (e.g., shadow)

32. Edge Detection Edge Types Searching for Edges: Step edge (ramp)
Line edge (roof)
Searching for Edges:
Filter: Smooth image
Enhance: Apply numerical derivative approximation
Detect: Threshold to find strong edges
Localize/analyze: Reject spurious edges, include weak but justified edges

33. Step edge detection First derivative edge detectors: Look for extrema
Sobel operator (Matlab: edge(I, ‘sobel’))
Prewitt, Roberts cross
Derivative of Gaussian
Second derivative: Look for zero-crossings
Laplacian : Isotropic
Second directional derivative
Laplacian of Gaussian/Difference of Gaussians -1 1 -2 2 -1 -2 1 2
Sobel x
Sobel y

34. Derivative of Gaussian

35. Laplacian of Gaussian
Matlab: fspecial(‘log’,…)

36. Edge Filtering Example1 -1 2 -2 2
Rotate 1 -1 2 -2

37. -1 1 2 2
Step 1 -2 2 2 2 -1 1 2 2 2 2 -1 1 -2 1 2 -1

38. -1 1 2 2
Step 2 -2 2 2 2 -1 1 2 2 2 2 -1 1 1 2 3 4 6 2

39. -1 1 2 2
Step 3 -2 2 2 2 -1 1 2 2 2 2 -1 1 1 2 3 4 6 2

40. -1 1 2 2
Step 4 -2 2 2 2 -1 1 2 2 2 2 -1 1 2 3 -4 2 6 -6
edge
effect -2 1

41. Sobel Edge Detection: Gradient Approximation
Horizontal
Vertical

42. Sobel vs. LoG Edge Detection: Matlab Automatic Thresholds

43. Canny Edge Detection Derivative of Gaussian Non-maximum suppression
Thin multi-pixel wide “ridges” down to single pixel
Thresholding
Low, high edge-strength thresholds
Accept all edges over low threshold that are connected to edge over high threshold
Matlab: edge(I, ‘canny’)

44. Canny Edge Detection: Example
note painted lines on road preserved, whereas others didn’t
(Matlab automatically set thresholds)

45. Corner Detection
Basic idea: Find points where two edges meet—i.e., high gradient in orthogonal directions
Examine gradient over window (Shi & Tomasi, 1994)
Edge strength encoded by eigenvalues ; corner is where over threshold
Harris corners (Harris & Stephens, 1988), Susan corners (Smith & Brady, 1997)

46. Example: Corner Detection
courtesy of S. Smith
SUSAN corners

47. Edge-Based Image Comparison
Chamfer, Hausdorff distance, etc.
Transform edge map based on distance to nearest edge before correlating as usual
courtesy of D. Gavrila

48. Scale Space How thick an edge? How big a dot?
Must consider what scale we are interested in when designing filters
Efficiency a major consideration: Fine-grained template matching is expensive over a full image

49. Image Pyramids
Idea: Represent image at different scales, allowing efficient coarse-to-fine search
Downsampling:
Simplest scale change: Decimation—just downsample
from Forsyth & Ponce

50. Gaussian, Laplacian Pyramids
Gaussian pyramid of image: and
Laplacian pyramid
Difference of image and Gaussian at each level of
Upsample with interpolation to reconstruct
DoG approximates Laplacian
courtesy
of Wolfram
Gaussian pyramid
Laplacian pyramid

51. Color-based Image Comparison
Color histograms (Swain & Ballard, 1991)
Steps
Histogram RGB/HSV triplets over two images to be compared
Normalize each histogram by respective total number of pixels to get frequencies
Similarity is Euclidean distance between color frequency vectors
Sensitive to lighting changes
Works for different-sized images
Matlab: imhist, hist