1. Vision Review: Image Processing Course web page: www.cis.udel.edu/~cer/arv September 17, 2002

2. Announcements Homework and paper presentation guidelines are up on web page Readings for next Tuesday: Chapters 6, 11.1, and 18 For next Thursday: “Stochastic Road Shape Estimation”

3. Computer Vision Review Outline Image formation Image processing Motion & Estimation Classification

4. Outline Images Binary operators Filtering –Smoothing –Edge, corner detection Modeling, matching Scale space

5. Images An image is a matrix of pixels Note: Matlab uses Resolution –Digital cameras: 1600 X 1200 at a minimum –Video cameras: ~640 X 480 Grayscale: generally 8 bits per pixel  Intensities in range [0…255] RGB color: 3 8-bit color planes

6. Image Conversion RGB  Grayscale: Mean color value, or weight by perceptual importance (Matlab: rgb2gray ) Grayscale  Binary: Choose threshold based on histogram of image intensities (Matlab: imhist )

7. Color Representation RGB, HSV (hue, saturation, value), YUV, etc. Luminance: Perceived intensity Chrominance: Perceived color –HS(V), (Y)UV, etc. –Normalized RGB removes some illumination dependence:

8. Binary Operations Dilation, erosion (Matlab: imdilate, imerode ) –Dilation: All 0’s next to a 1  1 (Enlarge foreground) –Erosion: All 1’s next to a 0  0 (Enlarge background) Connected components –Uniquely label each n-connected region in binary image –4- and 8-connectedness –Matlab: bwfill, bwselect Moments: Region statistics –Zeroth-order: Size –First-order: Position (centroid) –Second-order: Orientation

9. Image Transformations Geometric: Compute new pixel locations –Rotate –Scale –Undistort (e.g., radial distortion from lens) Photometric: How to compute new pixel values when non-integral –Nearest neighbor: Value of closest pixel –Bilinear interpolation (2 x 2 neighborhood) –Bicubic interpolation (4 x 4)

10. Bilinear Interpolation Idea: Blend four pixel values surrounding source, weighted by nearness Vertical blend Horizontal blend

11. Image Comparison: SSD Given a template image and an image, how to quantify the similarity between them for a given alignment? Sum of squared differences (SSD)

12. Cross-Correlation for Template Matching Note that SSD formula can be written: First two terms fixed  last term measures mismatch—the cross-correlation: In practice, normalize by image magnitude when shifting template to search for matches

13. Filtering Idea: Analyze neighborhood around some point in image with filter function ; put result in new image at corresponding location System properties –Shift invariance: Same inputs give same outputs, regardless of location –Superposition: Output on sum of images = –Sum of outputs on separate images –Scaling: Output on scaled image = Scaled output on image Linear shift invariance  Convolution

14. Convolution Definition: Shorter notation: Properties –Commutative –Associative Fourier theorem: Convolution in spatial domain = Multiplication in frequency domain –More on Fourier transforms on Thursday

15. Discrete Filtering Linear filter: Weighted sum of pixels over rectangular neighborhood—kernel defines weights Think of kernel as template being matched by correlation (Matlab: imfilter, filter2 ) Convolution: Correlation with kernel rotated 180  –Matlab: conv2 Dealing with image edges –Zero-padding –Border replication 111 21 1

16. Filtering Example 1: 1 12 111 2223 2133 2212 1322 Rotate 1 12 111

17. Step 1 3 2 1 2 2 1 3 2 32 21 22 325 3 2 1 2 2 1 3 2 32 21 22 32 1-2 24 111 1 12 111

18. Step 2 3 2 1 2 2 1 3 2 32 21 22 3245 3 2 1 2 2 1 3 2 32 21 22 32 3-2 24 111 1 12 111

19. Step 3 3 2 1 2 2 1 3 2 32 21 22 32445 3 2 1 2 2 1 3 2 32 21 22 32 3-3 34-2 111 1 12 111

20. Step 4 3 2 1 2 2 1 3 2 32 21 22 3244-25 3 2 1 2 2 1 3 2 32 21 22 32 1-3 16-2 111 1 12 111

21. Step 5 3 2 1 2 2 1 3 2 32 21 22 3244 9 -25 3 2 1 2 2 1 3 2 32 21 22 32 2 14 221 1 12 111

22. Step 6 3 2 1 2 2 1 3 2 32 21 22 32 6 44 9 -25 3 2 1 2 2 1 3 2 32 21 22 32 1 32 222 1 12 111

23. Final Result 12 7 6 4 8 6 14 4 59 59 511 -25

24. Separability Definition: 2-D kernel can be written as convolution of two 1-D kernels Advantage: Efficiency—Convolving image with kernel requires multiplies vs. for non- separable kernel

25. Smoothing (Low-Pass) Filters Replace each pixel with average of neighbors Benefits: Suppress noise, aliasing Disadvantage: Sharp features blurred Types –Mean filter (box) –Median (nonlinear) –Gaussian 111 111 111 3 x 3 box filter

26. Box Filter: Smoothing 7 x 7 kernelOriginal image

27. Gaussian Kernel Idea: Weight contributions of neighboring pixels by nearness Matlab: fspecial(‘gaussian’,…)

28. Gaussian: Benefits Rotational symmetry treats features of all orientations equally (isotropy) Smooth roll-off reduces “ringing” Efficient: Rule of thumb is kernel width  5  –Separable –Cascadable: Approach to large  comes from identity

29. Gaussian: Smoothing  = 1  = 3 7 x 7 kernel Original image

30. Gradient Think of image intensities as a function. Gradient of image is a vector field as for a normal 2-D height function: Edge: Place where gradient magnitude is high; orthogonal to gradient direction

31. Edge Causes Depth discontinuity Surface orientation discontinuity Reflectance discontinuity (i.e., change in surface material properties) Illumination discontinuity (e.g., shadow)

32. Edge Detection Edge Types –Step edge (ramp) –Line edge (roof) Searching for Edges: –Filter: Smooth image –Enhance: Apply numerical derivative approximation –Detect: Threshold to find strong edges –Localize/analyze: Reject spurious edges, include weak but justified edges

33. Step edge detection First derivative edge detectors: Look for extrema –Sobel operator (Matlab: edge(I, ‘sobel’) ) –Prewitt, Roberts cross –Derivative of Gaussian Second derivative: Look for zero-crossings –Laplacian : Isotropic –Second directional derivative –Laplacian of Gaussian/Difference of Gaussians -2 000 121 01 -202 01 Sobel xSobel y

34. Derivative of Gaussian

35. Laplacian of Gaussian Matlab: fspecial(‘log’,…)

36. Edge Filtering Example 10 20-2 10 0022 0022 0022 0022 Rotate 10 20-2 10

37. Step 1 0 0 1 2 2 1 2 2 22 20 20 220 0 0 0 0 2 2 2 2 20 20 20 20 00 00-2 10 10 20-2 10

38. Step 2 0 0 1 2 2 2 3 2 22 20 20 2260 0 0 0 0 2 2 2 2 20 20 20 20 200 400 10 10 20-2 10

39. Step 3 0 0 1 2 2 2 3 2 30 20 20 30660 0 0 0 0 2 2 2 2 20 20 20 20 200 400 10 10 20-2 10

40. Step 4 0 0 0 0 2 2 3 2 30 20 20 3066-60 0 0 0 0 2 2 2 2 20 20 20 20 10-2 20-4 10 10 20-2 10 edge effect

41. Sobel Edge Detection: Gradient Approximation HorizontalVertical

42. Sobel vs. LoG Edge Detection: Matlab Automatic Thresholds SobelLoG

43. Canny Edge Detection Derivative of Gaussian Non-maximum suppression –Thin multi-pixel wide “ridges” down to single pixel Thresholding –Low, high edge-strength thresholds –Accept all edges over low threshold that are connected to edge over high threshold Matlab: edge(I, ‘canny’)

44. Canny Edge Detection: Example (Matlab automatically set thresholds)

45. Corner Detection Basic idea: Find points where two edges meet—i.e., high gradient in orthogonal directions Examine gradient over window (Shi & Tomasi, 1994) Edge strength encoded by eigenvalues ; corner is where over threshold Harris corners (Harris & Stephens, 1988), Susan corners (Smith & Brady, 1997)

46. Example: Corner Detection courtesy of S. Smith SUSAN corners

47. Edge-Based Image Comparison Chamfer, Hausdorff distance, etc. –Transform edge map based on distance to nearest edge before correlating as usual courtesy of D. Gavrila

48. Scale Space How thick an edge? How big a dot? Must consider what scale we are interested in when designing filters Efficiency a major consideration: Fine- grained template matching is expensive over a full image

49. Image Pyramids Idea: Represent image at different scales, allowing efficient coarse-to-fine search Downsampling: Simplest scale change: Decimation—just downsample from Forsyth & Ponce

50. Gaussian, Laplacian Pyramids Gaussian pyramid of image: and Laplacian pyramid –Difference of image and Gaussian at each level of courtesy of Wolfram Gaussian pyramidLaplacian pyramid

51. Color-based Image Comparison Color histograms (Swain & Ballard, 1991) –Steps Histogram RGB/HSV triplets over two images to be compared Normalize each histogram by respective total number of pixels to get frequencies Similarity is Euclidean distance between color frequency vectors –Sensitive to lighting changes –Works for different-sized images –Matlab: imhist, hist