1. Spatial Filtering Dan Witzner Hansen

2. Recap Exercises??? Feedback Last lectures?

3. Digital images In computer vision we operate on digital (discrete) images: – Sample the 2D space on a regular grid – Quantize each sample (round to nearest integer) Image thus represented as a matrix of integer values. Adapted from S. Seitz 2D 1D

4. What is point processing? Only one pixel in the input has an effect on the output For example: – Changing the brightness, thresholding, histogram stretching 02 12 121 253 13 22 01 120 214 101 InputOutput

5. Image neighborhoods Q: What happens if we reshuffle all pixels within the images? A: Its histogram won’t change. Point-wise processing unaffected. Need to measure properties relative to small neighborhoods of pixels

6. Why use proximity Noise reduction Smoothing Feature detection Correlation

7. What are they good for? Improve Search – Search over translations Classic coarse-to-fine strategy – Search over scale Template matching E.g. find a face at different scales Precomputation – Need to access image at different blur levels – Useful for texture mapping at different resolutions (called mip- mapping) Image Processing – Editing frequency bands separately – E.g. image blending…

8. Motivation: noise reduction

9. Common types of noise – Salt and pepper noise: random occurrences of black and white pixels – Impulse noise: random occurrences of white pixels – Gaussian (additive) noise: variations in intensity drawn from a Gaussian normal distribution Source: S. Seitz

10. Gaussian noise Fig: M. Hebert >> noise = randn(size(im)).*sigma; >> output = im + noise; Ideal Image Noise process

11. Changing Sigma and zero mean

12. First attempt at a solution Replace each pixel with an average of all the values in its neighborhood Assumptions: – Expect pixels to be like their neighbors – Expect noise processes to be independent from pixel to pixel

13. First attempt at a solution Let’s replace each pixel with an average of all the values in its neighborhood Moving average in 1D: Source: S. Marschner

14. Weighted Moving Average Can add weights to our moving average Weights [1, 1, 1, 1, 1] / 5 Source: S. Marschner

15. Weighted Moving Average Non-uniform weights [1, 4, 6, 4, 1] / 16 Source: S. Marschner

16. Neighborhood processing Several pixels in the input has an effect on the output 02 12 121 253 13 22 01 120 214 101 Input Output

17. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

18. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

19. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

20. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

21. Moving Average In 2D 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

22. Moving Average In 2D 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

23. Correlation filtering Say the averaging window size is 2k+1 x 2k+1: Loop over all pixels in neighborhood around image pixel F[i,j] Attribute uniform weight to each pixel Now generalize to allow different weights depending on neighboring pixel’s relative position: Non-uniform weights

24. Correlation filtering Filtering an image: replace each pixel with a linear combination of its neighbors. The filter “kernel” or “mask” H[u,v] is the prescription for the weights in the linear combination. Assume the kernel H has weights 0 ‘outside’ the finite region This is called cross-correlation, denoted

25. Averaging filter What values belong in the kernel H for the moving average example? 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 111 111 111 “box filter” ?

26. Smoothing by averaging depicts box filter: white = high value, black = low value original filtered

27. Gaussian filter What if we want nearest neighboring pixels to have the most influence on the output? 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 121 242 121 This kernel is an approximation of a Gaussian function: Source: S. Seitz

28. Smoothing with a Gaussian

29. Gaussian filters What parameters matter here? Size of kernel or mask – Note, Gaussian function has infinite support, but discrete filters use finite kernels σ = 5 with 10 x 10 kernel σ = 5 with 30 x 30 kernel

30. Gaussian filters Variance of Gaussian: determines extent of smoothing σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel

31. Matlab >> hsize = 10; >> sigma = 5; >> h = fspecial(‘gaussian’ hsize, sigma); >> mesh(h); >> imagesc(h); >> outim = imfilter(im, h); >> imshow(outim);

32. The effect of smoothing and noise More noise  Wider smoothing kernel 

33. Boundary issues What is the size of the output? MATLAB: filter2(g, f, shape) / conv2 – shape = ‘full’: output size is sum of sizes of f and g – shape = ‘same’: output size is same as f – shape = ‘valid’: output size is the difference of sizes of f and g f gg gg f gg g g f gg gg full samevalid Source: S. Lazebnik

34. Boundary issues What about near the edge? – the filter window falls off the boundaries of the image – methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner

35. Boundary issues in Matlab clip filter (black): imfilter(f, g, 0) wrap around:imfilter(f, g, ‘circular’) copy edge: imfilter(f, g, ‘replicate’) reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner

36. Filtering an impulse signal What is the result of filtering the impulse signal (image) F with the arbitrary kernel H? 0000000 0000000 0000000 0001000 0000000 0000000 0000000 abc def ghi ?

37. Convolution Convolution: – Flip the filter in both dimensions (bottom to top, right to left) – Then apply correlation Notation for convolution operator F H

38. Convolution vs. correlation Convolution Cross-correlation

39. Shift invariant linear system Shift invariant: Operator behaves the same everywhere, i.e. the value of the output depends on the pattern in the image neighborhood, not the position of the neighborhood. Linear: Superposition: h * (f1 + f2) = (h * f1) + (h * f2) Scaling: h * (k f) = k (h * f)

40. Properties of convolution Linear & shift invariant Commutative: f * g = g * f Associative (f * g) * h = f * (g * h) Identity: unit impulse e = […, 0, 0, 1, 0, 0, …]. f * e = f Differentiation:

41. How to do color blurring?

42. Predict the filtered outputs 000 010 000 * = ? 000 100 000 * 111 111 111 000 020 000 - *

43. Practice with linear filters 000 010 000 Original ? Source: D. Lowe

44. Practice with linear filters 000 010 000 Original Filtered (no change) Source: D. Lowe

45. Practice with linear filters 000 100 000 Original ? Source: D. Lowe

46. Practice with linear filters 000 100 000 Original Shifted left by 1 pixel with correlation Source: D. Lowe

47. Practice with linear filters Original ? 111 111 111 Source: D. Lowe

48. Practice with linear filters Original 111 111 111 Blur (with a box filter) Source: D. Lowe

49. Practice with linear filters Original 111 111 111 000 020 000 - ? Source: D. Lowe

50. Practice with linear filters Original 111 111 111 000 020 000 - Sharpening filter - Accentuates differences with local average Source: D. Lowe

51. Filtering examples: sharpening

52. Convolution with special filters for iris recognition

53. Template Matching The filter is called a template or a mask The brighter the value in the output, the better the match Implemented as – the correlation coefficient – Sums of squared differences (SSD) - sum(sum( (I-F).^2)) 53 Input image Template Output Output as 3D

54. Template Matching 54 Output

55. Template matching Scene Template What if the template is not identical to some subimage in the scene?

56. Template matching Detected template Template Match can be meaningful, if scale, orientation, and general appearance is right.

57. So, what scale to choose? It depends what we’re looking for. Too fine of a scale…can’t see the forest for the trees. Too coarse of a scale…can’t tell the maple grain from the cherry.

58. Smoothing with a Gaussian for sigma=1:3:10 h = fspecial('gaussian‘, fsize, sigma); out = imfilter(im, h); imshow(out); pause; end … Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.

59. Gaussian pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why?

60. Subsampling with Gaussian pre- filtering G 1/4G 1/8Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? How can we speed this up?

61. Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983]

62. A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose Figure from David Forsyth

63. So what about speed Convolution is What now if we have to use many scales? Can it be done faster? – Seperability – Integral images (somewhat opposite of scale- space / pyramid) – Fourier Transform (brief introduction)

64. Separability In some cases, filter is separable, and we can factor into two steps: – Convolve all rows – Convolve all columns

65. Haar-Like Features Haar-Wavelet

66. Image Features “Rectangle filters” Value = ∑ (pixels in white area) – ∑ (pixels in black area)

67. Fast computation with integral images The integral image computes a value at each pixel (x,y) that is the sum of the pixel values above and to the left of (x,y), inclusive This can quickly be computed in one pass through the image Calculated once per image (x,y)

68. Computing the integral image

69. Cumulative row sum: s(x, y) = s(x–1, y) + i(x, y) Integral image: ii(x, y) = ii(x, y−1) + s(x, y) ii(x, y-1) s(x-1, y) i(x, y) MATLAB: ii = cumsum(cumsum(double(i)), 2);

70. Computing sum within a rectangle Let A,B,C,D be the values of the integral image at the corners of a rectangle Then the sum of original image values within the rectangle can be computed as: – sum = A – B – C + D Only 3 additions are required for any size of rectangle! DB C A

71. Example +1 +2 -2 +1 Integral Image

72. Integral images Exceptionally fast when using many features at different scales Notice the difference to image pyramids Not all filters can be (efficiently) used with Integral images (e.g. Gaussian, but some approximation can)

73. Feature selection For a 24x24 detection region, the number of possible rectangle features is ~160,000! Can we make one approximation filter for the eye?

74. Boosting for face detection First two features selected by boosting: This feature combination can yield 100% detection rate and 50% false positive rate

75. Jean Baptiste Joseph Fourier (1768- 1830) Had crazy idea (1807): Any periodic function can be rewritten as a weighted sum of Sines and Cosines of different frequencies. Don’t believe it? – Neither did Lagrange, Laplace, Poisson and other big wigs – Not translated into English until 1878! But it’s true! – called Fourier Series – Possibly the greatest tool – used in Engineering

76. = 3 sin(x) A + 1 sin(3x) B A+B + 0.8 sin(5x) C A+B+C + 0.4 sin(7x)D A+B+C+D A sum of sines and cosines sin(x) A

77. Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)

78. Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t) = +

79. Higher frequencies due to sharp image variations (e.g., edges, noise, etc.)

80. Frequency Spectra example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t) = +

81. Fourier Transform and Convolution Spatial Domain (x)Frequency Domain (u) So, we can find g(x) by Fourier transform FT IFT

82. The Continuous Fourier Transform

83. Complex Numbers Real Imaginary Z=(a,b) a b |Z| 

84. x – The wavelength is 1/u. – The frequency is u. 1 The 1D Basis Functions 1/u

85. The Fourier Transform 1D Continuous Fourier Transform: The Inverse Fourier Transform The Continuous Fourier Transform 2D Continuous Fourier Transform: The Inverse Transform The Transform

86. The wavelength is. The direction is u/v. The 2D Basis Functions u=0, v=0 u=1, v=0u=2, v=0 u=-2, v=0u=-1, v=0 u=0, v=1u=1, v=1u=2, v=1 u=-2, v=1u=-1, v=1 u=0, v=2u=1, v=2u=2, v=2 u=-2, v=2u=-1, v=2 u=0, v=-1u=1, v=-1u=2, v=-1 u=-2, v=-1u=-1, v=-1 u=0, v=-2u=1, v=-2u=2, v=-2 u=-2, v=-2u=-1, v=-2 U V

87. (u = 0,..., N-1) (x = 0,..., N-1) 1D Discrete Fourier Transform: The Discrete Fourier Transform 2D Discrete Fourier Transform: (x = 0,..., N-1; y = 0,…,M-1) (u = 0,..., N-1; v = 0,…,M-1)

88. Fourier spectrum log(1 + |F(u,v)|) Image f The Fourier Image Fourier spectrum |F(u,v)|

89. Frequency Bands Percentage of image power enclosed in circles (small to large) : 90%, 95%, 98%, 99%, 99.5%, 99.9% ImageFourier Spectrum

90. Low pass Filtering 90% 95% 98% 99% 99.5% 99.9%

91. Noise Removal Noisy image Fourier Spectrum Noise-cleaned image

92. Noise Removal Noisy imageFourier SpectrumNoise-cleaned image

93. High Pass Filtering OriginalHigh Pass Filtered

94. High Frequency Emphasis + OriginalHigh Pass Filtered

95. High Frequency Emphasis OriginalHigh Frequency Emphasis Original High Frequency Emphasis

96. OriginalHigh pass Filter High Frequency Emphasis High Frequency Emphasis + Histogram Equalization High Frequency Emphasis