1. Slide credit Fei Fei Li

2. Slide credit Fei Fei Li

3. Slide credit Fei Fei Li

4. Image filtering
Image filtering: compute function of local neighborhood at each position
Really important!
Enhance images
Denoise, resize, increase contrast, etc.
Extract information from images
Texture, edges, distinctive points, etc.
Detect patterns
Template matching

5. Example: box filter 1
Slide credit: David Lowe (UBC)

6. Image filtering 1 90 90
ones, divide by 9
Credit: S. Seitz

7. Image filtering 1 90 90 10
ones, divide by 9
Credit: S. Seitz

8. Image filtering 1 90 90 10 20
ones, divide by 9
Credit: S. Seitz

9. Image filtering 1 90 90 10 20 30
ones, divide by 9
Credit: S. Seitz

10. Image filtering 1 90 10 20 30
ones, divide by 9
Credit: S. Seitz

11. Image filtering 1 90 10 20 30 ?
ones, divide by 9
Credit: S. Seitz

12. Image filtering 1 90 10 20 30 50 ?
ones, divide by 9
Credit: S. Seitz

13. Image filtering 1 90 10 20 30 40 60 90 50 80
Credit: S. Seitz

14. Box Filter What does it do?
Replaces each pixel with an average of its neighborhood
Achieve smoothing effect (remove sharp features) 1
Slide credit: David Lowe (UBC)

15. Smoothing with box filter

16. Practice with linear filters1 ?
Original
Source: D. Lowe

17. Practice with linear filters1
Original
Filtered
(no change)
Source: D. Lowe

18. Practice with linear filters1 ?
Original
Source: D. Lowe

19. Practice with linear filters1
Original
Shifted left
By 1 pixel
Source: D. Lowe

20. Practice with linear filters- 2 1 ?
(Note that filter sums to 1)
Original
Source: D. Lowe

21. Practice with linear filters- 2 1
Original
Sharpening filter
Accentuates differences with local average
Source: D. Lowe

22. Sharpening
Source: D. Lowe

23. Other filters -1 1 -2 2
Sobel
Vertical Edge
(absolute value)

24. Other filters -1 -2 1 2 Horizontal Edge (absolute value) Sobel1 2
Questions at this point?
Sobel
Horizontal Edge
(absolute value)

25. Filtering vs. Convolution
g=filter
f=image
2d filtering
2d convolution

26. Key properties of linear filters
Linearity:
filter(f1 + f2) = filter(f1) + filter(f2)
Shift invariance: same behavior regardless of pixel location
filter(shift(f)) = shift(filter(f))
Any linear, shift-invariant operator can be represented as a convolution
f is image
Source: S. Lazebnik

27. More properties Commutative: a * b = b * a
Conceptually no difference between filter and signal
Associative: a * (b * c) = (a * b) * c
Often apply several filters one after another: (((a * b1) * b2) * b3)
This is equivalent to applying one filter: a * (b1 * b2 * b3)
Distributes over addition: a * (b + c) = (a * b) + (a * c)
Scalars factor out: ka * b = a * kb = k (a * b)
Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a
Source: S. Lazebnik

28. Important filter: Gaussian
Weight contributions of neighboring pixels by nearness
5 x 5,  = 1
e^0 = 1, e^-1 = .37, e^-2 = .14, etc.
Slide credit: Christopher Rasmussen

29. Smoothing with Gaussian filter

30. Smoothing with box filter

31. Gaussian filters
Remove “high-frequency” components from the image (low-pass filter)
Images become smoother
Convolution with self is another Gaussian
So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have
Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2
Separable kernel
Factors into product of two 1D Gaussians
Linear vs. quadratic in mask size
Source: K. Grauman

32. Separability of the Gaussian filter
Source: D. Lowe

33. Separability example * 2D convolution (center location only)
The filter factors into a product of 1D filters: * =
Perform convolution along rows:
Followed by convolution along the remaining column:
Source: K. Grauman

34. Separability Why is separability useful in practice?
Because you can do it faster. Filtering an M-by-N image with a P-by-Q filter kernel requires roughly MNPQ multiplies and adds.
With separable kernels, you can filter in two steps.
The first step requires about MNP multiplies and adds. The second requires about MNQ multiplies and adds, for a total of MN(P + Q). Performance Advantage  PQ/(P+Q)
The computational advantage of separable convolution versus nonseparable convolution is therefore: For a 9-by-9 filter kernel, that's a theoretical speed-up of 4.5.

35. Practical matters What about near the edge?
the filter window falls off the edge of the image
need to extrapolate
methods:
clip filter (black)
wrap around
copy edge
reflect across edge
Source: S. Marschner

36. Practical matters What is the size of the output?
MATLAB: filter2(g, f, shape)
shape = ‘full’: output size is sum of sizes of f and g
shape = ‘same’: output size is same as f
shape = ‘valid’: output size is difference of sizes of f and g
full
same
valid g g g g f f g f g g g g g g g
Source: S. Lazebnik

37. Median filters
A Median Filter operates over a window by selecting the median intensity in the window.
What advantage does a median filter have over a mean filter?
Is a median filter a kind of convolution?
Better at salt’n’pepper noise
Not convolution: try a region with 1’s and a 2, and then 1’s and a 3
© 2006 Steve Marschner • 37
Slide by Steve Seitz

38. Comparison: salt and pepper noise
© 2006 Steve Marschner • 38
Slide by Steve Seitz

39. Why does the Gaussian give a nice smooth image, but the square filter give edgy artifacts?
Box filter
Things we can’t understand without thinking in frequency

40. A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006

41. Why do we get different, distance-dependent interpretations of hybrid images?
Slide: Hoiem

42. Why does a lower resolution image still make sense to us
Why does a lower resolution image still make sense to us? What do we lose?
Image:
Slide: Hoiem

43. Thinking in terms of frequency

44. Jean Baptiste Joseph Fourier (1768-1830)
...the manner in which the author arrives at these equations is not exempt of difficulties and...his analysis to integrate them still leaves something to be desired on the score of generality and even rigour.
had crazy idea (1807):
Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies.
Don’t believe it?
Neither did Lagrange, Laplace, Poisson and other big wigs
Not translated into English until 1878!
But it’s (mostly) true!
called Fourier Series
there are some subtle restrictions
Laplace
Legendre
Lagrange

45. A sum of sines Our building block:
Add enough of them to get any signal g(x) you want!

46. Frequency Spectra example : g(t) = sin(2πf t) + (1/3)sin(2π(3f) t) = +
Slides: Efros

47. Frequency Spectra

48. Frequency Spectra = + =

49. Frequency Spectra = + =

50. Frequency Spectra = + =

51. Frequency Spectra = + =

52. Frequency Spectra = + =

53. Frequency Spectra =

54. Example: Music
We think of music in terms of frequencies at different magnitudes
Slide: Hoiem

55. Other signals
We can also think of all kinds of other signals the same way
xkcd.com

56. Fourier analysis in images
Intensity Image
Fourier Image

57. Signals can be composed+ =
More:

58. Fourier Transform
Fourier transform stores the magnitude and phase at each frequency
Magnitude encodes how much signal there is at a particular frequency
Phase encodes spatial information (indirectly)
For mathematical convenience, this is often notated in terms of real and complex numbers
Amplitude:
Phase:

59. Computing the Fourier Transform
Continuous
Discrete
k = -N/2..N/2
Fast Fourier Transform (FFT): NlogN

60. The Convolution Theorem
The Fourier transform of the convolution of two functions is the product of their Fourier transforms
Convolution in spatial domain is equivalent to multiplication in frequency domain!

61. Properties of Fourier Transforms
Linearity
Fourier transform of a real signal is symmetric about the origin
The energy of the signal is the same as the energy of its Fourier transform
See Szeliski Book (3.4)

62. Filtering in spatial domain-1 1 -2 2 * =

63. Filtering in frequency domain
FFT
FFT =
Inverse FFT
Slide: Hoiem

64. Filtering
Why does the Gaussian give a nice smooth image, but the square filter give edgy artifacts?
Gaussian
Box filter
Things we can’t understand without thinking in frequency

65. Gaussian

66. Box Filter

67. Sampling
Why does a lower resolution image still make sense to us? What do we lose?
Image:

68. Subsampling by a factor of 2
Throw away every other row and column to create a 1/2 size image

69. Aliasing problem
1D example (sinewave):
Source: S. Marschner

70. Aliasing problem
1D example (sinewave):
Source: S. Marschner

71. Aliasing problem Sub-sampling may be dangerous….
Characteristic errors may appear:
“Wagon wheels rolling the wrong way in movies”
“Checkerboards disintegrate in ray tracing”
“Striped shirts look funny on color television”
Source: D. Forsyth

72. Aliasing in video
Slide by Steve Seitz

73. Aliasing in graphics
Source: A. Efros

74. Sampling and aliasing

75. Nyquist-Shannon Sampling Theorem
When sampling a signal at discrete intervals, the sampling frequency must be  2  fmax
fmax = max frequency of the input signal
This will allows to reconstruct the original perfectly from the sampled version v v v
good
bad

76. Anti-aliasing Solutions: Sample more often
Get rid of all frequencies that are greater than half the new sampling frequency
Will lose information
But it’s better than aliasing
Apply a smoothing filter

77. Algorithm for downsampling by factor of 2
Start with image(h, w)
Apply low-pass filter
im_blur = imfilter(image, fspecial(‘gaussian’, 7, 1))
Sample every other pixel
im_small = im_blur(1:2:end, 1:2:end);

78. Anti-aliasing
Forsyth and Ponce 2002

79. Subsampling without pre-filtering
1/2
1/4 (2x zoom)
1/8 (4x zoom)
Slide by Steve Seitz

80. Subsampling with Gaussian pre-filtering
Slide by Steve Seitz

81. Why does a lower resolution image still make sense to us
Why does a lower resolution image still make sense to us? What do we lose?
Image:

82. Why do we get different, distance-dependent interpretations of hybrid images?

83. Salvador Dali invented Hybrid Images?
“Gala Contemplating the Mediterranean Sea,
which at 30 meters becomes the portrait
of Abraham Lincoln”, 1976

85. Application: Hybrid Images
A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006

86. Clues from Human Perception
Early processing in humans filters for various orientations and scales of frequency
Perceptual cues in the mid-high frequencies dominate perception
When we see an image from far away, we are effectively subsampling it
Early Visual Processing: Multi-scale edge and blob filters

87. Hybrid Image in FFT
Hybrid Image
Low-passed Image
High-passed Image

88. Perception
Why do we get different, distance-dependent interpretations of hybrid images? ?

89. Things to Remember
Sometimes it makes sense to think of images and filtering in the frequency domain
Fourier analysis
Can be faster to filter using FFT for large images (N logN vs. N2 for auto-correlation)
Images are mostly smooth
Basis for compression
Remember to low-pass before sampling

90. Sharpening revisited What does blurring take away? Let’s add it back:
original
smoothed (5x5)
detail = –
Let’s add it back:
original
detail
+ α
sharpened =