1. Lecture: Pixels and Filters
Juan Carlos Niebles and Ranjay Krishna
Stanford Vision Lab

2. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

3. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

4. Types of Images

5. Binary image representation
Slide credit: Ulas Bagci

6. Grayscale image representation
Slide credit: Ulas Bagci

7. Color Image - one channel

8. Color image representation
Slide credit: Ulas Bagci

9. Images are sampled
What happens when we zoom into the images we capture?

10. Errors due Sampling
Slide credit: Ulas Bagci

11. Resolution
is a sampling parameter, defined in dots per inch (DPI) or equivalent measures of spatial pixel density, and its standard value for recent screen technologies is 72 dpi
Slide credit: Ulas Bagci

12. Images are Sampled and Quantized
An image contains discrete number of pixels
A simple example
Pixel value:
“grayscale”
(or “intensity”): [0,255] 75
231
Last time: Images as Matrices
intensity
temperature
depth
148

13. Images are Sampled and Quantized
An image contains discrete number of pixels
A simple example
Pixel value:
“grayscale”
(or “intensity”): [0,255]
“color”
RGB: [R, G, B]
Lab: [L, a, b]
HSV: [H, S, V]
[90, 0, 53]
[249, 215, 203]
[213, 60, 67]

14. With this loss of information (from sampling and quantization), Can we still use images for useful tasks?

15. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

16. Histogram
Histogram of an image provides the frequency of the brightness (intensity) value in the image.
def histogram(im):
h = np.zeros(255)
for row in im.shape[0]:
for col in im.shape[1]:
val = im[row, col]
h[val] += 1

17. Histogram
Histogram captures the distribution of gray levels in the image.
How frequently each gray level occurs in the image Q

18. Histogram
Slide credit: Dr. Mubarak Shah

19. Histogram – use case
Slide credit: Dr. Mubarak Shah

20. Histogram – another use case
Slide credit: Dr. Mubarak Shah

21. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

22. Images as discrete functions
Images are usually digital (discrete):
Sample the 2D space on a regular grid
Represented as a matrix of integer values
pixel

23. Images as coordinates Cartesian coordinates Notation for discrete
functions

24. Images as coordinates Cartesian coordinates Notation for discrete
functions

25. Images as functions An Image as a function f from R2 to RM:
f( x, y ) gives the intensity at position ( x, y )
Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ]  [0,255]
Domain
support
range

26. Images as functions An Image as a function f from R2 to RM:
f( x, y ) gives the intensity at position ( x, y )
Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ]  [0,255]
Domain
support
range
Each r, g, b are functions that like the one above
A color image:

27. Histograms are a type of image function

28. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

29. Systems and Filters Filtering: Goals:
Forming a new image whose pixel values are transformed from original pixel values
Goals:
Goal is to extract useful information from images, or transform images into another domain where we can modify/enhance image properties
Features (edges, corners, blobs…)
super-resolution; in-painting; de-noising

30. Super-resolution
De-noising
In-painting
Bertamio et al

31. System and Filters
we define a system as a unit that converts an input function f[n,m] into an output (or response) function g[n,m], where (n,m) are the independent variables.
In the case for images, (n,m) represents the spatial position in the image.

32. Images as coordinates Cartesian coordinates Notation for discrete
functions

33. 2D discrete-space systems (filters)
S is the system operator, defined as a mapping or assignment of a member of the set of possible outputs g[n,m] to each member of the set of possible inputs f[n,m].

34. Filter example #1: Moving Average

35. Filter example #1: Moving Average
2D DS moving average over a 3 × 3 window of neighborhood h 1
Board:
Input image
Output image
each pixel results from a neighborhood operation

36. Filter example #1: Moving Average90 90
Courtesy of S. Seitz

37. Filter example #1: Moving Average90 90 10 37

38. Filter example #1: Moving Average90 90 10 20

39. Filter example #1: Moving Average90 90 10 20 30

40. Filter example #1: Moving Average90 10 20 30

41. Filter example #1: Moving Average90 10 20 30 40 60 90 50 80
What is the effect on the output?
What happens to the sharp edges?

42. Filter example #1: Moving Average
In summary:
This filter creates a new pixel with an average of its neighborhood.
Achieve smoothing effect (remove sharp features) 1

43. Filter example #1: Moving Average

44. Filter example #2: Image Segmentation
Image segmentation based on a simple threshold:
255,

45. Example Properties of systems
Amplitude properties:
Additivity
Homogeneity
Superposition
Stability
Invertibility

46. Example Properties of linear systems
Spatial properties
Causality
Shift invariance:

47. These properties have to be true for Linear Systems
Amplitude properties:
Additivity
Homogeneity
Superposition

48. These properties have to be true for Linear SHIFT INVARIANT Systems
Amplitude properties:
Additivity
Homogeneity
Superposition
Shift invariance:

49. What does shifting an image look like?
Cartesian coordinates

50. Is the moving average system is shift invariant?90 10 20 30 40 60 90 50 80

51. Is the moving average system is shift invariant?
𝑔 𝑛,𝑚 = 𝑘=𝑛−1 𝑛+1 𝑙=𝑚−1 𝑚+1 𝑓[𝑘,𝑙]
= 𝑘=−1 1 𝑙=−1 1 𝑓[𝑛−𝑘,𝑚−𝑙]
Let 𝑓[𝑛− 𝑛 0 ,𝑚− 𝑚 0 ] be a shifted input of 𝑓[𝑛,𝑚]
Now let’s pass 𝑓[𝑛− 𝑛 0 ,𝑚− 𝑚 0 ] through the system:
𝑓[𝑛− 𝑛 0 ,𝑚− 𝑚 0 ] 𝑆 𝑘=−1 1 𝑙=−1 1 𝑓[𝑛− 𝑛 0 −𝑘,𝑚− 𝑚 0 −𝑙]
Yes!
= g[𝑛− 𝑛 0 ,𝑚− 𝑚 0 ]

52. Is the moving average system is casual?
𝑔[𝑛,𝑚]= 𝑘=−1 1 𝑙=−1 1 𝑓[𝑛−𝑘,𝑚−𝑙]
f[𝑛,𝑚]
g[𝑛,𝑚] 90 10 20 30 40 60 90 50 80

53. Linear Systems (filters)
Linear filtering:
Form a new image whose pixels are a weighted sum of original pixel values
Use the same set of weights at each point
S is a linear system (function) iff it S satisfies
Moving average is an example of a linear filter.
Here a more general definition.
𝑆[ 𝛼𝑓 𝑖 𝑛,𝑚 + 𝛽 𝑓 𝑗 𝑘,𝑙 ] = 𝛼 𝑆[ 𝑓 𝑖 𝑛,𝑚 ] + 𝛽 𝑆[ 𝑓 𝑗 𝑘,𝑙 ]
superposition property

54. Linear Systems (filters)
Is the moving average a linear system?
Is thresholding a linear system?
f1[n,m] + f2[n,m] > T
f1[n,m] < T
f2[n,m]<T
Show on the board equations for moving average.
No!

55. How do we characterize a linear shift invariant (LSI) system?
Mathematically, how do you calculate g[n,m] from f[n,m]

56. 2D impulse function
1 at [0,0].
0 everywhere else

57. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response

58. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙] ?
h[0,0]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
g[0,0]

59. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9 ?
h[0,0]
h[0,1]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
g[0,0]

60. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9
1/9
h[0,0]
h[0,1]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response ?
g[0,0]
h[1,1]

61. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9
1/9
h[0,0]
h[0,1]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
1/9
g[0,0]
h[1,1]

62. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9
1/9 ?
h[0,0]
h[0,1]
h[0,2]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
1/9
g[0,0]
h[1,1]

63. Impulse response to the moving filter
𝛿 𝑆 ℎ 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9
1/9
h[0,0]
h[0,1]
h[0,2]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
1/9
g[0,0]
h[1,1]

64. Impulse response to the moving filter
1/9
1/9
1/9
h[-1,-1]
𝛿 𝑆 𝑔 𝑛,𝑚
= 𝑘=−1 1 𝑙=−1 1 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
1/9
1/9
1/9
h[0,0]
h[0,1]
h[0,2]
LSI systems can be characterized by the impulse response.
\delta_2 is the impulse (an image with 1 pixel =1, all the rest =0)
h is the response
1/9
1/9
1/9
g[0,0]
h[1,1]

65. Impulse response of the 3 by 3 moving average filter1

66. Any linear shift invariant system
By passing an impulse into a linear system, we get it’s impulse response.
So, if we don’t know what the linear system is doing, we can pass an impulse into it to get a filter ℎ 𝑛,𝑚 that tells us what the system is actually doing.
But how do we use ℎ 𝑛,𝑚 to calculate g 𝑛,𝑚 from f 𝑛,𝑚
𝛿 2 [𝑛,𝑚]
ℎ[𝑛,𝑚]

67. General linear shift invariant system
Let’s say our input f is a 3x3 image:
We can rewrite 𝑓 𝑛,𝑚 as a sum of delta functions:
𝑓 𝑛,𝑚 =𝑓 0,0 × 𝛿 2 [𝑛−0,𝑚−0]
+ 𝑓 0,1 × 𝛿 2 [𝑛−0,𝑚−1]
+ 𝑓 0,2 × 𝛿 2 [𝑛−0,𝑚−2]
+ …
f[2,2]
f[2,1]
f[2,0]
f[1,2]
f[1,1]
f[1,0]
f[0,1]
f[0,0]
𝑓 𝑛,𝑚 = 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
Or you can write it as:

68. 3 properties we need:
Now, we know what happens when we send a delta function through a LSI system:
𝛿 2 [𝑛,𝑚]
ℎ[𝑛,𝑚]
We also know that LSI systems shift the output if the input is shifted:
𝛿 2 [𝑛−𝑘,𝑚−𝑙]
ℎ[𝑛−𝑘,𝑚−𝑙]
Finally, the superposition principle:
𝑆[ 𝛼𝑓 𝑖 𝑛,𝑚 + 𝛽 𝑓 𝑗 𝑘,𝑙 ] = 𝛼 𝑆[ 𝑓 𝑖 𝑛,𝑚 ] + 𝛽 𝑆[ 𝑓 𝑗 𝑘,𝑙 ]

69. We can generalize this superposition principle…
𝑆[ 𝛼𝑓 𝑖 𝑛,𝑚 + 𝛽 𝑓 𝑗 𝑘,𝑙 ] = 𝛼 𝑆[ 𝑓 𝑖 𝑛,𝑚 ] + 𝛽 𝑆[ 𝑓 𝑗 𝑘,𝑙 ]
… with our summation of deltas…
𝑓 𝑛,𝑚 = 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
… as follows:
𝑆 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
= 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝑆 𝛿 2 [𝑛−𝑘,𝑚−𝑙]

70. Linear shift invariant system
𝑆 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
= 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 × 𝑆 𝛿 2 [𝑛−𝑘,𝑚−𝑙]
= 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑘,𝑙 ×ℎ[𝑛−𝑘,𝑚−𝑙]

71. Linear Shift Invariant systems
An LSI system is completely specified by its impulse response.
We can write any function as a sum of impulses…
Let’s pass this function over our system..
Using superposition, we get a sum over the impulse responses…
Final equation: we call this a convolution. Applying the impulse response h to f at all locations.
Discrete convolution
𝑓 𝑛,𝑚 ∗ℎ[𝑛,𝑚]

72. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

73. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

74. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

75. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

76. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

77. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

78. 2D convolution 2D convolution is very similar to 1D.
The main difference is that we now have to iterate over 2 axis instead of 1.
Assume we have a filter(h[,]) that is 3x3. and an image (f[,]) that is 7x7.

79. Slide credit: Ulas Bagci

80. 2D convolution example
Slide credit: Song Ho Ahn

81. 2D convolution example
Slide credit: Song Ho Ahn

82. 2D convolution example
Slide credit: Song Ho Ahn

83. 2D convolution example
Slide credit: Song Ho Ahn

84. 2D convolution example
Slide credit: Song Ho Ahn

85. 2D convolution example
Slide credit: Song Ho Ahn

86. 2D convolution example
Slide credit: Song Ho Ahn

87. Convolution in 2D - examples1 ? = *
Original

88. Convolution in 2D - examples1 = *
Original
Filtered
(no change)

89. Convolution in 2D - examples1 ? = *
Original

90. Convolution in 2D - examples1 = *
Original
Shifted right
By 1 pixel

91. Convolution in 2D - examples? 1 = *
Original

92. Convolution in 2D - examples1 = *
Original
Blur (with a
box filter)

93. Convolution in 2D - examples2 1
= ?
(Note that filter sums to 1)
Original
“details of the image”
Original image – blur = details! - 1 1 1 +

94. What does blurring take away?– =
original
smoothed (5x5)
Detailed
Let’s add it back: = +
original
Detailed
Sharpened

95. Convolution in 2D – Sharpening filter- 2 1 =
Original
Sharpening filter: Accentuates differences with local average

96. Image support and edge effect
A computer will only convolve finite support signals.
That is: images that are zero for n,m outside some rectangular region
numpy’s convolution performs 2D DS convolution of finite-support signals. = *
(N1 + N2 − 1) × (M1 +M2 − 1)
N2 ×M2
N1 ×M1

97. Image support and edge effect
A computer will only convolve finite support signals.
What happens at the edge?
• zero “padding”
• edge value replication
• mirror extension
• more (beyond the scope of this class) f h

98. Slide credit: Wolfram Alpha

99. What we will learn today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation
Some background reading:
Forsyth and Ponce, Computer Vision, Chapter 7

100. (Cross) correlation (symbol: )
Cross correlation of two 2D signals f[n,m] and h[n,m]:
𝑓 𝑛,𝑚 ∗∗ℎ[𝑛,𝑚]= 𝑘=−∞ ∞ 𝑙=−∞ ∞ 𝑓 𝑛,𝑚 × ℎ[𝑛+𝑘,𝑚+𝑙]
Equivalent to a convolution without the flip
Equation is about the same.
Difference is with the complex conjugate)

101. (Cross) correlation – example
Use: compare similarity between “kernel” and image.
Courtesy of J. Fessler

102. (Cross) correlation – example
Use: compare similarity between “kernel” and image.
Courtesy of J. Fessler

103. (Cross) correlation – example
Use: compare similarity between “kernel” and image.
Courtesy of J. Fessler
numpy’s correlate
Courtesy of J. Fessler

104. (Cross) correlation – example
Left
Right
scanline
Norm. cross corr. score

105. f*h
f**h

106. Cross Correlation Application: Vision system for TV remote control
- uses template matching
This is a figure from the book, figure Read the caption for the story!
Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE

107. Properties • Commutative property: • Associative property:
• Distributive property:
The order doesn’t matter!

108. Properties
• Shift property:
• Shift-invariance:

109. Convolution vs. (Cross) Correlation
A convolution is an integral that expresses the amount of overlap of one function as it is shifted over another function.
convolution is a filtering operation
Correlation compares the similarity of two sets of data. Correlation computes a measure of similarity of two input signals as they are shifted by one another. The correlation result reaches a maximum at the time when the two signals match best .
correlation is a measure of relatedness of two signals

110. What we have learned today?
Image sampling and quantization
Image histograms
Images as functions
Linear systems (filters)
Convolution and correlation