1. Lecture 2: Image filtering
CS6670: Computer Vision
Noah Snavely
Lecture 2: Image filtering
Hybrid Images, Oliva et al.,

2. Lecture 2: Image filtering
CS6670: Computer Vision
Noah Snavely
Lecture 2: Image filtering
Hybrid Images, Oliva et al.,

3. Lecture 2: Image filtering
CS6670: Computer Vision
Noah Snavely
Lecture 2: Image filtering
Hybrid Images, Oliva et al.,

4. Lecture 2: Image filtering
CS6670: Computer Vision
Noah Snavely
Lecture 2: Image filtering
Hybrid Images, Oliva et al.,

5. Reading
Szeliski, Chapter

6. What is an image?

7. What is an image? We’ll focus on these in this class
Digital Camera
We’ll focus on these in this class
(More on this process later)
The Eye
Source: A. Efros

8. = What is an image? A grid (matrix) of intensity values
(common to use one byte per value: 0 = black, 255 = white)
255 20 75 95 96
127
145
175
200 47 74 =

9. What is an image?
We can think of a (grayscale) image as a function, f, from R2 to R (or a 2D signal):
f (x,y) gives the intensity at position (x,y)
A digital image is a discrete (sampled, quantized) version of this function x y
f (x, y)
snoop
3D view

10. Image transformations
As with any function, we can apply operators to an image
We’ll talk about a special kind of operator, convolution (linear filtering)
g (x,y) = f (x,y) + 20
g (x,y) = f (-x,y)

11. Question: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Answer: take lots of images, average them
Take lots of images and average them!
What’s the next best thing?
Source: S. Seitz

12. Image filtering
Modify the pixels in an image based on some function of a local neighborhood of each pixel 5 1 4 7 3 10
Some function 7
Local image data
Modified image data
Source: L. Zhang

13. Linear filtering
One simple version: linear filtering (cross-correlation, convolution)
Replace each pixel by a linear combination of its neighbors
The prescription for the linear combination is called the “kernel” (or “mask”, “filter”) 6 1 4 8 5 3 10
0.5 1 8
Local image data
kernel
Modified image data
Source: L. Zhang

14. Cross-correlation
Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image
This is called a cross-correlation operation:

15. Convolution
Same as cross-correlation, except that the kernel is “flipped” (horizontally and vertically)
Convolution is commutative and associative
This is called a convolution operation:

16. Convolution
Adapted from F. Durand

17. Mean filtering 90 10 20 30 40 60 90 50 80 * = 1

18. Linear filters: examples* 1 =
Original
Identical image
Source: D. Lowe

19. Linear filters: examples* 1 =
Original
Shifted left
By 1 pixel
Source: D. Lowe

20. Linear filters: examples1 * =
Original
Blur (with a mean filter)
Source: D. Lowe

21. Linear filters: examples
Sharpening filter (accentuates edges) 1 2 - * =
Original
Source: D. Lowe

22. Sharpening
Source: D. Lowe

23. Smoothing with box filter revisited
I always walk through the argument on the left rather carefully; it gives some insight into the
significance of impulse responses or point spread functions.
Source: D. Forsyth

24. Gaussian Kernel
Source: C. Rasmussen

25. Gaussian filters
= 1 pixel
= 5 pixels
= 10 pixels
= 30 pixels

26. Gaussian filter
Removes “high-frequency” components from the image (low-pass filter)
Convolution with self is another Gaussian
Convolving two times with Gaussian kernel of width = convolving once with kernel of width * =
Linear vs. quadratic in mask size
Source: K. Grauman

27. Sharpening
Source: D. Lowe

28. Sharpening revisited = = What does blurring take away? – + α
original
smoothed (5x5)
detail = –
Let’s add it back:
original
detail
+ α
sharpened =
Source: S. Lazebnik

29. unit impulse (identity)
Sharpen filter
blurred image
image
unit impulse (identity)
Gaussian
scaled impulse
Laplacian of Gaussian
f + a(f - f * g) = (1+a)f-af*g = f*((1+a)e-g)

30. Sharpen filter
unfiltered
filtered

31. Convolution in the real world
Camera shake = *
Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006
Bokeh: Blur in out-of-focus regions of an image.
Source:

32. Questions?

33. Edge detection Convert a 2D image into a set of curves
Extracts salient features of the scene
More compact than pixels
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAA

34. Origin of Edges Edges are caused by a variety of factors
surface normal discontinuity
depth discontinuity
surface color discontinuity
illumination discontinuity
Edges are caused by a variety of factors

35. Characterizing edges
An edge is a place of rapid change in the image intensity function
intensity function (along horizontal scanline)
image
first derivative
edges correspond to extrema of derivative
Source: L. Lazebnik

36. Image derivatives How can we differentiate a digital image F[x,y]?
Option 1: reconstruct a continuous image, f, then compute the derivative
Option 2: take discrete derivative (finite difference)
How would you implement this as a linear filter? 1 -1 -1 1 : :
Source: S. Seitz

37. Image gradient The gradient of an image:
The gradient points in the direction of most rapid increase in intensity
The edge strength is given by the gradient magnitude:
The gradient direction is given by:
how does this relate to the direction of the edge?
give definition of partial derivative: lim h->0 [f(x+h,y) – f(x,y)]/h
Source: Steve Seitz

38. Image gradient
Source: L. Lazebnik

39. Image gradient

40. Effects of noise Where is the edge? Noisy input image Source: S. Seitz
How to fix?
Where is the edge?
Source: S. Seitz

41. Solution: smooth first
f * h
To find edges, look for peaks in
Source: S. Seitz

42. Associative property of convolution
Differentiation is convolution, and convolution is associative:
This saves us one operation: f
Source: S. Seitz

43. 2D edge detection filters
derivative of Gaussian (x)
Gaussian
How many 2nd derivative filters are there? There are four 2nd partial derivative filters.
In practice, it’s handy to define a single 2nd derivative filter—the Laplacian

44. Derivative of Gaussian filter
x-direction
y-direction

45. Side note: How would you compute a directional derivative?
(From vector calculus)
Directional deriv. is a linear combination of partial derivatives
= ? + =

46. Derivative of Gaussian filter
x-direction
y-direction + =

47. The Sobel operator Common approximation of derivative of Gaussian-1 1 -2 2 1 2 -1 -2
Q: Why might these work better?
A: more stable when there is noise
The standard defn. of the Sobel operator omits the 1/8 term
doesn’t make a difference for edge detection
the 1/8 term is needed to get the right gradient value 47

48. Sobel operator: example
Source: Wikipedia

49. Example
original image (Lena)

50. Finding edges
gradient magnitude

51. Finding edges
where is the edge?
thresholding

52. Non-maximum supression
Check if pixel is local maximum along gradient direction
requires interpolating pixels p and r

53. Finding edges
thresholding

54. (non-maximum suppression)
Finding edges
thinning
(non-maximum suppression)

55. Source: D. Lowe, L. Fei-Fei
Canny edge detector
MATLAB: edge(image,‘canny’)
Filter image with derivative of Gaussian
Find magnitude and orientation of gradient
Non-maximum suppression
Linking and thresholding (hysteresis):
Define two thresholds: low and high
Use the high threshold to start edge curves and the low threshold to continue them
Source: D. Lowe, L. Fei-Fei

56. Canny edge detector
Still one of the most widely used edge detectors in computer vision
Depends on several parameters:
J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8: , 1986.
: width of the Gaussian blur
high threshold
low threshold

57. Canny edge detector original Canny with Canny with
The choice of depends on desired behavior
large detects “large-scale” edges
small detects fine edges
Source: S. Seitz

58. Scale space (Witkin 83) larger
first derivative peaks
larger
Gaussian filtered signal
Properties of scale space (w/ Gaussian smoothing)
edge position may shift with increasing scale ()
two edges may merge with increasing scale
an edge may not split into two with increasing scale

59. Questions?
3-minute break

60. Images as vectors Very important idea!1
2D image
Scanline (1D signal)
(A 2D, n x m image can be represented by a vector of length nm formed by concatenating the rows)
Vector

61. Multiplying row and column vectors
= ?

62. Filtering as matrix multiplication
What kind of filter is this?

63. Filtering as matrix multiplication=

64. Another matrix transformation
2D DCT basis
1D Discrete cosine transform (DCT) basis

65. Another matrix transformation
2D DCT basis
1D Discrete cosine transform (DCT) basis