1. - photometric aspects of image formation gray level images
linear/nonlinear filtering
edge detection
corner detection
CS223, Jana Kosecka

2. Image
Brightness values
I(x,y)
CS223, Jana Kosecka

3. Analog intensity function Temporal/spatial sampled function
Image model
Mathematical tools
Analog intensity function
Temporal/spatial sampled function
Quantization of the gray levels
Point sets
Random fields
List of image features, regions
Analysis
Linear algebra
Numerical methods
Set theory, morphology
Stochastic methods
Geometry, AI, logic
CS223, Jana Kosecka

4. Basic Photometry Radiometric model of image formation
CS223, Jana Kosecka

5. Basic ingredients
Radiance – amount of energy emitted along certain direction
Iradiance – amount of energy received along certain direction
BRDF – bidirectional reflectance distribution –
portion of the energy coming from direction
reflected to direction
Lambertian surfaces – the appearance depends only on radiance, not
on the viewing direction
Image intensity for a Lambertian surface
CS223, Jana Kosecka

6. Images Images contain noise – sources sensor quality, light
fluctuations, quantization effects
CS223, Jana Kosecka

7. Image Noise Models Additive noise: most commonly used
Multiplicative noise:
Impulsive noise (salt and pepper):
Noise models: gaussian, uniform
Noise Amount: SNR = s/ n
CS223, Jana Kosecka

8. Image filtering How can we reduce the noise in the image
Image acquisition noise due to light fluctuations and sensor noise can be reduced by acquiring a sequence of images and averaging them
Computation of simple features
First stage of visual processing
CS223, Jana Kosecka

9. Image Processing 1D signal and its sampled version
f = { f(1), f(2), f(3), …, f(n)}
f = {0, 1, 2, 3, 4, 5, 5, 6, 10 }
CS223, Jana Kosecka

10. Discrete time system Maps 1 discrete time signal to another
f[x]
g[x] h
Special class of systems – linear , time-invariant systems
Superposition principle
Shift (time) invariant – shift in input causes shift in output
CS223, Jana Kosecka

11. unit impluse – if x = 0 it’s 1 and zero everywhere else
Convolution sum:
unit impluse – if x = 0 it’s 1 and zero everywhere else
Every discrete time signal can be written as a sum of scaled
and shifted impulses
The output the linear system is related to the input and the transfer
function via convolution h f g
filter f g
Convolution sum:
Notation:
CS223, Jana Kosecka

12. Averaging filter
Original image
Smoothed image
CS223, Jana Kosecka

13. Averaging filter center pixel weighted more
and 0 everywhere else
Box filter
Ex. cont.
Averaging filter center pixel weighted more
CS223, Jana Kosecka

14. Convolution in 2D X 10 10 11 9 1 2 99 f g h 1 1 1 1 1 1
1/9 1 1 1
1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) =
1/9.( 90) = 10
CS223, Jana Kosecka

15. Example: X X X X X X 10 11 10 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 91 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 10 2 1 X X 11 10 9 9 11 10 X X 9 10 11 9 99 11 F X X X X X X 10 9 9 11 10 10 1 1 1 1 1 1
1/9 1 1 1
1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) =
1/9.( 34) =
CS223, Jana Kosecka

16. Example: X X X X X X 10 11 10 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 91 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 10 2 1 X X 11 10 9 9 11 10 1 F X 20 X 9 10 11 9 99 11 X X X X X X 10 9 9 11 10 10
1/9
1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) =
1/9.( 180) = 20
CS223, Jana Kosecka

17. How big should the mask be?
The bigger the mask,
more neighbors contribute
bigger noise spread.
more blurring.
more expensive to compute
Limitations of averaging
Signal frequencies shared with noise are lost
Impulsive noise is diffused but not removed
CS223, Jana Kosecka

18. Frequency Domain Interpretation:
Sinc(u) = sin u F i
Low-pass filter
The secondary lobes of the sinc let noise into the filtered image.
CS223, Jana Kosecka

19. Gaussian Filter Better option for blurring
The coefficients are samples of a 1D Gaussian.
Gives more weight at the central pixel and less weights to the neighbors.
The further away the neighbors, the smaller the weight.
Gaussian filter
Samples from the continuous Gaussian
Gaussian filter is the only one that has the same shape
in the space and frequency domains.
There are no secondary lobes – i.e. a truly low-pass filter
CS223, Jana Kosecka

20. How big should the mask be?
The std. dev of the Gaussian  determines the amount of smoothing.
The samples should adequately represent a Gaussian
For a 98.76% of the area, we need
m = 5
5.(1/)  2    0.796, m 5
5-tap filter
g[x] = [0.136, , 1.00, 0.606, 0.136]
CS223, Jana Kosecka

21. Image Smoothing Convolution with a 2D Gaussian filter
Gaussian filter is separable, convolution can be accomplished as two 1-D convolutions
CS223, Jana Kosecka

22. Non-linear Filtering
Replace each pixel with the MEDIAN value of all the pixels in the neighborhood
Non-linear
Does not spread the noise
Can remove spike noise
Expensive to run
CS223, Jana Kosecka

23. Example: 10 11 9 1 2 99 I X 10 O median sort 10,11,10,9,10,11,10,9,1099 I X 10 O
median
sort
10,11,10,9,10,11,10,9,10
9,9,10,10,10,10,10,11,11
CS223, Jana Kosecka

24. Example: 10 11 10 1 X X X X X X 9 10 11 1 1 10 10 1 1 X X O 10 9 10 I 2 1 X X 11 10 9 9 11 X X 10 9 10 11 9 99 11 X X 10 9 9 11 10 10 X X X X X X
median
sort
11,10,0,10,11,1,9,10,0
0,0,1,9,10,10,10,11,11
CS223, Jana Kosecka

25. Example: 10 11 10 1 X X X X X X 9 10 11 1 1 10 X X O 10 9 10 2 1 X X I1 X X X X X X 9 10 11 1 1 10 X X O 10 9 10 2 1 X X I 11 10 9 9 11 X 9 X 10 9 10 11 9 99 11 X 10 X 10 9 9 11 10 10 X X X X X X
median
sort
10,9,11,9,99,11,11,10,10
9,9,10,10,10,11,11,11,99
CS223, Jana Kosecka

26. Image Features Local, meaningful, detectable parts of the image.
We will look at edges and corners
CS223, Jana Kosecka

27. Image Features – Edges, Corners
Look for detectable, meaningful parts of the image
Edges are detected at places where the image values exhibit sharp variation
Gray value
column
column
Edge
Edge
CS223, Jana Kosecka

28. Edge detection (1D) F(x) Edge= sharp variation x F ’(x)
Large first derivative x
CS223, Jana Kosecka

29. Digital Approximation of 1st derivatives
f(x)
X-1
X+1 -1 1
Convolve with:
CS223, Jana Kosecka

30. Edge Detection (2D) Vertical Edges: -1 1 Convolve with: Convolve with:1
Convolve with:
Convolve with: -1 1
Horizontal Edges:
CS223, Jana Kosecka

31. Noise cleaning and Edge Detection
I(x,y)
E(x,y)
Noise
Filter
Edge
Detection
We need to also deal with noise
-> Combine Linear Filters
CS223, Jana Kosecka

32. Noise Smoothing & Edge Detection-1 1
Convolve with:
Noise Smoothing
Vertical Edge Detection
This mask is called the (vertical) Prewitt Edge Detector
Outer product of box filter [1 1 1]T and [-1 0 1]
CS223, Jana Kosecka

33. Noise Smoothing & Edge Detection-1 1
Convolve with:
Horizontal Edge Detection
Noise Smoothing
This mask is called the (horizontal) Prewitt Edge Detector
CS223, Jana Kosecka

34. Sobel Edge Detector -1 1 -2 2 Convolve with: Gives more weight1 -2 2
Convolve with:
Gives more weight
to the center pixels
and -1 -2 1 2
CS223, Jana Kosecka

35. Example 50 50
100
150 50
100
150 -1 1 -1 1
CS223, Jana Kosecka

36. Image Derivatives We know better alternative to smoothing
Smooth using Gaussian filter
g(x) is a 1-D Gaussian filter, g(x,y) – 2-D Gaussian filter
Taking a derivative – linear operation (take the
derivative of the filter)
CS223, Jana Kosecka

37. Gaussian and its derivative
CS223, Jana Kosecka

38. Vertical edges
First derivative
CS223, Jana Kosecka

39. Horizontal edges
Gradient Magnitude
Image Gradient
CS223, Jana Kosecka

40. Gradient Orientation
CS223, Jana Kosecka

41. Orientation histogram
CS223, Jana Kosecka

42. Corner detection A point on a line is hard to match. Intuition:
Right at corner, gradient is ill defined.
Near corner, gradient has two different values.
CS223, Jana Kosecka

43. Formula for Finding Corners
We look at matrix:
Gradient with respect to x, times gradient with respect to y
Sum over a small region, the hypothetical corner
Matrix is symmetric
CS223, Jana Kosecka

44. First, consider case where:
This means all gradients in neighborhood are:
(k,0) or (0, c) or (0, 0) (or off-diagonals cancel).
What is region like if: ?
and ?
CS223, Jana Kosecka

45. General Case:
In general case from linear algebra, it follows that because C is symmetric:
With R being a rotation matrix. So every case can be intrepreted like one on last slide.
CS223, Jana Kosecka

46. Corner Detection Filter image.
Compute magnitude of the gradient everywhere.
We construct C in a window.
Compute eigenvalues l1 and l2.
If they are both big, we have a corner
Or if smalest eigenvalue  of C is bigger than  - mark pixel as candidate feature point
CS223, Jana Kosecka

47. Point Feature Extraction Harris Corner Detector
Alternatively feature quality function (Harris Corner Detector)
CS223, Jana Kosecka

48. Harris Corner Detector - Example
CS223, Jana Kosecka

49. Edge Detection Compute image derivatives
original image
gradient magnitude
Compute image derivatives
if gradient magnitude >  and the value is a local max. along gradient
direction – pixel is an edge candidate
how to detect one pixel thin edges ?
CS223, Jana Kosecka

50. Canny Edge Detector
The magnitude image Es has the magnitudes of the smoothed gradient.
Sigma determines the amount of smoothing.
Es has large values at edges:
Find local maxima Th
CS223, Jana Kosecka

51. Nonmaximum supression
The inputs are Es & Eo Magnitude and orientation
Consider 4 directions D={ 0,45,90,135} wrt x
For each pixel (i,j) do:
Find the direction dD s.t. d Eo(i,j) (normal to the edge)
If {Es(i,j) is smaller than at least one of its neigh. along d}
IN(i,j)=0
Otherwise, IN(i,j)= Es(i,j)
The output is the thinned edge image IN x
CS223, Jana Kosecka

52. Thresholding
Edges are found by thresholding the output of NONMAX_SUPRESSION
If the threshold is too high:
Very few (none) edges
High MISDETECTIONS, many gaps
If the threshold is too low:
Too many (all pixels) edges
High FALSE POSITIVES, many extra edges
CS223, Jana Kosecka

53. Hysteresis Thresholding
Es(i,j)> H
Es(i,j)<H
Es(i,j)>L
Es(i,j)<L
Strong edges reinforce adjacent weak edges
CS223, Jana Kosecka

54. Finding lines in an image
Option 1:
Search for the line at every possible position/orientation
What is the cost of this operation?
Option 2:
Use a voting scheme: Hough transform
CS223, Jana Kosecka

55. Other edge detectors - second-order derivative filters (1D)
Peaks of the first-derivative of the input signal, correspond to “zero-crossings” of the second-derivative of the input signal.
F(x)
F ’(x) x
F’’(x)
CS223, Jana Kosecka

56. Edge Detection (2D) x y 1D 2D I(x) I(x,y) dI(x) dx > Th
F(x) x 1D 2D
I(x)
I(x,y)
dI(x) dx
> Th
|I(x,y)| =(Ix 2(x,y) + Iy2(x,y))1/2 > Th
tan  = Ix(x,y)/ Iy(x,y)
d2I(x)
dx2
= 0
2I(x,y) =Ix x (x,y) + Iyy (x,y)=0
Laplacian
CS223, Jana Kosecka

57. Notes about the Laplacian:
2I(x,y) is a SCALAR
 Can be found using a SINGLE mask
 Orientation information is lost
2I(x,y) is the sum of SECOND-order derivatives
But taking derivatives increases noise
Very noise sensitive!
It is always combined with a smoothing operation:
Filter Laplacian of Gaussian LOG filter
Smooth
Laplacian
I(x,y)
O(x,y)
CS223, Jana Kosecka

58. 1D Gaussian 2D
“Mexican Hat”
CS223, Jana Kosecka

59. Derivative of Gaussian Filters
Measure the image gradient and its direction at different scales (use a pyramid).
CS223, Jana Kosecka

60. The Laplacian Pyramid Building a Laplacian pyramid:
Create a Gaussian pyramid
Take the difference between one Gaussian pyramid level and the next (before subsampling)
Properties
Also known as the difference-of-Gaussian function, which is a close approximation to the Laplacian
It is a band pass filter - each level represents a different band of spatial frequencies
Reconstructing the original image:
Reconstruct the Gaussian pyramid starting at top layer
CS223, Jana Kosecka

61. This is a Gaussian pyramid
CS223, Jana Kosecka

62. Laplacian Pyramid (note top image is from Gaussian)
And this a Laplacian pyramid. Notice that the lowest layer is the same in each case,
and that if I upsample the lowest layer and add to the next, I get the Gauss layer.
Notice also that each layer shows detail at a particular scale --- these are, basically, bandpass filtered versions of the image.
Laplacian
Pyramid
(note top image
is from Gaussian)
CS223, Jana Kosecka

63. Add more oriented filters (Malik & Perona, 1990)
Spots and bars at a fine scale for the butterfly; images show squared response for
corresponding filter
CS223, Jana Kosecka

64. Alternative: Gabor filters
Gabor filters: Product of a Gaussian with sine or cosine
Top row shows anti-symmetric
(or odd) filters, bottom row the
symmetric (or even) filters.
No obvious advantage to any one type of oriented filters.
Gabor filters are just another way to generate spots and bars.
CS223, Jana Kosecka

65. An edge is not a line...
How can we detect lines ?
CS223, Jana Kosecka