1. Edges and Binary Images
Dan Witzner Hansen

2. Previously???

3. What filter is likely to have produced this image output?
original
filtered output

4. Is the meadian filter a linear filter (convolution)?
5 x 5 median filter
What is the purpose of smoothing?
Is that a linear filter – can it be done with covolution
25 x 25 median filter
50 x 50 median filter

5. Edge detection
Goal: map image from 2d array of pixels to a set of curves or line segments or contours.
Why?
Main idea: look for strong gradients, post-process
Figure from J. Shotton et al., PAMI 2007

6. What can cause an edge? Depth discontinuity: object boundary
Reflectance change: appearance information, texture
Cast shadows
Change in surface orientation: shape

7. Derivatives and edges
An edge is a place of rapid change in the image intensity function.
intensity function (along horizontal scanline)
image
first derivative
edges correspond to extrema of derivative
Source: L. Lazebnik

8. Recall : Images as functions
Edges look like steep cliffs
Source: S. Seitz

9. Differentiation and convolution
For 2D function, f(x,y), the partial derivative is:
For discrete data, we can approximate using finite differences:
To implement above as convolution, what would be the associated filter?

10. Partial derivatives of an image? or
Which shows changes with respect to x?
(showing flipped filters)

11. Assorted finite difference filters
>> My = fspecial(‘sobel’);
>> outim = imfilter(double(im), My);
>> imagesc(outim);
>> colormap gray;

12. Image gradient The gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction (orientation of edge normal) is given by:
The edge strength is given by the gradient magnitude
Slide credit S. Seitz 12

13. Eye Image example

14. Effects of noise Consider a single row or column of the image
Plotting intensity as a function of position gives a signal
Where is the edge?

15. Solution: smooth first
Where is the edge?
Look for peaks in

16. Derivative theorem of convolution
Differentiation property of convolution.

17. Derivative of Gaussian filter
Handles both in one go
Less Computation
Why is this preferable?

18. Derivative of Gaussian filters
x-direction
y-direction
Cells in human vision system
Bemærk Haar features frpm last week
Source: L. Lazebnik

19. Laplacian of Gaussian Consider Laplacian of Gaussian operator
David marr
Why is this possible (diff with convolution)
Where is the edge?
Zero-crossings of bottom graph

20. 2D edge detection filters
Laplacian of Gaussian
Gaussian
derivative of Gaussian
is the Laplacian operator (trace of Hessian): 20

21. Derivatives of Gaussian

22. Mask properties Smoothing Derivatives Filters act as templates
Values positive
Sum to 1  constant regions same as input
Amount of smoothing proportional to mask size
Remove “high-frequency” components; “low-pass” filter
Derivatives
Opposite signs used to get high response in regions of high contrast
Sum to 0  no response in constant regions
High absolute value at points of high contrast
Filters act as templates
Highest response for regions that “look the most like the filter”
Dot product as correlation

23. Gradients -> edges Primary edge detection steps:
1. Smoothing: suppress noise
2. Edge enhancement: filter for contrast
3. Edge localization
Determine which local maxima from filter output are actually edges vs. noise
Threshold, Thin ( more about this later)

24. Smoothing with a Gaussian
Recall: parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing. …

25. Effect of on derivatives
σ = 1 pixel
σ = 3 pixels
The apparent structures differ depending on Gaussian’s scale parameter.
Larger values: larger scale edges detected
Smaller values: finer features detected

26. So, what scale to choose?
Too fine of a scale…can’t see the forest for the trees.
Too coarse of a scale…can’t tell the maple grain from the cherry.

27. Original image

28. Thresholding gradient with a lower threshold

29. Source: D. Lowe, L. Fei-Fei
Canny edge detector
Filter image with derivative of Gaussian
Find magnitude and orientation of gradient
Non-maximum suppression:
Thin multi-pixel wide “ridges” down to single pixel width
Linking and thresholding (hysteresis):
Define two thresholds: low and high
Use the high threshold to start edge curves and the low threshold to continue them
MATLAB: edge(image, ‘canny’);
>>help edge
Source: D. Lowe, L. Fei-Fei

30. The Canny edge detector
original image (Lena)

31. The Canny edge detector
norm of the gradient

32. The Canny edge detector
thresholding

33. The Canny edge detector
How to turn these thick regions of the gradient into curves?
thresholding

34. Non-maximum suppression
Check if pixel is local maximum along gradient direction, select single max across width of the edge
requires checking interpolated pixels p and r

35. The Canny edge detector
Problem: pixels along this edge didn’t survive the thresholding
thinning
(non-maximum suppression)

36. Hysteresis thresholding
Check that maximum gradient value is sufficiently large
drop-outs? use hysteresis
use a high threshold to start edge curves and a low threshold to continue them.
Source: S. Seitz

37. Hysteresis thresholding
original image
high threshold
(strong edges)
low threshold
(weak edges)
hysteresis threshold
Source: L. Fei-Fei

38. Object boundaries vs. edges
Background
Texture
Shadows

39. Seam Carving uses edges to be able to resize images
Try it.

40. Thresholding What is thresholding? Choose a threshold value t
Set any pixels less than t to zero (off)
Set any pixels greater than or equal to t to one (on)

41. Binary images

42. Binary images Two pixel values Foreground and background
Mark region(s) of interest

43. Thresholding
Given a grayscale image or an intermediate matrix  threshold to create a binary output.
Example: edge detection
Gradient magnitude
fg_pix = find(gradient_mag > t);
Looking for pixels where gradient is strong.

44. Thresholding
Given a grayscale image or an intermediate matrix  threshold to create a binary output.
Example: background subtraction = -
Looking for pixels that differ significantly from the “empty” background.
fg_pix = find(diff > t);

45. fg_pix = find(im < 65);
Thresholding
Given a grayscale image or an intermediate matrix  threshold to create a binary output.
Example: intensity-based detection
fg_pix = find(im < 65);
Looking for dark pixels

46. Thresholding
Given a grayscale image or an intermediate matrix  threshold to create a binary output.
Example: color-based detection
fg_pix = find(hue > t1 & hue < t2);
Looking for pixels within a certain hue range.

47. Issues How to indentify multiple regions of interest?
Count objects
Compute further features per object
What to do with “noisy” binary outputs?
Holes
Extra small fragments

48. Issues How to demarcate multiple regions of interest?
Count objects
Compute further features per object
What to do with “noisy” binary outputs?
Holes
Extra small fragments

49. Morphological operators
Change the shape of the foreground regions/ objects.
Useful to clean up result from thresholding
Basic operators are:
Dilation
Erosion
Combine
Opening
Closing

50. Dilation Expands connected components Grow features Fill holes
Before dilation
After dilation

51. Erosion Erode connected components Shrink features
Remove bridges, branches, noise
Before erosion
After erosion

52. Structuring elements
Masks of varying shapes and sizes used to perform morphology, for example:
Scan mask across foreground pixels to transform the binary image
>> help strel

53. Dilation vs. Erosion At each position:
Dilation: if current pixel is foreground, OR the structuring element with the input image.
Eroding foreground = dilating background

54. Example for Dilation (1D)
Input image 1 1
Structuring Element
Output Image 1
Adapted from T. Moeslund

55. Example for Dilation 1 1 1 1 1 Input image Structuring Element
Structuring Element 1 1 1
Output Image 1

56. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1

57. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1

58. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1

59. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1

60. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1

61. Example for Dilation 1 1 1 1 1 Input image Structuring Element
Structuring Element 1 1 1
Output Image 1

62. Example for Dilation 1 1 1 Input image Structuring Element1
Structuring Element
Output Image 1
Note that the object gets bigger and holes are filled.
>> help imdilate

63. Dilation vs. Erosion At each position:
Dilation: if current pixel is foreground, OR the structuring element with the input image.
Erosion: if every pixel under the structuring element’s nonzero entries is foreground, OR the current pixel with S.

64. Example for Erosion (1D)
Input image 1 1
Structuring Element _
Output Image

65. Example for Erosion (1D)
Input image 1 1
Structuring Element _
Output Image

66. Example for Erosion
Input image 1 1
Structuring Element
Output Image

67. Example for Erosion
Input image 1 1
Structuring Element
Output Image

68. Example for Erosion
Input image 1 1
Structuring Element
Output Image

69. Example for Erosion
Input image 1 1
Structuring Element
Output Image 1

70. Example for Erosion
Input image 1 1
Structuring Element
Output Image 1

71. Example for Erosion
Input image 1 1
Structuring Element
Output Image 1

72. Example for Erosion
Input image 1 1
Structuring Element
Output Image 1

73. Example for Erosion 1 1 1 Input image Structuring Element Output Image1
Structuring Element
Output Image 1
Note that the object gets smaller
>> help imerode

74. Erosion example

75. Opening Erode, then dilate Remove small objects, keep original shape
Idempotent: Repeated operations has no further effects!
Before opening
After opening

76. Closing Dilate, then erode Fill holes, but keep original shape
Idempotent
Before closing
After closing

77. Combine Open and close

78. Morphology operators on grayscale images
Dilation and erosion typically performed on binary images.
If image is grayscale: for dilation take the neighborhood max, for erosion take the min.
original
dilated
eroded

79. Matlab N = hist(Y,M) L = bwlabel (BW,N);
STATS = regionprops(L,PROPERTIES) ;
'Area'
'Centroid'
'BoundingBox'
'Orientation‘, …
IM2 = imerode(IM,SE);
IM2 = imdilate(IM,SE);
IM2 = imclose(IM, SE);
IM2 = imopen(IM, SE);

80. Example using binary image analysis: segmentation of a liver
Slide credit: Li Shen

81. Example using binary image analysis: Bg subtraction + blob detection…
How to find the location on the floor?

82. Connected components Identify distinct regions of “connected pixels”
Shapiro and Stockman

83. Connectedness Defining which pixels are considered neighbors

84. Connected components
We’ll consider a sequential algorithm that requires only 2 passes over the image.
Input: binary image
Output: “label” image, where pixels are numbered per their component
[Note, in this example, “foreground” is denoted with black.]

85. Depth First Search Goal. Find all vertices connected to s.
Depth first search. To visit a vertex v:
Mark v as visited.
Recursively visit all unmarked vertices w adjacent to v.
Running time. O(E) since each edge examined at most twice.

86. Reachability Application: Flood Fill
Flood fill. Given lime green pixel in an image, change color of entire blob of neighboring lime pixels to blue.
Vertex: pixel.
Edge: two neighboring lime pixels.
Blob: all pixels reachable from chosen lime pixel.
recolor lime green blob to blue
recolor lime green blob to blue

87. Sequential connected components
For an 8-neighborhood, the nw, n, ne, and w neighbors are considered.
Slide from J. Neira

88. Sequential connected components

89. Region properties
Given connected components, can compute simple features per blob, such as:
Area (number of pixels in the region)
Centroid (average x and y position of pixels in the region)
Bounding box (min and max coordinates)
How could such features be useful?
A2=170
A1=200

90. Summary
Filters allow local image neighborhood to influence our description and features
Smoothing to reduce noise
Derivatives to locate contrast, gradient
Templates, matched filters to find designated pattern.
Edge detection processes the image gradient to find curves, or chains of edges.
Binary image analysis useful to manipulate regions of interest
Connected components
Morphological operators (local operator, but different than convolution)

91. Assignment1 – Eye /Gaze tracking

92. What to be done Week 5. Simple eye tracking
Week 6. Contour detection & Fitting
Week 7. Gaze estimation (may be subject to change)

93. Try the techniques learned today in the assignment to see if they could can improve your results . You might be lucky 

94. Summary Operations, tools Features, representations Derivative filters
Smoothing, morphology
Thresholding
Connected components
Matched filters
Histograms
Edges, gradients
Blobs/regions
Color distributions
Local patterns