1. Neighborhood Operations

2. Objectives Why are neighborhoods important?
What is linear convolution?
discrete
templates, masks or filters
algorithm mechanics
graphical interpretation
Describe non-linear operators
maximum
minimum
median
What is tiling?

3. Why are neighbourhoods important?
pixel

4. Because… Provide context for individual pixels.
Relationships between neighbors determine image features.
Neighborhood operations:
noise reduction
edge enhancement
zooming

5. Noise reduction
Edge Enhancement
Zooming

6. Neighbourhood Operations
Linear convolution (*)
A*B*C*D = B*C*D*A = ….
Non-linear operators
median, max, min, ...

7. Convolution versus Spectral
We learnt two methods of processing images:
Convolution
Spectral
We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution.
1D convolution is used in speech processing and in polynomial multiplication.
We will use visualized animations now to show in more detail how 2D convolution works for images.
This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.

8. 2D Convolution
We will show more examples of convolution now, especially for 2D data
Consists of filtering an image A using a filter (mask, template) B.
Mask is a small image whose pixel values are called weights.
Weights modify relationships between pixels.

9.  = A C B Filter, mask or template Input image Convolved Image A1,1
2 2
3 3
4 4

10. A1,1
A1,2
A2,1
A2,2
B1,1
B1,2
B2,1
B2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
A1,1B1,1
A1,2B1,2
A2,1B2,1
A2,2B2,2
C1,1=
A1,1B1,1 
A1,2B1,2 
A2,1B2,1 
A2,2B2,2

11. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
A1,2B1,1
A1,3B1,2
A2,2B2,1
A2,3B2,2
C1,2=
A1,2B1,1 
A1,3B1,2 
A2,2B2,1 
A2,3B2,2

12. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
A1,3B1,1
A1,4B1,2
A2,3B2,1
A2,4B2,2
C1,3=
A1,3B1,1 
A1,4B1,2 
A2,3B2,1 
A2,4B2,2

13. A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
A1,1
A1,2
A2,1
A2,2
A2,1B1,1
A2,2B1,2
B1,1
B1,2
B2,1
B2,2
A3,1B2,1
A3,2B2,2
C2,1=
A2,1B1,1 
A2,2B1,2 
A3,1B2,1 
A3,2B2,2

14. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
B1,1
B1,2
B2,1
B2,2
B1,1
B1,2
B2,1
B2,2

15. Mathematical Notation
A1,1B1,1 
A1,2B1,2 
A2,1B2,1 
A2,2B2,2

16.  Convolution = A C B Filter, mask or template Input image Convolved4 7 9 6 23 21  -1 2 4 3 8 9 = 9 26 19 -1 2 3 5 9 6 10 16 27 17
2 2
3 3
4 4

17. Convolution size Image size = M1 N1 Mask size = M2 N2
M1- M2 +1 N1-N2+1 N1 N2
N1-N2+1
Typical Mask sizes
= 33, 5 5, 77,
9 9, 1111
What is the convolved
image size for a 128 
 128 image and
7  7 mask?

18. * =
We convolve with 9*9 averaging filter

19. Nonlinear Neighbourhood Operations
Maximum
Minimum
Median
We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators

20. Max and Min Operations C1,2= 61 62 57 60 59 65 63 56 55 58 49 53 45 1
63=max, 59=min

21. Median Operation 9 8 7 6 5 4 3 2 1 65 63 62 61 62 57 60 59 65 63 56 55 58 49 53 45 1 60 62 60 59 63 65 56 55 58 57
rank 59 58 57 56
C1,2= 59 55

22. 9x9 Median

23. Edge Detection
What do we mean by edge detection?
What is an edge?

24. What is Edge Detection? Redraws the image with only the edges showing
Detects large intensity transitions between pixels
Redraws the image with only the edges showing

25. What is an Edge?
Edge easy to find

26. What is an Edge?
Where is edge? Single pixel wide or multiple pixels?

27. What is an Edge? Noise is here
Noise: have to distinguish noise from actual edge

28. What is an Edge?
Is this one edge or two?

29. What is an Edge?
Texture discontinuity

30. Edge Detection – so what is an edge to be detected?
A large change in image brightness of a short spatial distance
Edge strength = (I(x,y)-I(x+dx,y))/dx
But this general definition still allows for many theories, software implementation and hardware architectures.

31. Now we will discuss and illustrate various kinds of filter operators

32. Edge Detection Filters
High - Pass Filtering Eliminates Uniform Regions (Low Frequencies)
edge “detection” or “enhancement”

33. Edge Detection Filters

34. Edge Detection Filters
Edge Detection Continued
Sum of Kernel Coefficients = 0
differences in signs emphasize differences in pixel values
reduces average image intensity
Negative pixel values in output?

35. Edge Direction
vertical
horizontal
diagonal

36. Directional High Pass Filters

37. Convolution Edge Detection using Sobel and similar operators

38. Example of Sobel Operator

39. Sobel Edge Detection

40. Convolution Application Examples
We apply Sobel Operator
--Edge Detection
Column Mask
Row Mask
as mask to a sub-field of a picture
p0, p1, p2
p3, p4, p5
p6, p7, p8 *
= (p6-p0)+2(p7-p1)+(p8-p2)
The final step of the convolution equation, dividing by the weight , must be ignored
We can learn from the result obviously
The result of the above calculation for column mask is horizontal difference
With Row Mask we will get vertical difference

41. Convolution Application Examples
--Edge Detection with Sobel Operator
The weight of a mask determines the grey level of the
image after convolution.
Like the weight of Sobel Mask W
W= (-1) + (-2) + (-1) = 0
The resulting image lost its “lightness” to be dark.

42. Sobel Operator

43. Sobel Operator -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 S2 = S1=
S2 =
S1=
Edge Magnitude =
Edge Direction =

44. Comparison of Edge Detection Algorithms
Sobel
Canny
Prewitt
Ticbetts

45. Edge Direction
Assymetric kernels detect edges from specific directions
NorthEast
East
North

46. Robinson Operator

47. Robinson Compass Masks
Arrows show edge directions

48. Roberts Operator

49. Roberts Operator or
1 0
0 -1
0 1 +
Does not return any information about the orientation of the edge

50. Prewitt Operator -1 -1 -1 0 0 0 1 1 1 -1 0 1 P2 = P1= Edge Magnitude =
P2 =
P1=
Edge Magnitude =
Edge Direction =

51. Edge Detection Filters
Prewitt Row

52.
Original and filtered cow

53. Edge Detection Filters: compare Prewitt and Sobel
Edge Detection (continued)
First Order (Gradient) Kernels
Prewitt Row
Sobel Row
1 0 -1
2 0 -2
Combine Row and Column Operators

54. first derivative
1D Laplacian Operator
second derivative

55. 2D Laplacian Operator Convolution masks approximating a Laplacian
This is just one example of Laplacian, we can use much larger window

56.
Input Mask Output

57. Image Processing Operations for Early Vision:
Edge Detection

58. Reminder: Effect of Filters
low
high

59. Edges … are the important part of images intensity color edges
simplest, least robust
intensity
color
edges
textures
contours
condensation...
most difficult, most robust
There are many letters B occluded by black shape here. How to find them?

60. Image Processing Operations
Edge Detection
Edges are curves in the image plane across which there is a “significant” change in image brightness.
The goal of edge detection is the construction of an idealized line drawing

61. Pixels on edges

62. Edges found

63. Edge effects: rarely ideal edges
Not all information is created equal...

64. Causes of edges Depth discontinuity Surface orientation discontinuity
One surface occludes another
Surface orientation discontinuity
the edge of a block
reflectance discontinuity
texture or color changes
illumination discontinuity
shadows

65. Edges: causes What are they? Why?
four physical events that cause image edges...

66. Edges: causes What are they? Why? discontinuities in
four physical events that cause image edges...
surface color/intensity
surface normal
discontinuities in
depth
lighting (specularities)

67. Edges: causes
Edges are image locations with a local maximum in image gradient in the direction of that gradient
(steepness)

68. Formal Model of Edge (cont)

69. Formal Model of Edge: Roberts
Formal Model of Edge (cont)
Formal Model of Edge: Roberts

70. Formal Model of Edge: Laplacian and Marr-Hildreth
Formal Model of Edge (cont)
Formal Model of Edge: Laplacian and Marr-Hildreth

71. Formal Model of Edge (cont)

72. Formal Model of Edge (cont)

73. Thresholds
Thresholds are important, done before or during edge detection.
original image
very high threshold

74. Thresholds
Thresholds
original image
very high threshold

75. Thresholds
original image
very high threshold

76. Thresholds
original image
very high threshold
reasonable

77. Thresholds original image very high threshold reasonable too low !
this all takes time...