Neighborhood Operations

Objectives Why are neighborhoods important? What is linear convolution? discrete templates, masks or filters algorithm mechanics graphical interpretation Describe non-linear operators maximum minimum median What is tiling?

Why are neighbourhoods important? pixel

Because… Provide context for individual pixels. Relationships between neighbors determine image features. Neighborhood operations: noise reduction edge enhancement zooming

Noise reduction Edge Enhancement Zooming

Neighbourhood Operations Linear convolution (*) A*B*C*D = B*C*D*A = …. Non-linear operators median, max, min, ...

Convolution versus Spectral We learnt two methods of processing images: Convolution Spectral We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution. 1D convolution is used in speech processing and in polynomial multiplication. We will use visualized animations now to show in more detail how 2D convolution works for images. This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.

2D Convolution We will show more examples of convolution now, especially for 2D data Consists of filtering an image A using a filter (mask, template) B. Mask is a small image whose pixel values are called weights. Weights modify relationships between pixels.

 = A C B Filter, mask or template Input image Convolved Image A1,1 2 2 3 3 4 4

A1,1 A1,2 A2,1 A2,2 B1,1 B1,2 B2,1 B2,2 A1,3 A1,4 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4 A1,1B1,1 A1,2B1,2 A2,1B2,1 A2,2B2,2 C1,1= A1,1B1,1  A1,2B1,2  A2,1B2,1  A2,2B2,2

A1,1 A1,2 A2,1 A2,2 A1,3 A1,4 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4 B1,1 B1,2 B2,1 B2,2 A1,2B1,1 A1,3B1,2 A2,2B2,1 A2,3B2,2 C1,2= A1,2B1,1  A1,3B1,2  A2,2B2,1  A2,3B2,2

A1,1 A1,2 A2,1 A2,2 A1,3 A1,4 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4 B1,1 B1,2 B2,1 B2,2 A1,3B1,1 A1,4B1,2 A2,3B2,1 A2,4B2,2 C1,3= A1,3B1,1  A1,4B1,2  A2,3B2,1  A2,4B2,2

A1,3 A1,4 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4 A1,1 A1,2 A2,1 A2,2 A2,1B1,1 A2,2B1,2 B1,1 B1,2 B2,1 B2,2 A3,1B2,1 A3,2B2,2 C2,1= A2,1B1,1  A2,2B1,2  A3,1B2,1  A3,2B2,2

A1,1 A1,2 A2,1 A2,2 A1,3 A1,4 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4 B1,1 B1,2 B2,1 B2,2 B1,1 B1,2 B2,1 B2,2 B1,1 B1,2 B2,1 B2,2

Mathematical Notation A1,1B1,1  A1,2B1,2  A2,1B2,1  A2,2B2,2

 Convolution = A C B Filter, mask or template Input image Convolved 4 7 9 6 23 21  -1 2 4 3 8 9 = 9 26 19 -1 2 3 5 9 6 10 16 27 17 2 2 3 3 4 4

Convolution size Image size = M1 N1 Mask size = M2 N2 M1- M2 +1 N1-N2+1 N1 N2 N1-N2+1 Typical Mask sizes = 33, 5 5, 77, 9 9, 1111 What is the convolved image size for a 128   128 image and 7  7 mask?

1 1 1 * = We convolve with 9*9 averaging filter

Nonlinear Neighbourhood Operations Maximum Minimum Median We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators

Max and Min Operations C1,2= 61 62 57 60 59 65 63 56 55 58 49 53 45 1 63=max, 59=min

Median Operation 9 8 7 6 5 4 3 2 1 65 63 62 61 62 57 60 59 65 63 56 55 58 49 53 45 1 60 62 60 59 63 65 56 55 58 57 rank 59 58 57 56 C1,2= 59 55

9x9 Median

Edge Detection What do we mean by edge detection? What is an edge?

What is Edge Detection? Redraws the image with only the edges showing Detects large intensity transitions between pixels Redraws the image with only the edges showing 0 0 0 33 0 0 45 78 0 45 23 33 0 0 42 76 0 0 0 38

What is an Edge? Edge easy to find

What is an Edge? Where is edge? Single pixel wide or multiple pixels?

What is an Edge? Noise is here Noise: have to distinguish noise from actual edge

What is an Edge? Is this one edge or two?

What is an Edge? Texture discontinuity

Edge Detection – so what is an edge to be detected? A large change in image brightness of a short spatial distance Edge strength = (I(x,y)-I(x+dx,y))/dx But this general definition still allows for many theories, software implementation and hardware architectures.

Now we will discuss and illustrate various kinds of filter operators

Edge Detection Filters High - Pass Filtering Eliminates Uniform Regions (Low Frequencies) edge “detection” or “enhancement”

Edge Detection Filters

Edge Detection Filters Edge Detection Continued Sum of Kernel Coefficients = 0 differences in signs emphasize differences in pixel values reduces average image intensity Negative pixel values in output?

Edge Direction vertical horizontal diagonal

Directional High Pass Filters

Convolution Edge Detection using Sobel and similar operators

Example of Sobel Operator

Sobel Edge Detection

Convolution Application Examples We apply Sobel Operator --Edge Detection -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 Column Mask Row Mask as mask to a sub-field of a picture p0, p1, p2 p3, p4, p5 p6, p7, p8 -1 2 -1 0 0 0 1 2 1 * = (p6-p0)+2(p7-p1)+(p8-p2) The final step of the convolution equation, dividing by the weight , must be ignored We can learn from the result obviously The result of the above calculation for column mask is horizontal difference With Row Mask we will get vertical difference

Convolution Application Examples --Edge Detection with Sobel Operator The weight of a mask determines the grey level of the image after convolution. Like the weight of Sobel Mask W W= (-1) + (-2) + (-1) + 0 + 0 + 0 +1 + 2 +1= 0 The resulting image lost its “lightness” to be dark.

Sobel Operator

Sobel Operator -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 S2 = S1= -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 S2 = S1= Edge Magnitude = Edge Direction =

Comparison of Edge Detection Algorithms Sobel Canny Prewitt Ticbetts

Edge Direction Assymetric kernels detect edges from specific directions NorthEast 1 -1 -1 1 -2 -1 1 1 -1 East 1 1 -1 1 -2 -1 North -1 -1 -1 1 -2 1 1 1 1

Robinson Operator

Robinson Compass Masks -1 0 1 -2 0 2 0 1 2 -1 0 1 -2 -1 0 1 2 1 0 0 0 -1 -2 -1 2 1 0 1 0 -1 0 -1 -2 1 0 -1 2 0 -2 1 1 -1 0 -1 -2 -1 0 -1 2 1 0 -1 -2 -1 0 0 0 1 2 1 -2 -1 0 -1 0 1 0 1 2 Arrows show edge directions

Roberts Operator

Roberts Operator or 1 0 0 -1 0 1 -1 0 + Does not return any information about the orientation of the edge

Prewitt Operator -1 -1 -1 0 0 0 1 1 1 -1 0 1 P2 = P1= Edge Magnitude = -1 -1 -1 0 0 0 1 1 1 -1 0 1 P2 = P1= Edge Magnitude = Edge Direction =

Edge Detection Filters Prewitt Row

0 1 2 -1 0 1 -2 -1 0 1 2 1 0 0 0 -1 -2 -1 Original and filtered cow

Edge Detection Filters: compare Prewitt and Sobel Edge Detection (continued) First Order (Gradient) Kernels Prewitt Row 1 0 -1 Sobel Row 1 0 -1 2 0 -2 Combine Row and Column Operators

first derivative 1D Laplacian Operator second derivative

2D Laplacian Operator Convolution masks approximating a Laplacian 0 -1 0 -1 4 -1 1 -2 1 -2 4 -2 -1 -1 -1 -1 8 -1 This is just one example of Laplacian, we can use much larger window

0 -1 0 -1 4 -1 Input Mask Output

Image Processing Operations for Early Vision: Edge Detection

Reminder: Effect of Filters low high

Edges … are the important part of images intensity color edges simplest, least robust intensity color edges textures contours condensation... most difficult, most robust There are many letters B occluded by black shape here. How to find them?

Image Processing Operations Edge Detection Edges are curves in the image plane across which there is a “significant” change in image brightness. The goal of edge detection is the construction of an idealized line drawing

Pixels on edges

Edges found

Edge effects: rarely ideal edges Not all information is created equal...

Causes of edges Depth discontinuity Surface orientation discontinuity One surface occludes another Surface orientation discontinuity the edge of a block reflectance discontinuity texture or color changes illumination discontinuity shadows

Edges: causes What are they? Why? four physical events that cause image edges...

Edges: causes What are they? Why? discontinuities in four physical events that cause image edges... surface color/intensity surface normal discontinuities in depth lighting (specularities)

Edges: causes Edges are image locations with a local maximum in image gradient in the direction of that gradient (steepness)

Formal Model of Edge (cont)

Formal Model of Edge: Roberts Formal Model of Edge (cont) Formal Model of Edge: Roberts

Formal Model of Edge: Laplacian and Marr-Hildreth Formal Model of Edge (cont) Formal Model of Edge: Laplacian and Marr-Hildreth

Formal Model of Edge (cont)

Formal Model of Edge (cont)

Thresholds Thresholds are important, done before or during edge detection. original image very high threshold

Thresholds Thresholds original image very high threshold

Thresholds original image very high threshold

Thresholds original image very high threshold reasonable

Thresholds original image very high threshold reasonable too low ! this all takes time...