1. Lecture 11 Neighbourhood Operations (1)
TK3813
Lecture 11 Neighbourhood Operations (1)
DR MASRI AYOB

2. Outlines Convolution and Correlation. Linear Filtering:
Low pass filtering
Mean Filtering
Gaussian Filtering
High pass filtering
High boost filtering.

3. Neighborhood Operations
Neighbourhood operations modify pixel values based on the values of nearby pixels. Convolution and Correlation are fundamental neighborhood operations.
Convolution is used to filter images for specific reasons – to remove noise, to remove motion blur, to enhance image features, etc…
Correlation is used to determine the similarity of regions of an image to other regions of interest. Used in pattern recognition and image registration.

4. Principle Objective of Enhancement
Process an image so that the result will be more suitable than the original image for a specific application.
The suitableness is up to each application.
A method which is quite useful for enhancing an image may not necessarily be the best approach for enhancing another images.

5. 2 domains Spatial Domain : (image plane) Frequency Domain :
Techniques are based on direct manipulation of pixels in an image
Frequency Domain :
Techniques are based on modifying the Fourier transform of an image
There are some enhancement techniques based on various combinations of methods from these two categories.

6. Good images For human visual For machine perception
The visual evaluation of image quality is a highly subjective process.
It is hard to standardize the definition of a good image.
For machine perception
The evaluation task is easier.
A good image is one which gives the best machine recognition results.
A certain amount of trial and error usually is required before a particular image enhancement approach is selected.

7. Spatial Domain Procedures that operate directly on pixels.
g(x,y) = T[f(x,y)]
where
f(x,y) is the input image
g(x,y) is the processed image
T is an operator on f defined over some neighborhood of (x,y)

8. Mask/Filter
Neighborhood of a point (x,y) can be defined by using a square/rectangular (common used) or circular subimage area centered at (x,y)
The center of the subimage is moved from pixel to pixel starting at the top of the corner •
(x,y)

9. Mask Processing or Filter
Neighborhood is bigger than 1x1 pixel
Use a function of the values of f in a predefined neighborhood of (x,y) to determine the value of g at (x,y)
The value of the mask coefficients determine the nature of the process.
Used in techniques
Image Sharpening
Image Smoothing

10. Terminology
Neighborhood operation work with values of the image pixels and corresponding values of a sub image.
The sub image is called:
Filter
Mask
Kernel
Template
Window
The values in a filter sub image are referred to as coefficients, rather than pixels.
Our focus will be on masks of odd sizes, e.g. 3x3, 5x5,…

11. Spatial Filtering Process
simply move the filter mask from point to point in an image.
at each point (x,y), the response of the filter at that point is calculated using a predefined relationship.

12. Smoothing Spatial Filters
used for blurring and for noise reduction
blurring is used in preprocessing steps, such as:
removal of small details from an image prior to object extraction
bridging of small gaps in lines or curves
noise reduction can be accomplished by blurring with a linear filter and also by a nonlinear filter.

13. Given a “kernel” of weights to be centered on the pixel of interest
Convolution
The value of a pixel is determined by computing a weighted sum of nearby pixels. 82 78 88 65 56 76 60 53 72
Compute the new value of the center pixel by “overlaying” the kernel and computing the weighted sum 1 -1 2 -2
Given a “kernel” of weights to be centered on the pixel of interest
Note: that this only applies to kernels that have dimensions 3x3. Since the operations revolve around a particular pixel, neighbourhoods are always of odd dimensions (3x3, 5x5, 7x7...). The neighbourhoods are nearly always square too.

14. Convolution -1 0 +1 Location (x,y) +1 0 -1 56 j x g(x,y) = -1 * 82 +-1 2 -2 j k x y
Location (x,y) 72 53 60 76 56 65 88 78 82
g(x,y) = -1 * 82 +
0 * 78 +
1 * 88 +
-2 * 65 +
0 * 56 +
2 * 76 +
-1 * 60 +
0 * 53 +
1 * 72
g(x,y) = h(-1, -1) * f(x+1, y+1) +
h(0, -1) * f(x, y+1) +
h(1, -1) * f(x-1, y+1) +
h(-1, 0) * f(x+1, y) +
h(0, 0) * f(x, y) +
h(1, 0) * f(x-1, y) +
h(-1, 1) * f(x+1, y-1) +
h(0, 1) * f(x, y-1) +
h(1, 1) * f(x-1, y-1).

15. Convolution

16. Convolution The summation is expressed as:
The book is “technically” correct in using this formulation but most implementations and many books directly align the kernel with the image and compute the weighted sum.
For a kernel of width M and height N the more general formula becomes:
(7.5)

17. Convolution
Convolution is used so frequently in certain domains that it has been given the following shorthand notation:
The above formula implies convolution at a single pixel. To indicate convolution of an entire image with a kernel we write:

18. Convolution (Implementation Details)
Consider the following issues whether they affect implementation
The weights may be real-values numbers
The range of values of the output may be significantly changed by the weights
What to do about corners and edges?

19. Convolution (Implementation Details)
The weights may be real-valued numbers
The resulting pixel values must either be quantized or maintained with real-valued pixel intensities!
The range of values of the output may be significantly changed by the weights
Must increase the pixel resolution or
re-scale the computed image

20. Convolution (Implementation Details)
What about borders?
Ignore pixels where the kernel “falls off” the image.
Output pixels may be set to zero
Input pixel may be copied to the output
Truncate the output image
Truncate the kernel!
The kernel is made smaller for processing borders, edges
Copy last line
Use “circular” or “reflected” indexing
The most common way is just to ignore them and have an output image slightly smaller than the input. Other techniques include truncating the kernel to process the edge pixels correctly, but this can be complex to implement.

21. Convolution (Implementation Details)
Reflected Indexing
Pretends the image is “tiled” at each border by a mirror
Imagine a mirror vertically placed at each border that “reflects” the image back upon itself
Reflection of X component
Let M be the width of the image
if x < 0 then
x = -x-1
else if x >= M then
x = 2M-x-1
end

22. Convolution

23. Convolution Circular Indexing
Pretends the image is infinitely repeated at each border.
Sometimes a good theoretical reason for doing this.
Circular indexing of X component
Let M be the width of the image
if x < 0 then
x = x+M
else if x >= M then
x = x-M
end

24. Convolution Computation

25. Convolution Coding Java has built-in classes to support convolution.
The code is typically (at least on Windows boxes) implemented in native code (usually C).

26. Convolution Coding
int width = 3, height = 3;
float[] coeffs = new float[width*height];
for(int i=0; i<coeffs.length; i++) {
coeffs[i] = 1.0f/coeff.length; }
Kernel kernel = new Kernel(width, height, coeffs);
The Kernel class is used specifically for convolution operations.
The ConvolveOp implements the BufferedImageOp and filters images by performing a convolution on an image.

27. Convolution Coding
ConvolveOp op = new ConvolveOp(kernel);
BufferedImage image = op.filter(inputImage, null);
The “filter” method returns a gray-scale image if the input is gray-scale
The ConvolveOp class places “zeros” at borders by default. One other option is available by using a different constructor
ConvolveOp op1 = new ConvolveOp(kernel, ConvolveOp.EDGE_ZERO_FILL, null);
ConvolveOp op2 = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null);
EDGE_ZERO_FILL: default. Place zeros on the border
EDGE_NO_OP: copy border pixels from input directly to output

28. Convolution Code The built-in convolution code has some limitations:
Only two ways of dealing with borders:
EDGE_ZERO_FILL
EDGE_NO_OP
Would like to do
COPY_BORDER_PIXELS
REFLECTED_INDEXING
CIRCULAR_INDEXING
Always truncates (without rescaling) all pixel values
Would like to rescale in various ways.
Doesn’t take advantage of separable kernels
Maybe we can write more flexible code!

29. StandardGreyOp Review
public class StandardGreyOp implements BufferedImageOp {
public BufferedImage filter(BufferedImage src, BufferedImage dest) {
checkImage(src);
if (dest == null)
dest = createCompatibleDestImage(src, null);
WritableRaster raster = dest.getRaster();
src.copyData(raster);
return dest; }
public BufferedImage createCompatibleDestImage(BufferedImage src,
ColorModel destModel) {
if (destModel == null)
destModel = src.getColorModel();
int width = src.getWidth();
int height = src.getHeight();
BufferedImage image = new BufferedImage(destModel,
destModel.createCompatibleWritableRaster(width, height),
destModel.isAlphaPremultiplied(), null);
return image;
// other methods here ///

30. Convolution Code public class NeighbourhoodOp extends StandardGreyOp {
public static final int NO_BORDER_OP = 1;
public static final int COPY_BORDER_PIXELS = 2;
public static final int REFLECTED_INDEXING = 3;
public static final int CIRCULAR_INDEXING = 4;
protected int width, height, size;
protected int borderStrategy;
public NeighbourhoodOp(int w, int h, int strategy) {
if (w < 1 || h < 1 || w%2 == 0 || h%2 == 0)
throw new ImagingOpException("invalid neighbourhood dimensions");
width = w; height = h; size = w*h;
borderStrategy = strategy; }
public static final int refIndex(int i, int n) {
if (i < 0) return -i-1;
else if (i >= n) return 2*n-i-1;
else return i;
public static final int circIndex(int i, int n) {
if (i < 0) return i+n;
else if (i >= n) return i-n;

31. Convolution Code public class NeighbourhoodOp extends StandardGreyOp {
protected void copyBorders(Raster src, WritableRaster dest) {
int w = src.getWidth();
int h = src.getHeight();
int m = width/2;
int n = height/2;
for (int x = 0; x < w; ++x) { // copy top and bottom
for (int y = 0; y < n; ++y)
dest.setSample(x, y, 0, src.getSample(x, y, 0));
for (int y = h-n; y < h; ++y) }
for (int y = 0; y < h; ++y) { // copy left and right
for (int x = 0; x < m; ++x)
for (int x = w-m; x < w; ++x)

32. Convolution Code public class ConvolutionOp extends NeighbourhoodOp {
public static final int SINGLE_PASS = 1;
public static final int SEPARABLE = 2;
public static final int NO_RESCALING = 1;
/** Indicates that maximum output value should be scaled to 255. */
public static final int RESCALE_MAX_ONLY = 2;
/** Indicates that range should be scaled to */
public static final int RESCALE_MIN_AND_MAX = 3;
private Kernel kernel;
private int calculation; /** Calculation method (single pass or separable). */
private int rescaleStrategy;
public ConvolutionOp(Kernel kernel) {
this(kernel, NO_BORDER_OP, SINGLE_PASS, NO_RESCALING); }
public ConvolutionOp(Kernel kernel, int border, int calc, int rescale) {
super(kernel.getWidth(), kernel.getHeight(), border);
this.kernel = kernel;
calculation = calc;
rescaleStrategy = rescale;
///…other stuff here…///

33. Convolution Code public class ConvolutionOp extends NeighbourhoodOp {
public float[] convolve(BufferedImage image) {
int w = image.getWidth();
int h = image.getHeight();
Raster raster = image.getRaster();
float[] result = new float[w*h];
float[] coeff = kernel.getKernelData(null);
int m = width/2, n = height/2;
float sum;
int i, j, k, x, y;
switch (borderStrategy) {
case REFLECTED_INDEXING:
for (y = 0; y < h; ++y)
for (x = 0; x < w; ++x) {
for (sum = 0.0f, i = 0, k = -n; k <= n; ++k)
for (j = -m; j <= m; ++j, ++i)
sum += coeff[i] * raster.getSample(refIndex(x-j, w), refIndex(y-k, h), 0);
result[y*w+x] = sum; }
break;
case CIRCULAR_INDEXING:
sum += coeff[i] * raster.getSample(circIndex(x-j, w), circIndex(y-k, h), 0);
///…rest of code here…//

34. Convolution Code public class ConvolutionOp extends NeighbourhoodOp {
public BufferedImage filter(BufferedImage src, BufferedImage dest) {
checkImage(src);
if (dest == null)
dest = createCompatibleDestImage(src, null);
float[] rawData;
if (calculation == SEPARABLE) {
rawData = separableConvolve(src);
} else {
rawData = convolve(src); }
DataBufferByte buffer = (DataBufferByte) dest.getRaster().getDataBuffer();
convertToBytes(rawData, buffer.getData());
return dest;
protected void convertToBytes(float[] in, byte[] out) {
if (rescaleStrategy == NO_RESCALING) {
for (int i = 0; i < in.length; ++i) {
int value = Math.round(in[i]);
if (value < 0)
out[i] = (byte) 0;
else if (value > 255)
out[i] = (byte) 255;
else
out[i] = (byte) value;
// other cases go here //

35. Convolution

36. Convolution

37. Convolution

38. Frequency
Frequency of a sound wave or audio signal – referring to the rate at which the signal changes with time.
Frequency in an image - referring to changes occuring in space.
Spatial frequency is a measure of how rapidly brightness or colours varies as we traverse an image.
Images in which grey level varies slowly and smoothly are characterised solely by components with low spatial frequncy.
Images containing sudden grey level transitions, fine detail or strong texture will also contain components with high spatial frequencies.

40. Filtering
Convolution will have different effects depending upon the values of the Kernel. Convolution is an operation taken between images. (1) image data (2) kernel. Primary technique for spatial filtering. Convolution is a linear operation that can be undone.
Filtering is a way of tuning image frequencies – much like a graphic equalizer.
Linear filtering

41. Filtering

42. Filtering

43. Filtering Low pass filter: High pass filter:
Allows low spatial frequencies to pass unchanged.
Suppresses high frequencies.
Smoothes or blurs the image.
Tend to reduce noise but also obscures fine detail.
High pass filter:
Allows high spatial frequencies to pass unchanged.
Suppresses low frequencies.
Preserves sudden variation, such as those that occur at the boundaries of objects.
But suppresses the more gradual variation.

44. Filtering

45. Filtering

46. Low-Pass Filtering

47. Low-Pass Filtering

48. Low-Pass Filtering

49. Consider the two kernels shown above. What do they do?
Low-Pass Filtering
Average kernel : Equally weighted sum of all pixels in a neighborhood
Any kernel having all positive coefficients will act as a low-pass filter.
0.04
.111
Consider the two kernels shown above. What do they do?
Their coefficient sum to 1.
Convolution with them will not result in an overall brightening of the image.

50. Low-Pass Filtering = 1/9 *
Any kernel having all positive coefficients will act as a low-pass filter.
.111 1
= 1/9 *
The center pixel becomes the average of all neighboring pixels. Also known as a mean filter.

51. Low-Pass Filter
Pixel values from the neighbourhood are summed without being weighted.
The sum is divided by the number of pixels in the neighbourhood.
Convolution with these kernels is therefore equivalent to computing the mean grey level over the neighbourhood defined by the kernel.
These kernels are sometimes described as mean filters.

52. Low-Pass Filtering

53. Low-Pass Filtering

54. Mean Filtering Mean filters are good at removing noise.
Mean filters “blur” or “smooth” edges (by damping high frequency components and resisting fast changes in intensities)
Kernels are typically normalized so that they sum to 1

55. Mean Filtering
( )/9

56. Gaussian Filter
The Gaussian filter is a 2-D convolution operator that is used to `blur' images and remove detail and noise much like the mean filter.
It is similar to the mean filter, but it uses a different kernel that represents the shape of a Gaussian (`bell-shaped') hump. This kernel has some special properties which are detailed below.
The degree of smoothing is determined by the standard deviation of the Gaussian.
Larger standard deviation Gaussians, of course, require larger convolution kernels in order to be accurately represented.
The Gaussian outputs a `weighted average' of each pixel's neighborhood, with the average weighted more towards the value of the central pixels.
This is in contrast to the mean filter's uniformly weighted average.
A Gaussian provides gentler smoothing and preserves edges better than an identically sized mean filter.

57. 3D plot of the Gaussian filter

58. Gaussian Filter The Gaussian kernel is separable and symmetric.
To construct the kernel we must sample and quantize! (basically, we “image” the function)
The kernel below is an example where sigma = 1. 1 4 7 16 28 49

59. Gaussian Filter
15 x 15
Larger kernels result -> more blur

60. Gaussian Filter

61. 5x5 Mean Kernel
5x5 Gaussian Kernel

62. Noise Reduction
Gaussian and mean filters are usually used to reduce “noise” in images.
The above image has been corrupted by impulse noise.

63. Noise Reduction

64. Noise Reduction

65. Sharpening Filters (High-Pass Filtering)
These filters highlight fine image detail or de-blur an image.
Highpass filter: allows only high-frequency information through.
Main feature is a positive center coefficient and negative perimeter values.
The sum of the coefficients is zero, which means that areas of constant intensity are completely eliminated. -1 8

66. High-Pass Filtering
The sum of the coefficients in this kernel is zero.
This means that, when the kernel is over an area of constant or slowly varying grey level, the result of convolution is zero or some very small number.
However, when grey level is varying rapidly within the neighbourhood, the result of convolution can be a large number (+ve or –ve).
Need to choose an output image representation that support negative numbers.
If we wish to display/print image, we must map the pixel values onto a range.
Usually map 0 onto the middle of the range.
Thus, negative filter responses will show up as dark tones, whereas positive responses will be represented by light tones.

67. Sharpening Filters

68. Sharpening Filters

69. HighPass Examples

70. HighPass Example

71. HighPass Example

72. High Boost Filtering
An image can be sharpened by high-boost filtering, an operation that emphasises the high spatial frequencies present in that image. In the spatial domain, this can be accomplished by convolution with a kernel of the form
where c > 8. Larger values give more weight to a pixel's true value and less to the difference between it and its surroundings, thereby reducing the sharpening effect.
As c get closer to 8, the degree of sharpening increases.
If c=8, the kernel becomes the high pass filter.
Keeps the “original” image while enhancing (boosting) the high-frequency components. -1 c

73. Example: High boost filtering

74. Thank you Q&A