Lecture 11 Neighbourhood Operations (1) TK3813 Lecture 11 Neighbourhood Operations (1) DR MASRI AYOB

Outlines Convolution and Correlation. Linear Filtering: Low pass filtering Mean Filtering Gaussian Filtering High pass filtering High boost filtering.

Neighborhood Operations Neighbourhood operations modify pixel values based on the values of nearby pixels. Convolution and Correlation are fundamental neighborhood operations. Convolution is used to filter images for specific reasons – to remove noise, to remove motion blur, to enhance image features, etc… Correlation is used to determine the similarity of regions of an image to other regions of interest. Used in pattern recognition and image registration.

Principle Objective of Enhancement Process an image so that the result will be more suitable than the original image for a specific application. The suitableness is up to each application. A method which is quite useful for enhancing an image may not necessarily be the best approach for enhancing another images.

2 domains Spatial Domain : (image plane) Frequency Domain : Techniques are based on direct manipulation of pixels in an image Frequency Domain : Techniques are based on modifying the Fourier transform of an image There are some enhancement techniques based on various combinations of methods from these two categories.

Good images For human visual For machine perception The visual evaluation of image quality is a highly subjective process. It is hard to standardize the definition of a good image. For machine perception The evaluation task is easier. A good image is one which gives the best machine recognition results. A certain amount of trial and error usually is required before a particular image enhancement approach is selected.

Spatial Domain Procedures that operate directly on pixels. g(x,y) = T[f(x,y)] where f(x,y) is the input image g(x,y) is the processed image T is an operator on f defined over some neighborhood of (x,y)

Mask/Filter Neighborhood of a point (x,y) can be defined by using a square/rectangular (common used) or circular subimage area centered at (x,y) The center of the subimage is moved from pixel to pixel starting at the top of the corner • (x,y)

Mask Processing or Filter Neighborhood is bigger than 1x1 pixel Use a function of the values of f in a predefined neighborhood of (x,y) to determine the value of g at (x,y) The value of the mask coefficients determine the nature of the process. Used in techniques Image Sharpening Image Smoothing

Terminology Neighborhood operation work with values of the image pixels and corresponding values of a sub image. The sub image is called: Filter Mask Kernel Template Window The values in a filter sub image are referred to as coefficients, rather than pixels. Our focus will be on masks of odd sizes, e.g. 3x3, 5x5,…

Spatial Filtering Process simply move the filter mask from point to point in an image. at each point (x,y), the response of the filter at that point is calculated using a predefined relationship.

Smoothing Spatial Filters used for blurring and for noise reduction blurring is used in preprocessing steps, such as: removal of small details from an image prior to object extraction bridging of small gaps in lines or curves noise reduction can be accomplished by blurring with a linear filter and also by a nonlinear filter.

Given a “kernel” of weights to be centered on the pixel of interest Convolution The value of a pixel is determined by computing a weighted sum of nearby pixels. 82 78 88 65 56 76 60 53 72 Compute the new value of the center pixel by “overlaying” the kernel and computing the weighted sum 1 -1 2 -2 Given a “kernel” of weights to be centered on the pixel of interest -1 0 +1 +1 0 -1 Note: that this only applies to kernels that have dimensions 3x3. Since the operations revolve around a particular pixel, neighbourhoods are always of odd dimensions (3x3, 5x5, 7x7...). The neighbourhoods are nearly always square too.

Convolution -1 0 +1 Location (x,y) +1 0 -1 56 j x g(x,y) = -1 * 82 + -1 2 -2 -1 0 +1 +1 0 -1 j k x y Location (x,y) 72 53 60 76 56 65 88 78 82 g(x,y) = -1 * 82 + 0 * 78 + 1 * 88 + -2 * 65 + 0 * 56 + 2 * 76 + -1 * 60 + 0 * 53 + 1 * 72 g(x,y) = h(-1, -1) * f(x+1, y+1) + h(0, -1) * f(x, y+1) + h(1, -1) * f(x-1, y+1) + h(-1, 0) * f(x+1, y) + h(0, 0) * f(x, y) + h(1, 0) * f(x-1, y) + h(-1, 1) * f(x+1, y-1) + h(0, 1) * f(x, y-1) + h(1, 1) * f(x-1, y-1).

Convolution

Convolution The summation is expressed as: The book is “technically” correct in using this formulation but most implementations and many books directly align the kernel with the image and compute the weighted sum. For a kernel of width M and height N the more general formula becomes: (7.5)

Convolution Convolution is used so frequently in certain domains that it has been given the following shorthand notation: The above formula implies convolution at a single pixel. To indicate convolution of an entire image with a kernel we write:

Convolution (Implementation Details) Consider the following issues whether they affect implementation The weights may be real-values numbers The range of values of the output may be significantly changed by the weights What to do about corners and edges?

Convolution (Implementation Details) The weights may be real-valued numbers The resulting pixel values must either be quantized or maintained with real-valued pixel intensities! The range of values of the output may be significantly changed by the weights Must increase the pixel resolution or re-scale the computed image

Convolution (Implementation Details) What about borders? Ignore pixels where the kernel “falls off” the image. Output pixels may be set to zero Input pixel may be copied to the output Truncate the output image Truncate the kernel! The kernel is made smaller for processing borders, edges Copy last line Use “circular” or “reflected” indexing The most common way is just to ignore them and have an output image slightly smaller than the input. Other techniques include truncating the kernel to process the edge pixels correctly, but this can be complex to implement.

Convolution (Implementation Details) Reflected Indexing Pretends the image is “tiled” at each border by a mirror Imagine a mirror vertically placed at each border that “reflects” the image back upon itself Reflection of X component Let M be the width of the image if x < 0 then x = -x-1 else if x >= M then x = 2M-x-1 end

Convolution

Convolution Circular Indexing Pretends the image is infinitely repeated at each border. Sometimes a good theoretical reason for doing this. Circular indexing of X component Let M be the width of the image if x < 0 then x = x+M else if x >= M then x = x-M end

Convolution Computation

Convolution Coding Java has built-in classes to support convolution. The code is typically (at least on Windows boxes) implemented in native code (usually C).

Convolution Coding int width = 3, height = 3; float[] coeffs = new float[width*height]; for(int i=0; i<coeffs.length; i++) { coeffs[i] = 1.0f/coeff.length; } Kernel kernel = new Kernel(width, height, coeffs); The Kernel class is used specifically for convolution operations. The ConvolveOp implements the BufferedImageOp and filters images by performing a convolution on an image.

Convolution Coding ConvolveOp op = new ConvolveOp(kernel); BufferedImage image = op.filter(inputImage, null); The “filter” method returns a gray-scale image if the input is gray-scale The ConvolveOp class places “zeros” at borders by default. One other option is available by using a different constructor ConvolveOp op1 = new ConvolveOp(kernel, ConvolveOp.EDGE_ZERO_FILL, null); ConvolveOp op2 = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null); EDGE_ZERO_FILL: default. Place zeros on the border EDGE_NO_OP: copy border pixels from input directly to output

Convolution Code The built-in convolution code has some limitations: Only two ways of dealing with borders: EDGE_ZERO_FILL EDGE_NO_OP Would like to do COPY_BORDER_PIXELS REFLECTED_INDEXING CIRCULAR_INDEXING Always truncates (without rescaling) all pixel values Would like to rescale in various ways. Doesn’t take advantage of separable kernels Maybe we can write more flexible code!

StandardGreyOp Review public class StandardGreyOp implements BufferedImageOp { public BufferedImage filter(BufferedImage src, BufferedImage dest) { checkImage(src); if (dest == null) dest = createCompatibleDestImage(src, null); WritableRaster raster = dest.getRaster(); src.copyData(raster); return dest; } public BufferedImage createCompatibleDestImage(BufferedImage src, ColorModel destModel) { if (destModel == null) destModel = src.getColorModel(); int width = src.getWidth(); int height = src.getHeight(); BufferedImage image = new BufferedImage(destModel, destModel.createCompatibleWritableRaster(width, height), destModel.isAlphaPremultiplied(), null); return image; // other methods here ///

Convolution Code public class NeighbourhoodOp extends StandardGreyOp { public static final int NO_BORDER_OP = 1; public static final int COPY_BORDER_PIXELS = 2; public static final int REFLECTED_INDEXING = 3; public static final int CIRCULAR_INDEXING = 4; protected int width, height, size; protected int borderStrategy; public NeighbourhoodOp(int w, int h, int strategy) { if (w < 1 || h < 1 || w%2 == 0 || h%2 == 0) throw new ImagingOpException("invalid neighbourhood dimensions"); width = w; height = h; size = w*h; borderStrategy = strategy; } public static final int refIndex(int i, int n) { if (i < 0) return -i-1; else if (i >= n) return 2*n-i-1; else return i; public static final int circIndex(int i, int n) { if (i < 0) return i+n; else if (i >= n) return i-n;

Convolution Code public class NeighbourhoodOp extends StandardGreyOp { protected void copyBorders(Raster src, WritableRaster dest) { int w = src.getWidth(); int h = src.getHeight(); int m = width/2; int n = height/2; for (int x = 0; x < w; ++x) { // copy top and bottom for (int y = 0; y < n; ++y) dest.setSample(x, y, 0, src.getSample(x, y, 0)); for (int y = h-n; y < h; ++y) } for (int y = 0; y < h; ++y) { // copy left and right for (int x = 0; x < m; ++x) for (int x = w-m; x < w; ++x)

Convolution Code public class ConvolutionOp extends NeighbourhoodOp { public static final int SINGLE_PASS = 1; public static final int SEPARABLE = 2; public static final int NO_RESCALING = 1; /** Indicates that maximum output value should be scaled to 255. */ public static final int RESCALE_MAX_ONLY = 2; /** Indicates that range should be scaled to 0-255. */ public static final int RESCALE_MIN_AND_MAX = 3; private Kernel kernel; private int calculation; /** Calculation method (single pass or separable). */ private int rescaleStrategy; public ConvolutionOp(Kernel kernel) { this(kernel, NO_BORDER_OP, SINGLE_PASS, NO_RESCALING); } public ConvolutionOp(Kernel kernel, int border, int calc, int rescale) { super(kernel.getWidth(), kernel.getHeight(), border); this.kernel = kernel; calculation = calc; rescaleStrategy = rescale; ///…other stuff here…///

Convolution Code public class ConvolutionOp extends NeighbourhoodOp { public float[] convolve(BufferedImage image) { int w = image.getWidth(); int h = image.getHeight(); Raster raster = image.getRaster(); float[] result = new float[w*h]; float[] coeff = kernel.getKernelData(null); int m = width/2, n = height/2; float sum; int i, j, k, x, y; switch (borderStrategy) { case REFLECTED_INDEXING: for (y = 0; y < h; ++y) for (x = 0; x < w; ++x) { for (sum = 0.0f, i = 0, k = -n; k <= n; ++k) for (j = -m; j <= m; ++j, ++i) sum += coeff[i] * raster.getSample(refIndex(x-j, w), refIndex(y-k, h), 0); result[y*w+x] = sum; } break; case CIRCULAR_INDEXING: sum += coeff[i] * raster.getSample(circIndex(x-j, w), circIndex(y-k, h), 0); ///…rest of code here…//

Convolution Code public class ConvolutionOp extends NeighbourhoodOp { public BufferedImage filter(BufferedImage src, BufferedImage dest) { checkImage(src); if (dest == null) dest = createCompatibleDestImage(src, null); float[] rawData; if (calculation == SEPARABLE) { rawData = separableConvolve(src); } else { rawData = convolve(src); } DataBufferByte buffer = (DataBufferByte) dest.getRaster().getDataBuffer(); convertToBytes(rawData, buffer.getData()); return dest; protected void convertToBytes(float[] in, byte[] out) { if (rescaleStrategy == NO_RESCALING) { for (int i = 0; i < in.length; ++i) { int value = Math.round(in[i]); if (value < 0) out[i] = (byte) 0; else if (value > 255) out[i] = (byte) 255; else out[i] = (byte) value; // other cases go here //

Convolution

Convolution

Convolution

Frequency Frequency of a sound wave or audio signal – referring to the rate at which the signal changes with time. Frequency in an image - referring to changes occuring in space. Spatial frequency is a measure of how rapidly brightness or colours varies as we traverse an image. Images in which grey level varies slowly and smoothly are characterised solely by components with low spatial frequncy. Images containing sudden grey level transitions, fine detail or strong texture will also contain components with high spatial frequencies.

Filtering Convolution will have different effects depending upon the values of the Kernel. Convolution is an operation taken between images. (1) image data (2) kernel. Primary technique for spatial filtering. Convolution is a linear operation that can be undone. Filtering is a way of tuning image frequencies – much like a graphic equalizer. Linear filtering

Filtering

Filtering

Filtering Low pass filter: High pass filter: Allows low spatial frequencies to pass unchanged. Suppresses high frequencies. Smoothes or blurs the image. Tend to reduce noise but also obscures fine detail. High pass filter: Allows high spatial frequencies to pass unchanged. Suppresses low frequencies. Preserves sudden variation, such as those that occur at the boundaries of objects. But suppresses the more gradual variation.

Filtering

Filtering

Low-Pass Filtering

Low-Pass Filtering

Low-Pass Filtering

Consider the two kernels shown above. What do they do? Low-Pass Filtering Average kernel : Equally weighted sum of all pixels in a neighborhood Any kernel having all positive coefficients will act as a low-pass filter. 0.04 .111 Consider the two kernels shown above. What do they do? Their coefficient sum to 1. Convolution with them will not result in an overall brightening of the image.

Low-Pass Filtering = 1/9 * Any kernel having all positive coefficients will act as a low-pass filter. .111 1 = 1/9 * The center pixel becomes the average of all neighboring pixels. Also known as a mean filter.

Low-Pass Filter Pixel values from the neighbourhood are summed without being weighted. The sum is divided by the number of pixels in the neighbourhood. Convolution with these kernels is therefore equivalent to computing the mean grey level over the neighbourhood defined by the kernel. These kernels are sometimes described as mean filters.

Low-Pass Filtering

Low-Pass Filtering

Mean Filtering Mean filters are good at removing noise. Mean filters “blur” or “smooth” edges (by damping high frequency components and resisting fast changes in intensities) Kernels are typically normalized so that they sum to 1

Mean Filtering (100+100+100+100+200+205+100+195+200)/9

Gaussian Filter The Gaussian filter is a 2-D convolution operator that is used to `blur' images and remove detail and noise much like the mean filter. It is similar to the mean filter, but it uses a different kernel that represents the shape of a Gaussian (`bell-shaped') hump. This kernel has some special properties which are detailed below. The degree of smoothing is determined by the standard deviation of the Gaussian. Larger standard deviation Gaussians, of course, require larger convolution kernels in order to be accurately represented. The Gaussian outputs a `weighted average' of each pixel's neighborhood, with the average weighted more towards the value of the central pixels. This is in contrast to the mean filter's uniformly weighted average. A Gaussian provides gentler smoothing and preserves edges better than an identically sized mean filter.

3D plot of the Gaussian filter

Gaussian Filter The Gaussian kernel is separable and symmetric. To construct the kernel we must sample and quantize! (basically, we “image” the function) The kernel below is an example where sigma = 1. 1 4 7 16 28 49

Gaussian Filter 15 x 15 Larger kernels result -> more blur

Gaussian Filter

5x5 Mean Kernel 5x5 Gaussian Kernel

Noise Reduction Gaussian and mean filters are usually used to reduce “noise” in images. The above image has been corrupted by impulse noise.

Noise Reduction

Noise Reduction

Sharpening Filters (High-Pass Filtering) These filters highlight fine image detail or de-blur an image. Highpass filter: allows only high-frequency information through. Main feature is a positive center coefficient and negative perimeter values. The sum of the coefficients is zero, which means that areas of constant intensity are completely eliminated. -1 8

High-Pass Filtering The sum of the coefficients in this kernel is zero. This means that, when the kernel is over an area of constant or slowly varying grey level, the result of convolution is zero or some very small number. However, when grey level is varying rapidly within the neighbourhood, the result of convolution can be a large number (+ve or –ve). Need to choose an output image representation that support negative numbers. If we wish to display/print image, we must map the pixel values onto a 0-255 range. Usually map 0 onto the middle of the range. Thus, negative filter responses will show up as dark tones, whereas positive responses will be represented by light tones.

Sharpening Filters

Sharpening Filters

HighPass Examples

HighPass Example

HighPass Example

High Boost Filtering An image can be sharpened by high-boost filtering, an operation that emphasises the high spatial frequencies present in that image. In the spatial domain, this can be accomplished by convolution with a kernel of the form where c > 8. Larger values give more weight to a pixel's true value and less to the difference between it and its surroundings, thereby reducing the sharpening effect. As c get closer to 8, the degree of sharpening increases. If c=8, the kernel becomes the high pass filter. Keeps the “original” image while enhancing (boosting) the high-frequency components. -1 c

Example: High boost filtering

Thank you Q&A