1. Introduction to Image Processing Grass Sky Tree ? ? Spatial Techniques (2) Filtering

2. Spatial Techniques Neighbourhood Operations Linear System Theory –Unit Pulse Response Convolution & Correlation Image Filtering ‒ Linear − Non-linear

3. Neighbourhood Operations Operation at a pixel depends on the grey level values of a region of interest surrounding that pixel Neighbourhood is assumed to be centred at a pixel, thus normally it is a square with odd dimensions For point operations, the original input pixel value is no more needed once it has been processed. Neighbourhood operators, however, need the original pixel value for subsequent calculations of other neighbourhoods that also include that specific pixel applications include blurring, sharpening, noise reduction, edge detection, feature identification, measuring the similarity of two images

4. A signal is a description of how one parameter varies with another, e.g. voltage changing over time in an electronic circuit, or brightness varying with distance in an image. A system is any process that produces an output signal in response to an input signal. Signals & Systems

5. A system is classified as linear if it has the following mathematical properties: – homogeneity – additivity – shift invariance x[n - s]  y[n - s], for all s Linear Systems

6. ● Any signal can be decomposed into a group of additive components x[i] ● Passing these components through a linear system produces signals, y[i] ● The synthesis of these output signals produces the same output signal y[n] as when x[n] is passed through the system Foundations of DSP Superposition

7. Characterising Linear Systems Any linear digital image processing system can be characterised by one of two following equivalent ways: – unit pulse (impulse) response – frequency response Unit pulse (impulse) response – when the unit pulse is passed through a linear system, the single non-zero point will be changed into some other 2-D pattern – since the only thing that can happen to a point is that it spreads out, it is often called the point spread function (PSF) in image processing jargon – it is an effective way to model an imaging system, as any image can be viewed as a sum of scaled and shifted unit pulses

8. Convolution and Correlation Many image processing operations can be modeled as a linear system. Convolution and correlation are the two fundamental neighbourhood operations of image processing Convolution takes two signals and produces a third signal as output. Its main application is the filtering of images, e.g. sharpening the edges of objects, reducing random noise, correcting unequal illumination, blur and motion. Correlation measures similarity of two images which is useful in feature recognition and in registration, where we wish to place one image relative to another at a position of maximum similarity.

9. Convolution In essence, if the impulse response, or point-spread function (PSF) h(x, y), of a system is known, then the output g(x, y) with an input image f(x, y) is given by where kernel h is of width m and height n; where m = 2a + 1 and n = 2b + 1 are odd and known as the support of h. For m=n=3, with index values from -1 to +1, the entire convolution computation is g(x, y) = h(-1, -1) f (x+1, y+1) + h(0, -1) f (x, y+1) + h(1, -1) f (x-1, y+1) + h(-1, 0) f (x+1, y) + h(0, 0) f (x, y) + h(1, 0) f (x-1, y) + h(-1, 1) f (x+1, y-1) + h(0, 1) f (x, y-1) + h(1, 1) f (x-1, y-1) While the kernel coefficients are taken in sequence from top-left corner to the bottom right corner, the pixels associated with these kernel coefficients are in precisely the opposite direction. If we were to rotate the kernel by 180 o, then both sequence would run in the same direction, i.e. each kernel coefficient would then pair with the pixel directly beneath it.

10. Using the same notation of convolution, if the template is represented as h(x, y), the output g(x, y) with an input image f (x, y) is given by where template h is of width m and height n and m = 2a + 1, n = 2b + 1. Mathematically, convolution is the same process, except that h is rotated by 180 degree. When h(x, y) is symmetric, convolution is equal to correlation. Correlation

11. Correlation for Comparing Images Sum of square differences used for similarity measure: First two terms fixed  last term measures mismatch – the cross-correlation: In practice, normalise by image magnitude when shifting template to search for matches

12. Template Matching The filter is called a template or a mask The brighter the value in the output, the better the match Input image Template Output

13. Removing Noise (Denoising) Spatial Domain Some sorts of processing are sensitive to noise We want to remove the effects of noise before going further However, we may lose information in doing so Most noise removal processes are called filtering They are applied to each point in an image through convolution They use information in a small local window of pixels

14. Image Filters Source imageTarget image Source pixel Window Local processing on the window Target pixel Replace each pixel by a linear combination of its neighbours The prescription for the linear combination is called the “kernel” (or “mask”, “filter”)

15. Neigbouring pixels are highly correlated, denoise by averaging nearby pixels – Box filter (mean filter)  Each pixel is set to the mean (average) over a local window – Weighted average filter This blurs the image, making it smoother and removes fine details Box Weighted Average Smoothing Spatial Filters

16. 4.3 Step 1: Move the window to the first location where we want to compute the average value and then select only pixels inside the window. 4 4 67 6 1 9 2 2 2 7 5 2 26 4 4 5 212 1 3 3 4 2 9 5 7 7 Step 2: Compute the average value Sub image p Original image 41 9 2 2 3 2 9 7 Output image Step 3: Place the result at the pixel in the output image Step 4: Move the window to the next location and go to Step 2 Basics of Spatial Filtering

17. Gaussian Filter Gaussian filters are based on the same function used for Gaussian noise To generate a filter it needs to be 2D − If we take the mean to be 0 then we get

18. Gaussian Filter The Gaussian filter has a 2D bell shape − It is symmetric around the centre − The centre value gets the highest weight − Values further from the centre get lower weights 2-D Gaussian distribution with mean (0,0) and standard deviation 1

19. Discrete Gaussian Filters The Gaussian − Extends infinitely in all directions, but we want to process just a local window − Has a volume underneath it of 1, which we want to maintain − is a function (continuous) We can approximate the Gaussian with a discrete filter − We restrict ourselves to a square window and sample the Gaussian function − We normalise the result so that the filter entries add to 1

20. Example Suppose we want to use a 5x5 window to apply a Gaussian filter with σ 2 = 1 − The centre of the window has x = y = 0 − We sample the Gaussian at each point − We then normalise it 01 2-2 0 1 2 -2 0.000.010.020.010.00 0.010.060.100.060.01 0.020.100.160.100.02 0.010.060.100.060.01 0.000.010.020.010.00 × 1 0.96

21. Averaging vs. Gaussian Filters Advantages of Gaussian filtering − rotationally symmetric (isotropic) − filter weights decrease monotonically from central peak, giving most weight to central pixels − separable

22. Separable Filters The Gaussian filter is separable − This means you can do a 2D Gaussian as 2 1D Gaussians − First you filter with a ‘horizontal’ Gaussian − Then with a ‘vertical’ Gaussian

23. Separable Filters This gives a more efficient way to do a Gaussian filter − Generate a 1D filter with the required variance − Apply this ‘horizontally’ − Then apply it to the result ‘vertically’ 0.060.240.400.240.06 0.24 0.40 0.24 0.06

24. Separable Filters The separated filter is more efficient − Given an N×N image and a n×n filter we need to do O(N 2 n 2 ) operations − Applying two n×1 filters to a N×N image takes O(2N 2 n) operations Example − A 600×400 image and a 5×5 filter − Applying it directly takes around 6,000,000 operations (600×400x5x5) − Using a separable filter takes around 2,400,000 operations (600×400x5+ 600×400x5) - less than half as many

25. Gaussian Filters How big should the filter window be? − With Gaussian filters this depends on the variance (σ 2 ) − Under a Gaussian curve we know that 98% of the area lies within 5σ of the mean − If we take the filter width to be at least 5σ we get more than 98% of the values we want 5σ5σ 98.8%

26. Spatial Non-Linear Filtering As in the case of linear filtering, we consider a subimage S xy centred at a certain pixel (x,y) of the image. The new value of the image at the pixel (x,y) is a non-linear function of the values of S xy, such as: Original image Processed image Centre Min Max Median

27. Example Image with salt noise Image denoised with a 3x3 min filter

28. Example Image with pepper noise Image denoised with a 3x3 max filter

29. Nonlinear Median Filter An alternative to the mean filter is the median filter − Statistically the median is the middle value in a set − Each pixel is set to the median value in a local window 123124125 1291279 126123131 1231241251291279126123131 Find the values in a local window 9123 124125126127129131 Sort them 9123 124125126127129131 Pick the middle one

30. Example Image with salt and pepper noise Image denoised with a 7x7 median filter

31. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that only smooths regions but preserves edges? - yes, bilateral filter

32. Illustration a 1D Image 1D image = line of pixels Better visualized as a plot pixel intensity pixel position

33. Definition spacerange normalization space Gaussian blur Bilateral filter [Aurich 95, Smith 97, Tomasi 98] only spatial distance, intensity ignored spatial and range distances weights sum to 1 space range p p q q

34. Gaussian Smoothing * * * input output Slides taken from Sylvain Paris, Siggraph 2007

35. Bilateral Filtering * * * input output Kernel shape depends on image content Avoids averaging across edges Slides taken from Sylvain Paris, Siggraph 2007

36. The Problem with Edges We have seen two ways to filter images − Mean filter - take the average value over a local window − Median filter - take the middle value over a local window Filters operate on a local window − We need to know the value of all the neighbours of each pixel − Pixels on the edges or corners of the image don’t have all their neighbours ??? ? ?

37. Dealing with Edges There are a number of ways we can deal with edges − Ignore them - don’t use filters near edges − Modify our filter to account for smaller neighbourhoods − Extend the image to predict the unknowns Ignoring the edges − This is simple − It means that we lose the border pixels each time we apply a filter − Might be OK for one or two small filters, but your image can shrink rapidly

38. Modifying the Filter Many filters can be modified to give a reasonable value near the edge − We may need different filters for top, bottom, left, and right edges, and for each corner Example: mean filter 1/6 1/4

39. Extending the Image Padding with a specific value Circular extension of the image

40. Extending the Image We need to predict what lies beyond the edges of the image − We can use the value of the nearest known pixel − We can apply a more detailed model to the image Nearest neighbour

41. Linear Extrapolation A model can be fitted to an image − We use known pixels to establish a model of the pixel value as a function of x and y − We use this model to find missing values − Models are usually fitted locally A simple example is a linear model through a pair of pixels ABX X = B + (B-A)

42. Linear Extrapolation 130133 136138 130 + (130-133) 136 + (136-138) 127 134 136 + (136-133) 139 136 + (136-130) 142 138 + (138-133) 143

43. Code for filters With a filter the processing at (x,y) depends on a neighbourhood, typically a square with ‘radius’ r (3x3 has radius 1, 5x5 radius 2…) int x, y, dx, dy, r; for (x = 0; x < image.getWidth(); x++) { for (y = 0; y < image.getHeight(); y++) { for (dx = -r; dx <= r; dx++) { for (dy = -r; dy <= r; dy++) { // Do something with (x+dx, y+dy) }}}} For each pixel, p For each pixel in the neighbourhood of p

44. function gaussianfilter() % Parameters of the Gaussian filter: n1=10;sigma1=3;n2=10;sigma2=3;theta =0; % The amplitude of the noise: noise=10; [w,map]=imread('lena.gif', 'gif'); x=ind2gray(w,map); x_rand=noise*randn(size(x)); [m,n]=size(x); for i=1:m for j=1:n y(i,j)=x(i,j)+x_rand(i,j); j=j+1; end i=i+1; end filter1=d2gauss(n1,sigma1,n2,sigma2,the ta); f1=conv2(x,filter1,'same'); f2=conv2(y,filter1,'same'); figure(1); subplot(2,2,1);imagesc(x);title('lena'); subplot(2,2,2);imagesc(y);title('noisy'); subplot(2,2,3);imagesc(f1);title('Gaussian filter lena - smooth'); subplot(2,2,4);imagesc(f2);title('Gaussian filter noisy - noise removal'); colormap(gray); return; % Function "d2gauss.m": % This function returns a 2D Gaussian filter with size n1*n2; theta is % the angle that the filter rotated counter clockwise; and sigma1 and sigma2 % are the standard deviation of the Gaussian functions. function h = d2gauss(n1,std1,n2,std2,theta) r=[cos(theta) -sin(theta); sin(theta) cos(theta)]; for i = 1 : n2 for j = 1 : n1 u = r * [j-(n1+1)/2 i-(n2+1)/2]'; h(i,j)=gauss(u(1),std1)*gauss(u(2),std2); end h = h / sqrt(sum(sum(h.*h))); % Function "gauss.m": function y = gauss(x,std) y = exp(-x^2/(2*std^2)) / (std*sqrt(2*pi)); http://en.wikipedia.org/wiki/Gaussian_function

45. Acknowlegements Slides are modified based on the original slide set from Dr Li Bai, The University of Nottingham, Jubilee Campus plus the following sources: Digital Image Processing, by Gonzalez and Woods Digital Image Processing: a practical introduction using Java, by Nick Efford http://gear.kku.ac.th/~nawapak/178353/Chapter03.ppt http://www.comp.dit.ie/bmacnamee/materials/dip/lectures /ImageProcessing5-SpatialFiltering1.ppt The Scientist and Engineer’s Guide to Digital Signal Processing, by Steven W. Smith