1 Ekstrahering og repræsentation af features MM2: Convolution and filtering of images

2 Agenda Point processing –Image computing Neighborhood processing –Convolution Applications

3 Fundamental Steps in Computer Vision Knowledge base Problem domain Image acquisition Preprocessing Segmentation Representation and description Recognition and Interpretation Result Point 1: 22,33 Point 2: 24, 39 ….. Actor sitting Today: Point processing –Filtering Convolution –Method Filtering –Application

4 Point processing Only one pixel in the input has an effect on the output For example: –Changing the brightness, thresholding, histogram stretching InputOutput

5 Image Computing Arithmetic: An image calculator: + - / etc. Logic operations (OR, AND, etc. ) Image computation is done pixel-wise For example: ……… …a4a3 …a2a1 ……… …b4b3 …b2b1 a4 +b4 a3 +b3 a2 +b2 a1 +b1 + = ……… …a4a3 …a2a1 a4 +10 a3 +10 a2 +10 a = 10

6 Image Computing The size of the output image Sometimes a program requires the images to have the same sizes +=

7 Image Computing: Caution!! Undesired effects may happen, when computing with images: Overflow / Underflow Example: Two Gray value images I 1, I 2 with pixel ranges: [0,255]. Their Pixel sum may be larger than 255!! For example: =326! Subtraction results may be smaller than 0!! For example: =-78!!

8 Solution to the Overflow/Underflow Problem Use an intermediate image: –Pixel values are float values (32 bits/ 4 bytes) –Can store almost any number 1.Write computation results into intermediate image 2.Rescale intermediate image to values 0…255 and write results into normal Gray value image.

9 Gray Value Scaling and Shifting ……0…1… min … max … 01…… ………… Intermediate float image Gray value image

10 Different operations Monadic (one image one constant) –AddIm + 17 –SubIm - 17 –MulIm * 17 –DivIm / 17 –AndIm AND 17 –OrIm OR 17 –XorIm XOR 17 –MinMin(Im,17) –MaxMax(Im,17) –xxx… Dynadic (two images) –AddA+B –SubA-B –MulA*B –DivA/B –AndA AND B –OrA OR B –XorA XOR B –MinMin(A,B) –MaxMax(A,B) –xxx…

11 In ImageJ Involving two images: Process => Image Calculator Involving an image and a number: Process => Math += + = 10

12 Measure noise in your setup Subtract two images of the same scene at different time instances => Noise in scene Help you to judge the nature of the noise -= Before After Noise - =

13 Image averaging: remove noise Assume noise has zero mean The noise tends to cancel out if we average a number of images Example: real value: 100. Value in different images: 103, 109, 98, 87, 95, 112. Average = (I 1 + I 2 + I 3 + I 4 + I 5 + I 6 ) / 6 = 104 The more images the better Requirement: nothing is moving (changing) in the scene! Original Noise added

14 Segmentation of the object Background subtraction Image containing object - Background image = Object Advanced background subtraction: used avg. bg. image - = BackgroundImage w. Obj. Object - =

15 Combine two images Used when adding text etc. to an image –Mixing two images by averaging –But what if we want to put more weight on one image? –Alpha blending (or weighted averaging): – –If then this operation is standard averaging –Many programs (CV+CG) support alpha blending Image representation: RGBA (4 bytes) per pixel

16 Neighborhood processing

17 Neighborhood processing Several pixels in the input has an effect on the output Input Output

18 Convolution

19 Convolution (1D) Filter Response Filter Input Signal/Image-row Output Signal/Image-row Filter coefficients (mask, kernel, template, window)

20 Normalise filter response … … 255 Max values in image Filter coefficientsA B C Max filter response = A B C 255 = 255 (A+B+C) If max filter response = 255 (one byte) then New filter response = filter response / (A+B+C)

21 Convolution (1D)

22 Convolution (1D) This process is called Convolution!!

23 Math of convolution g(x): output, h: filter, * means convolution, f(x): input, n = |_ width of filter / 2 _| |_ _|: rounds down, for example: |_ 1.7_| = 1 For example: Filter (h): width = 3 => n=1 121 h(-1)=1h(0)=2h(1)=1

24 Correlation (1D) Filter Response Filter Input Signal/Image-row Output Signal/Image-row Filter coefficients (mask, kernel, template, window)

25 Correlation versus Convolution Correlation Convolution In image processing we use CORRELATION but (nearly) always call it CONVOLUTION!!!!! Note: When the filter is symmetric: correlation = convolution!

26 Convolution/correlation on images The filter is now 2D Kernel (mask), kernel coefficients Size: 3x3, 5x5, 7x7, … Input Output

27 Math. of 2D Convolution/Correlation Convolution Correlation

28 Problems at the borders Input Output Why is the output image smaller than the input? –We are lacking information The bigger the kernel the bigger the problem Does it matter? Yes, if we are going to combine the images afterwards

29 Problems at the borders Input Output Solutions –Add a value: 0, 255, neighbor (input/output) Change histogram, very different value, new pattern, etc. –Truncate kernel: 3x3 => for example 2x3: Complex and not well-defined

30 Problems at the borders Input Solutions: –Complex and perhaps wrong Input Reflected (mirrored) Circular indexing

31 Applications of convolution/correlation Many many operations defined by the programmer…. and some standard operations –Object detection –Smoothing filters –Edge detection –Morphology (next time) –….

32 Simple Object Detection Finding a specific object in the image 1D example: An object is given (known) as an image, e.g., Task: Find this object in an image: Input Output For images this is called corelation or template matching!

33 Template Matching The filter is called a template or a mask The brighter the value in the output, the better the match Input image Template Output

34 Smoothing filters Also know as: Smoothing kernel, Mean filter, Low pass filter The simplest filter: –Spatial low pass filter Another mask: –Gaussian filter:

35 Applications of smoothing Blurring to remove identity or other details Degree of blurring = kernel size De-blurring / sharpening (show: camera, 3 x smooth, sharpen)

36 Applications of smoothing Preprocessing: enhance objects Smooth + Thresholding Remove background: g(x,y) = f(x,y) – f(x,y) (show: boat, 25mean)

37 Applications of smoothing Remove noise

38 Rank Filters Aka: order-statistics filters Not based on convolution but still neighborhood processing Principle: –Define a mask, e.g., 3x3 –Sort all pixel-values within the mask into ascending order –Select a pixel-value according to the filter type: Median, min., max., range, …

39 Median Filter For an image, mask symmetric: 3x3, 5x5, etc Input Output Sorted: 0,0,1,1,1,2,2,2,4 1

40 Median Filter –Good for cleaning salt-and-pepper noise Better than the mean filter as blurring is minimized and edges stay sharp

41 Edge Detection Powerfull application of convolution!

42 Edge detection What are edges? Why are they interesting? How do we find them? –Prewitt –Sobel –Laplacian –Canny

43 What are edges? Edge detection Local intensity change Strong edge = the steep areas in a 3D plot (show: blobs)

44 Why are they interesting? Edges can (many times) represent the information in the image (the objects) A higher level of abstraction (less data to process!) Edges are features: –“Independent” from illumination. As opposed to e.g., color information Object recognition and detection (many times) use edge features –This is true for computer vision as well as for biological vision systems!

45 Edges detectors Three steps: –Noise reduction –Edge enhancement –Edge localisation

46 Edges detectors Three steps: –Noise reduction E.g., median filter E.g., mean filter Dilemma: –Large filter => remove noise –Large filter => remove edges  –Small filter => keep edges –Small filter => keep noise  –Edge enhancement –Edge localisation

47 Edges detectors Three steps: –Noise reduction –Edge enhancement Calculate candidates for the edges –Edge localisation Decide which edge candidates to keep

48 Edges detectors We’ll look at four methods (but others exist!) With respect to complexity (simplest first): –Prewitt –Sobel –Laplacian –Canny

49 Edge Detectors: Prewitt and Sobel

50 Edge Detectors: Prewitt and Sobel Simple to implement. Fast Based on the grey-level gradient –A measure of the steepness of the “image-landscape” –Calculated for each pixel => Gradient image: g(x,y) –Use a 16-bit or 32-bit image to represent the gradients! The gradient is the first-order derivative: Approximated in the x- and y-direction by: y xx+1x-1 y+1 y-1

51 Edge Detectors: Prewitt and Sobel Gradient vector: –Magnitude: [g x, g y ]

52 Edge Detectors: Prewitt and Sobel Noise reduction –Mean of three gradients Edge enhancement –Implementation of: –Convolution with Prewitt kernels or Sobel kernels Edge localisation –If the magnitude: g(x,y) > TH => edge found: e(x,y)=255 –Else: no edge: e(x,y)=0 (Show: blob, 32-bit, convolution, abs) Prewitt kernels Sobel kernels

53 Prewitt and Sobel Conclusion –Pros: Simple to understand Simple to implement Fast –Cons: Produce wide edges –Used allot! (most common: Sobel)

54 Edge Detector: Laplacian

55 Edge Detector: Laplacian Where exactly is the edge (width)? Second-order derivative: –The variation of the variation of the gray-level value! –How fast does the gradient change? –Find the zero-crossing Center of edge 0 Zero-crossing

56 Edge Detector: Laplacian The second-order derivative is very sensitive to noise!

57 Laplacian of Gaussian Solution: –Combined with a 2D Gaussian smoothing filter –The filter is then called Laplacian of Gaussian (LoG)

58 Laplacian of Gaussian 2D Laplacian of Gaussian Second-order derivative of the 2D Gaussian Can also be approximated by a discrete mask which depends on the size of the Gaussian and the size of the kernel, for example: (show: blob, 32-bit, gauss, convolution)

59 Examples of LoG Gauss filter width: 5x5 pixels 9x9 pixels 12x12 pixels Input LoGEdges

60 Laplacian of Gaussian Conclusion –Pros: Edges well-defined due to zero-crossings –Cons: How do we find the zero-crossings? –I.e., transitions from black to white and vice versa Laplacian is too sensitive to noise due to second-order derivative Simple to implement except for the search for zero-crossings! –Not used so often

61 Canny Edge Detector

62 Canny Edge Detector Noise reduction –2D Gaussian used for smoothing Edge enhancement –Magnitude of gradient vector: Edge localisation

63 Canny Edge Detector Edge localisation –Wide edges: Edges give rise to ridges in the gradient image (show: blobs, edge, invert, region, 3D) Thin edges using the principle of non- maximal suppression

64 Thresholding Dilemma Define a threshold in the magnitude-image –Assume high magnitudes originate from objects Dilemma: –Too high threshold: Remove noise Remove small edges  –Too low threshold: Keep noise  Keep small edges How do we threshold the magnitude image so that noise is suppressed and the object-edges are not???

65 Hysteresis Thresholding Concept: think of object edges as a group of edges Use two thresholds: Th min and Th max –All magnitude edge-pixel below Th min are sat to zero Choose Th min so low that no object edges are eliminated –Find all grouped edges (non-zero edge-pixels) –Ignore all groups which are too small (in number) –Ignore all groups which do not contain at least one edge-pixel with a magnitude above Th max Choose Th max so high that (most) noise edges are eliminated Effect: –Isolated (noise) edge-pixels are ignored –Both strong and weak edge-pixels are kept

66 Examples Gauss 5x5, Th max = 255, Th min = 1 Gauss 5x5, Th max = 255, Th min = 220 Gauss 5x5, Th max = 128, Th min = 1 Gauss 9x9, Th max = 128, Th min = 1

67 Examples Comparison on noisy image CannySobel

68 Examples Gauss 5x5, Th max = 200, Th min = 1 Gauss 7x7, Th max = 200, Th min = 1

69 Canny edge detector Conclusion: Pros: –One pixel wide edges –Edges are grouped together (often good for segmentation) –Robust to noise! Cons: –Complicated to understand and implement –Slow Used allot!

70 What to remember Point processing: pixel-wise operations –An image calculator…(to image, image and scalar): AND, OR, +, -, … Neighborhood processing –Convolution and correlation Kernel, mask, filter, template –Template matching (object recognition) –Filtering Mean filter: blur, preprocessing Median filter (good at removing noise) Remove “background” Sharpening

71 What to remember Edge = Rapid intensity change Edge information is one of the most important in CV and Human Vision Three steps in edge detection: –Noise reduction –Edge enhancement –Edge localisation Three types were presented: –Based on first-order derivative Prewitt and Sobel –Based on second-order derivative Laplacian of Gaussian –Based on groups of edges Canny