1. Images  Camera models  Digital images  Colour images  Noise  Smoothing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014Slide 1

2. Camera models Components: A photosensitive image plane A housing A lenses Mathematical model needed The simple pinhole camera model Distortions Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 2

3. Camera models – Simple Pinhole Model Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 3

4. Camera models – Simple Pinhole Model 3-D point 2-D image point Scaling factor Combination of focal length and image coordinate system Location of the optical centre Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 4

5. Digital Images Theoretically images are continuous 2D functions of reflected scene brightness. (i, j) or (column, row) or (x, y) To process on a computer we need a discrete representation Sample Quantise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 5

6. Digital Images – Sampling Sample the continuous 2D function into discrete elements. Sensor 2D array Photosensitive elements Non photosensitive gaps Issues Elements have a fixed area Gaps Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 6

7. Digital Images – Sampling How many samples do we need ? Wasted space and computation time Enough for the objects of interest Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 7 Mat image, smaller_image; resize( image, smaller_image, Size( image1.cols/2, image.rows/2 ));

8. Digital Images – Quantisation Represent the individual image points as digital values. Typically 8 bits Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 8

9. Digital Images – Quantisation How many bits do we need ? Wasted space ? Losing the ability to distinguish objects Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 9 void ChangeQuantisationGrey( Mat &image, int num_bits ) { CV_Assert( (image.type() == CV_8UC1) && (num_bits >= 1) && (num_bits <= 8) ); uchar mask = 0xFF << (8-num_bits); for (int row=0; row < image.rows; row++) for (int col=0; col < image.cols; col++) image.at (row,col) = image.at (row,col) & mask; }

10. Colour Images Luminance only Simple representation Humans can understand Colour images ( luminance + chrominance ) Multiple channels (typically 3) Around 16.8 million colours More complex to process Facilitate certain operations Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 10

11. Colour Images – Processing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 11 void InvertColour( Mat& input_image, Mat& output_image ) { CV_Assert( input_image.type() == CV_8UC3 );` output_image = input_image.clone(); for (int row=0; row < input_image.rows; row++) for (int col=0; col < input_image.cols; col++) for (int channel=0; channel < input_image.channels(); channel++) output_image.at (row,col)[channel] = 255 – input_image.at (row,col)[channel]; }

12. Colour Images – Efficient processing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 12 int image_rows = image.rows; int image_columns = image.cols; for (int row=0; row < image_rows; row++) { uchar* value = image.ptr (row); uchar* result_value = result_image.ptr (row); for (int column=0; column < image_columns; column++) { *result_value++ = *value++ ^ 0xFF; }

13. Colour Images – RGB Images Red-Green-Blue images Most common Channels correspond roughly to Red (700nm) Green (546nm) Blue (436nm) Colours combined in display Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 13

14. Colour Images – RGB Images Converting to Greyscale Y = 0.299R + 0.587G + 0.114B Camera photosensitive elements Separate Red, Green & Blue elements Sometimes sensitive to all visible wavelengths Bayer pattern: Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 14

15. Colour Images – RGB Images Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 15 Mat bgr_image, grey_image; cvtColor(bgr_image, grey_image, CV_BGR2GRAY); vector bgr_images(3); split(bgr_image, bgr_images); Mat& blue_image = bgr_images[0];

16. Colour Images – CMY Images Cyan-Magenta-Yellow images Secondary colours Subtractive colour scheme C = 255 – R M = 255 – G Y = 255 – B Often used in printers Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 16 CMY is not directly supported in OpenCV.

17. Colour Images – YUV Images Used for analogue television signals PAL, NTSC 4 Y to 1 U to 1 V Conversion from RGB Y = 0.299R + 0.587G + 0.114B U = 0.492 * (B-Y) V = 0.877 * (R-Y) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 17

18. Colour Images – HLS Images Hue-Luminance-Saturation images Separates Luminance & Chrominance Values we humans can relate to… Hue 0 o..360 o Luminance 0..1 Saturation 0..1 Watch out for circular Hue… Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 18

19. Colour Images – HLS Images Conversion from RGB Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 19 cvtColor(bgr_image, hls_image, CV_BGR2HLS); // Hue ranges from 0 to 179.

20. Colour Images – Other colour spaces HSV YCrCb CIE XYZ CIE L*u*v* CIE L*a*b* Bayer Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 20

21. Colour Images – Skin detection (S >= 0.2) AND (0.5 < L/S <3.0) AND (H = 330 o ) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 21 uchar H = hls_image.at (row,col)[0]; uchar L = hls_image.at (row,col)[1]; uchar S = hls_image.at (row,col)[2]; double LS_ratio = ((double) L) / ((double) S); bool skin_pixel = (S >= 50) && (LS_ratio > 0.5) && (LS_ratio = 165));

22. Colour Images – Red eye detection (L >= 0.25) AND (S >= 0.4) AND (0.5 < L/S <1.5) AND (H = 324 o ) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 22 uchar H = hls_image.at (row,col)[0]; uchar L = hls_image.at (row,col)[1]; uchar S = hls_image.at (row,col)[2]; double LS_ratio = ((double) L) / ((double) S); bool skin_pixel = (L >= 64) && (S >= 100) && (LS_ratio > 0.5) && (LS_ratio < 1.5) && ((H = 162));

23. Noise Affects most images Degrades the image Can cause problems with processing Causes? Measuring noise: Correcting noise… Types Gaussian Salt and Pepper Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 23

24. Noise - Models Additive noise f(i,j) = g(i,j) + v(i,j) Multiplicative noise f(i,j) = g(i,j) + g(i,j).v(i,j) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 24

25. Noise – Salt and Pepper Noise Impulse noise Noise is maximum or minimum values Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 25

26. Noise – Salt and Pepper Noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 26 int noise_points = (int) (((double) image_rows* image.cols*image.channels())*noise_percentage/100.0); for (int count = 0; count < noise_points; count++) { int row = rand() % image.rows; int column = rand() % image.cols; int channel = rand() % image.channels(); uchar* pixel = image.ptr (row) + (column*image.channels()) + channel; *pixel = (rand()%2 == 1) ? 255 : 0; }

27. Noise – Gaussian Noise Good approximation to real noise Distribution is Gaussian (mean & s.d.) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 27

28. Noise – Gaussian Noise Generation Select value for σ Determine distribution -(i) 2 p(i) = 1.e 2σ 2 i = -(G-1),..,-1,0,1,…,G-1 σ  2  P cum (-(G-1)) = p(-(G-1)) P cum (i) = P cum (i-1) + p(i) For every pixel (x,y) f*(x,y) = g(x,y) + argmin i (abs( rand() - P cum [i] )) Set f(x,y) = 0 if f*(x,y) < 0 f(x,y) = G-1 if f*(x,y) > G-1 f(x,y) = f*(x,y) otherwise Truncation attenuates Gaussian nature of noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 28

29. Noise – Gaussian Noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 29 Mat noise_image(image.size(), CV_16SC3); randn(noise_image, Scalar::all(average), Scalar::all(standard_deviation)); Mat temp_image; image.convertTo(temp_image,CV_16SC3); addWeighted(temp_image, 1.0, noise_image, 1.0, 0.0, temp_image); temp_image.convertTo(image,image.type());

30. Smoothing Removing or reducing noise… Linear smoothing transformations Image Averaging Local Averaging Gaussian Smoothing Non-linear transformations Rotating Mask Median Filter Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 30

31. Smoothing – Image Averaging (linear) Average of n images f(i,j) = (1/ n). Σ (g k (i,j) + v k (i,j)) k = 1..,n Additive noise model v k (i,j) St. Dev. = σ /  n Assumptions? Static Statistical independence Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 31 addWeighted(image1,0.5,image2,0.5,0.0,average_image);

32. Smoothing – Image Averaging Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 32

33. Smoothing – Convolution Filtering / Convolution Linear transformation: f( i,j ) = Σ Σ h( i-m, j-n ).g( m,n ) (m,n)  Ο Convolution mask Non-linear transformation: Some logical operation based on a local region Smoothing Suppression of image noise Blurring sharp edges Large degradations? Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 33

34. Smoothing – Averaging Filters (linear) Available images? Local neighbourhood f( i,j ) = Σ Σ h( i-m, j-n ).g( m,n ) (m,n)  Ο Different masks… Local Average Gaussian Acceptable results? Size of noise Blurring of edges Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 34 blur(image,smoothed_image,Size(3,3)); GaussianBlur(image,smoothed_image,Size(5,5),1.5);

35. Smoothing – Examples Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 35

36. Smoothing – Rotating Mask (non-linear) Define a number of masks/regions Mask size & shape Alternatives: Use the average of one of the masks But which mask?? The most homogeneous Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 36

37. Smoothing – Rotating Mask (non-linear) Algorithm: For each image point (i,j) Calculate dispersions Assign output point average of mask with minimum dispersion Iterative application Convergence Effects of mask size Effects Noise suppression Image sharpening Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 37

38. Smoothing – Rotating Mask (non-linear) Dispersion σ 2 Lower computational complexity… Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 38

39. Smoothing – Median Filter (non-linear) Use the median value… Not affected by noise Doesn’t blur edges much Can be applied iteratively Damages thin lines and sharp corners Change region shape Computational expensive Standard – O(r 2 log r) Huang – O(r) Perreault (2007) – O(1) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 39 11 18 20 21 23 25 25 30 250 Median = 23 Average = 47 medianBlur(image, smoothed_image, 5);

40. Smoothing – Median Filter (Huang O(r)) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 40 Input image X (m * n), kernel radius k, Output image Y for row = 1 to m Initialise histogram H: X(-k, row-k).. X(k, row+k) for column = 1 to n Y(column, row) = Median of H Remove leftmost column from H: X(column-k, row-k).. X(column-k, row+k) Add new column to right of H: X(column+k+1, row-k).. X(column+k+1, row+k) Can determine median efficiently by maintaining and updating: - Median value - Number of points less than the median

41. Smoothing – Median Filter (Perreault O(1)) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 41 Input image X (m * n), kernel radius k, Output image Y for row = 1 to m Initialise column histograms h 1..n and histogram H for column = 1 to n Y(column, row) = Median of H Remove X(column+k+1, row-k-1) from h column+k+1 Add X(column+k+1, row+k) from h column+k+1 Remove leftmost column from H: h column-k Add new column to right of H: h column+k+1

42. Smoothing – Examples Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 42