Images  Camera models  Digital images  Colour images  Noise  Smoothing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014Slide 1

Camera models Components: A photosensitive image plane A housing A lenses Mathematical model needed The simple pinhole camera model Distortions Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 2

Camera models – Simple Pinhole Model Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 3

Camera models – Simple Pinhole Model 3-D point 2-D image point Scaling factor Combination of focal length and image coordinate system Location of the optical centre Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 4

Digital Images Theoretically images are continuous 2D functions of reflected scene brightness. (i, j) or (column, row) or (x, y) To process on a computer we need a discrete representation Sample Quantise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 5

Digital Images – Sampling Sample the continuous 2D function into discrete elements. Sensor 2D array Photosensitive elements Non photosensitive gaps Issues Elements have a fixed area Gaps Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 6

Digital Images – Sampling How many samples do we need ? Wasted space and computation time Enough for the objects of interest Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 7 Mat image, smaller_image; resize( image, smaller_image, Size( image1.cols/2, image.rows/2 ));

Digital Images – Quantisation Represent the individual image points as digital values. Typically 8 bits Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 8

Digital Images – Quantisation How many bits do we need ? Wasted space ? Losing the ability to distinguish objects Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 9 void ChangeQuantisationGrey( Mat &image, int num_bits ) { CV_Assert( (image.type() == CV_8UC1) && (num_bits >= 1) && (num_bits <= 8) ); uchar mask = 0xFF << (8-num_bits); for (int row=0; row < image.rows; row++) for (int col=0; col < image.cols; col++) image.at (row,col) = image.at (row,col) & mask; }

Colour Images Luminance only Simple representation Humans can understand Colour images ( luminance + chrominance ) Multiple channels (typically 3) Around 16.8 million colours More complex to process Facilitate certain operations Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 10

Colour Images – Processing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 11 void InvertColour( Mat& input_image, Mat& output_image ) { CV_Assert( input_image.type() == CV_8UC3 );` output_image = input_image.clone(); for (int row=0; row < input_image.rows; row++) for (int col=0; col < input_image.cols; col++) for (int channel=0; channel < input_image.channels(); channel++) output_image.at (row,col)[channel] = 255 – input_image.at (row,col)[channel]; }

Colour Images – Efficient processing Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 12 int image_rows = image.rows; int image_columns = image.cols; for (int row=0; row < image_rows; row++) { uchar* value = image.ptr (row); uchar* result_value = result_image.ptr (row); for (int column=0; column < image_columns; column++) { *result_value++ = *value++ ^ 0xFF; }

Colour Images – RGB Images Red-Green-Blue images Most common Channels correspond roughly to Red (700nm) Green (546nm) Blue (436nm) Colours combined in display Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 13

Colour Images – RGB Images Converting to Greyscale Y = 0.299R G B Camera photosensitive elements Separate Red, Green & Blue elements Sometimes sensitive to all visible wavelengths Bayer pattern: Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 14

Colour Images – RGB Images Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 15 Mat bgr_image, grey_image; cvtColor(bgr_image, grey_image, CV_BGR2GRAY); vector bgr_images(3); split(bgr_image, bgr_images); Mat& blue_image = bgr_images[0];

Colour Images – CMY Images Cyan-Magenta-Yellow images Secondary colours Subtractive colour scheme C = 255 – R M = 255 – G Y = 255 – B Often used in printers Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 16 CMY is not directly supported in OpenCV.

Colour Images – YUV Images Used for analogue television signals PAL, NTSC 4 Y to 1 U to 1 V Conversion from RGB Y = 0.299R G B U = * (B-Y) V = * (R-Y) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 17

Colour Images – HLS Images Hue-Luminance-Saturation images Separates Luminance & Chrominance Values we humans can relate to… Hue 0 o..360 o Luminance 0..1 Saturation 0..1 Watch out for circular Hue… Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 18

Colour Images – HLS Images Conversion from RGB Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 19 cvtColor(bgr_image, hls_image, CV_BGR2HLS); // Hue ranges from 0 to 179.

Colour Images – Other colour spaces HSV YCrCb CIE XYZ CIE L*u*v* CIE L*a*b* Bayer Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 20

Colour Images – Skin detection (S >= 0.2) AND (0.5 < L/S <3.0) AND (H = 330 o ) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 21 uchar H = hls_image.at (row,col)[0]; uchar L = hls_image.at (row,col)[1]; uchar S = hls_image.at (row,col)[2]; double LS_ratio = ((double) L) / ((double) S); bool skin_pixel = (S >= 50) && (LS_ratio > 0.5) && (LS_ratio = 165));

Colour Images – Red eye detection (L >= 0.25) AND (S >= 0.4) AND (0.5 < L/S <1.5) AND (H = 324 o ) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 22 uchar H = hls_image.at (row,col)[0]; uchar L = hls_image.at (row,col)[1]; uchar S = hls_image.at (row,col)[2]; double LS_ratio = ((double) L) / ((double) S); bool skin_pixel = (L >= 64) && (S >= 100) && (LS_ratio > 0.5) && (LS_ratio < 1.5) && ((H = 162));

Noise Affects most images Degrades the image Can cause problems with processing Causes? Measuring noise: Correcting noise… Types Gaussian Salt and Pepper Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 23

Noise - Models Additive noise f(i,j) = g(i,j) + v(i,j) Multiplicative noise f(i,j) = g(i,j) + g(i,j).v(i,j) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 24

Noise – Salt and Pepper Noise Impulse noise Noise is maximum or minimum values Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 25

Noise – Salt and Pepper Noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 26 int noise_points = (int) (((double) image_rows* image.cols*image.channels())*noise_percentage/100.0); for (int count = 0; count < noise_points; count++) { int row = rand() % image.rows; int column = rand() % image.cols; int channel = rand() % image.channels(); uchar* pixel = image.ptr (row) + (column*image.channels()) + channel; *pixel = (rand()%2 == 1) ? 255 : 0; }

Noise – Gaussian Noise Good approximation to real noise Distribution is Gaussian (mean & s.d.) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 27

Noise – Gaussian Noise Generation Select value for σ Determine distribution -(i) 2 p(i) = 1.e 2σ 2 i = -(G-1),..,-1,0,1,…,G-1 σ  2  P cum (-(G-1)) = p(-(G-1)) P cum (i) = P cum (i-1) + p(i) For every pixel (x,y) f*(x,y) = g(x,y) + argmin i (abs( rand() - P cum [i] )) Set f(x,y) = 0 if f*(x,y) < 0 f(x,y) = G-1 if f*(x,y) > G-1 f(x,y) = f*(x,y) otherwise Truncation attenuates Gaussian nature of noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 28

Noise – Gaussian Noise Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 29 Mat noise_image(image.size(), CV_16SC3); randn(noise_image, Scalar::all(average), Scalar::all(standard_deviation)); Mat temp_image; image.convertTo(temp_image,CV_16SC3); addWeighted(temp_image, 1.0, noise_image, 1.0, 0.0, temp_image); temp_image.convertTo(image,image.type());

Smoothing Removing or reducing noise… Linear smoothing transformations Image Averaging Local Averaging Gaussian Smoothing Non-linear transformations Rotating Mask Median Filter Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 30

Smoothing – Image Averaging (linear) Average of n images f(i,j) = (1/ n). Σ (g k (i,j) + v k (i,j)) k = 1..,n Additive noise model v k (i,j) St. Dev. = σ /  n Assumptions? Static Statistical independence Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 31 addWeighted(image1,0.5,image2,0.5,0.0,average_image);

Smoothing – Image Averaging Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 32

Smoothing – Convolution Filtering / Convolution Linear transformation: f( i,j ) = Σ Σ h( i-m, j-n ).g( m,n ) (m,n)  Ο Convolution mask Non-linear transformation: Some logical operation based on a local region Smoothing Suppression of image noise Blurring sharp edges Large degradations? Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 33

Smoothing – Averaging Filters (linear) Available images? Local neighbourhood f( i,j ) = Σ Σ h( i-m, j-n ).g( m,n ) (m,n)  Ο Different masks… Local Average Gaussian Acceptable results? Size of noise Blurring of edges Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 34 blur(image,smoothed_image,Size(3,3)); GaussianBlur(image,smoothed_image,Size(5,5),1.5);

Smoothing – Examples Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 35

Smoothing – Rotating Mask (non-linear) Define a number of masks/regions Mask size & shape Alternatives: Use the average of one of the masks But which mask?? The most homogeneous Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 36

Smoothing – Rotating Mask (non-linear) Algorithm: For each image point (i,j) Calculate dispersions Assign output point average of mask with minimum dispersion Iterative application Convergence Effects of mask size Effects Noise suppression Image sharpening Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 37

Smoothing – Rotating Mask (non-linear) Dispersion σ 2 Lower computational complexity… Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 38

Smoothing – Median Filter (non-linear) Use the median value… Not affected by noise Doesn’t blur edges much Can be applied iteratively Damages thin lines and sharp corners Change region shape Computational expensive Standard – O(r 2 log r) Huang – O(r) Perreault (2007) – O(1) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide Median = 23 Average = 47 medianBlur(image, smoothed_image, 5);

Smoothing – Median Filter (Huang O(r)) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 40 Input image X (m * n), kernel radius k, Output image Y for row = 1 to m Initialise histogram H: X(-k, row-k).. X(k, row+k) for column = 1 to n Y(column, row) = Median of H Remove leftmost column from H: X(column-k, row-k).. X(column-k, row+k) Add new column to right of H: X(column+k+1, row-k).. X(column+k+1, row+k) Can determine median efficiently by maintaining and updating: - Median value - Number of points less than the median

Smoothing – Median Filter (Perreault O(1)) Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 41 Input image X (m * n), kernel radius k, Output image Y for row = 1 to m Initialise column histograms h 1..n and histogram H for column = 1 to n Y(column, row) = Median of H Remove X(column+k+1, row-k-1) from h column+k+1 Add X(column+k+1, row+k) from h column+k+1 Remove leftmost column from H: h column-k Add new column to right of H: h column+k+1

Smoothing – Examples Images Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc Slide 42