- photometric aspects of image formation gray level images linear/nonlinear filtering edge detection corner detection CS223, Jana Kosecka

Image Brightness values I(x,y) CS223, Jana Kosecka

Analog intensity function Temporal/spatial sampled function Image model Mathematical tools Analog intensity function Temporal/spatial sampled function Quantization of the gray levels Point sets Random fields List of image features, regions Analysis Linear algebra Numerical methods Set theory, morphology Stochastic methods Geometry, AI, logic CS223, Jana Kosecka

Basic Photometry Radiometric model of image formation CS223, Jana Kosecka

Basic ingredients Radiance – amount of energy emitted along certain direction Iradiance – amount of energy received along certain direction BRDF – bidirectional reflectance distribution – portion of the energy coming from direction reflected to direction Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction Image intensity for a Lambertian surface CS223, Jana Kosecka

Images Images contain noise – sources sensor quality, light fluctuations, quantization effects CS223, Jana Kosecka

Image Noise Models Additive noise: most commonly used Multiplicative noise: Impulsive noise (salt and pepper): Noise models: gaussian, uniform Noise Amount: SNR = s/ n CS223, Jana Kosecka

Image filtering How can we reduce the noise in the image Image acquisition noise due to light fluctuations and sensor noise can be reduced by acquiring a sequence of images and averaging them Computation of simple features First stage of visual processing CS223, Jana Kosecka

Image Processing 1D signal and its sampled version f = { f(1), f(2), f(3), …, f(n)} f = {0, 1, 2, 3, 4, 5, 5, 6, 10 } CS223, Jana Kosecka

Discrete time system Maps 1 discrete time signal to another f[x] g[x] h Special class of systems – linear , time-invariant systems Superposition principle Shift (time) invariant – shift in input causes shift in output CS223, Jana Kosecka

unit impluse – if x = 0 it’s 1 and zero everywhere else Convolution sum: unit impluse – if x = 0 it’s 1 and zero everywhere else Every discrete time signal can be written as a sum of scaled and shifted impulses The output the linear system is related to the input and the transfer function via convolution h f g filter f g Convolution sum: Notation: CS223, Jana Kosecka

Averaging filter Original image Smoothed image CS223, Jana Kosecka

Averaging filter center pixel weighted more and 0 everywhere else Box filter Ex. cont. Averaging filter center pixel weighted more CS223, Jana Kosecka

Convolution in 2D X 10 10 11 9 1 2 99 f g h 1 1 1 1 1 1 1/9 1 1 1 1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) = 1/9.( 90) = 10 CS223, Jana Kosecka

Example: X X X X X X 10 11 10 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 10 2 1 X X 11 10 9 9 11 10 X X 9 10 11 9 99 11 F X X X X X X 10 9 9 11 10 10 1 1 1 1 1 1 1/9 1 1 1 1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) = 1/9.( 34) = 3.7778 CS223, Jana Kosecka

Example: X X X X X X 10 11 10 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 1 10 7 4 1 X X 9 10 11 1 1 O X X I 10 9 10 2 1 X X 11 10 9 9 11 10 1 F X 20 X 9 10 11 9 99 11 X X X X X X 10 9 9 11 10 10 1/9 1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) = 1/9.( 180) = 20 CS223, Jana Kosecka

How big should the mask be? The bigger the mask, more neighbors contribute bigger noise spread. more blurring. more expensive to compute Limitations of averaging Signal frequencies shared with noise are lost Impulsive noise is diffused but not removed CS223, Jana Kosecka

Frequency Domain Interpretation: Sinc(u) = sin u F i Low-pass filter The secondary lobes of the sinc let noise into the filtered image. CS223, Jana Kosecka

Gaussian Filter Better option for blurring The coefficients are samples of a 1D Gaussian. Gives more weight at the central pixel and less weights to the neighbors. The further away the neighbors, the smaller the weight. Gaussian filter Samples from the continuous Gaussian Gaussian filter is the only one that has the same shape in the space and frequency domains. There are no secondary lobes – i.e. a truly low-pass filter CS223, Jana Kosecka

How big should the mask be? The std. dev of the Gaussian  determines the amount of smoothing. The samples should adequately represent a Gaussian For a 98.76% of the area, we need m = 5 5.(1/)  2    0.796, m 5 5-tap filter g[x] = [0.136, 0.6065, 1.00, 0.606, 0.136] CS223, Jana Kosecka

Image Smoothing Convolution with a 2D Gaussian filter Gaussian filter is separable, convolution can be accomplished as two 1-D convolutions CS223, Jana Kosecka

Non-linear Filtering Replace each pixel with the MEDIAN value of all the pixels in the neighborhood Non-linear Does not spread the noise Can remove spike noise Expensive to run CS223, Jana Kosecka

Example: 10 11 9 1 2 99 I X 10 O median sort 10,11,10,9,10,11,10,9,10 99 I X 10 O median sort 10,11,10,9,10,11,10,9,10 9,9,10,10,10,10,10,11,11 CS223, Jana Kosecka

Example: 10 11 10 1 X X X X X X 9 10 11 1 1 10 10 1 1 X X O 10 9 10 I 2 1 X X 11 10 9 9 11 X X 10 9 10 11 9 99 11 X X 10 9 9 11 10 10 X X X X X X median sort 11,10,0,10,11,1,9,10,0 0,0,1,9,10,10,10,11,11 CS223, Jana Kosecka

Example: 10 11 10 1 X X X X X X 9 10 11 1 1 10 X X O 10 9 10 2 1 X X I 1 X X X X X X 9 10 11 1 1 10 X X O 10 9 10 2 1 X X I 11 10 9 9 11 X 9 X 10 9 10 11 9 99 11 X 10 X 10 9 9 11 10 10 X X X X X X median sort 10,9,11,9,99,11,11,10,10 9,9,10,10,10,11,11,11,99 CS223, Jana Kosecka

Image Features Local, meaningful, detectable parts of the image. We will look at edges and corners CS223, Jana Kosecka

Image Features – Edges, Corners Look for detectable, meaningful parts of the image Edges are detected at places where the image values exhibit sharp variation Gray value column column Edge Edge CS223, Jana Kosecka

Edge detection (1D) F(x) Edge= sharp variation x F ’(x) Large first derivative x CS223, Jana Kosecka

Digital Approximation of 1st derivatives f(x) X-1 X+1 -1 1 Convolve with: CS223, Jana Kosecka

Edge Detection (2D) Vertical Edges: -1 1 Convolve with: Convolve with: 1 Convolve with: Convolve with: -1 1 Horizontal Edges: CS223, Jana Kosecka

Noise cleaning and Edge Detection I(x,y) E(x,y) Noise Filter Edge Detection We need to also deal with noise -> Combine Linear Filters CS223, Jana Kosecka

Noise Smoothing & Edge Detection -1 1 Convolve with: Noise Smoothing Vertical Edge Detection This mask is called the (vertical) Prewitt Edge Detector Outer product of box filter [1 1 1]T and [-1 0 1] CS223, Jana Kosecka

Noise Smoothing & Edge Detection -1 1 Convolve with: Horizontal Edge Detection Noise Smoothing This mask is called the (horizontal) Prewitt Edge Detector CS223, Jana Kosecka

Sobel Edge Detector -1 1 -2 2 Convolve with: Gives more weight 1 -2 2 Convolve with: Gives more weight to the center pixels and -1 -2 1 2 CS223, Jana Kosecka

Example 50 50 100 150 50 100 150 -1 1 -1 1 CS223, Jana Kosecka

Image Derivatives We know better alternative to smoothing Smooth using Gaussian filter g(x) is a 1-D Gaussian filter, g(x,y) – 2-D Gaussian filter Taking a derivative – linear operation (take the derivative of the filter) CS223, Jana Kosecka

Gaussian and its derivative CS223, Jana Kosecka

Vertical edges First derivative CS223, Jana Kosecka

Horizontal edges Gradient Magnitude Image Gradient CS223, Jana Kosecka

Gradient Orientation CS223, Jana Kosecka

Orientation histogram CS223, Jana Kosecka

Corner detection A point on a line is hard to match. Intuition: Right at corner, gradient is ill defined. Near corner, gradient has two different values. CS223, Jana Kosecka

Formula for Finding Corners We look at matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region, the hypothetical corner Matrix is symmetric CS223, Jana Kosecka

First, consider case where: This means all gradients in neighborhood are: (k,0) or (0, c) or (0, 0) (or off-diagonals cancel). What is region like if: ? and ? CS223, Jana Kosecka

General Case: In general case from linear algebra, it follows that because C is symmetric: With R being a rotation matrix. So every case can be intrepreted like one on last slide. CS223, Jana Kosecka

Corner Detection Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Compute eigenvalues l1 and l2. If they are both big, we have a corner Or if smalest eigenvalue  of C is bigger than  - mark pixel as candidate feature point CS223, Jana Kosecka

Point Feature Extraction Harris Corner Detector Alternatively feature quality function (Harris Corner Detector) CS223, Jana Kosecka

Harris Corner Detector - Example CS223, Jana Kosecka

Edge Detection Compute image derivatives original image gradient magnitude Compute image derivatives if gradient magnitude >  and the value is a local max. along gradient direction – pixel is an edge candidate how to detect one pixel thin edges ? CS223, Jana Kosecka

Canny Edge Detector The magnitude image Es has the magnitudes of the smoothed gradient. Sigma determines the amount of smoothing. Es has large values at edges: Find local maxima Th CS223, Jana Kosecka

Nonmaximum supression The inputs are Es & Eo Magnitude and orientation Consider 4 directions D={ 0,45,90,135} wrt x For each pixel (i,j) do: Find the direction dD s.t. d Eo(i,j) (normal to the edge) If {Es(i,j) is smaller than at least one of its neigh. along d} IN(i,j)=0 Otherwise, IN(i,j)= Es(i,j) The output is the thinned edge image IN x CS223, Jana Kosecka

Thresholding Edges are found by thresholding the output of NONMAX_SUPRESSION If the threshold is too high: Very few (none) edges High MISDETECTIONS, many gaps If the threshold is too low: Too many (all pixels) edges High FALSE POSITIVES, many extra edges CS223, Jana Kosecka

Hysteresis Thresholding Es(i,j)> H Es(i,j)<H Es(i,j)>L Es(i,j)<L Strong edges reinforce adjacent weak edges CS223, Jana Kosecka

Finding lines in an image Option 1: Search for the line at every possible position/orientation What is the cost of this operation? Option 2: Use a voting scheme: Hough transform CS223, Jana Kosecka

Other edge detectors - second-order derivative filters (1D) Peaks of the first-derivative of the input signal, correspond to “zero-crossings” of the second-derivative of the input signal. F(x) F ’(x) x F’’(x) CS223, Jana Kosecka

Edge Detection (2D) x y 1D 2D I(x) I(x,y) dI(x) dx > Th F(x) x 1D 2D I(x) I(x,y) dI(x) dx > Th |I(x,y)| =(Ix 2(x,y) + Iy2(x,y))1/2 > Th tan  = Ix(x,y)/ Iy(x,y) d2I(x) dx2 = 0 2I(x,y) =Ix x (x,y) + Iyy (x,y)=0 Laplacian CS223, Jana Kosecka

Notes about the Laplacian: 2I(x,y) is a SCALAR  Can be found using a SINGLE mask  Orientation information is lost 2I(x,y) is the sum of SECOND-order derivatives But taking derivatives increases noise Very noise sensitive! It is always combined with a smoothing operation: Filter Laplacian of Gaussian LOG filter Smooth Laplacian I(x,y) O(x,y) CS223, Jana Kosecka

1D Gaussian 2D “Mexican Hat” CS223, Jana Kosecka

Derivative of Gaussian Filters Measure the image gradient and its direction at different scales (use a pyramid). CS223, Jana Kosecka

The Laplacian Pyramid Building a Laplacian pyramid: Create a Gaussian pyramid Take the difference between one Gaussian pyramid level and the next (before subsampling) Properties Also known as the difference-of-Gaussian function, which is a close approximation to the Laplacian It is a band pass filter - each level represents a different band of spatial frequencies Reconstructing the original image: Reconstruct the Gaussian pyramid starting at top layer CS223, Jana Kosecka

This is a Gaussian pyramid CS223, Jana Kosecka

Laplacian Pyramid (note top image is from Gaussian) And this a Laplacian pyramid. Notice that the lowest layer is the same in each case, and that if I upsample the lowest layer and add to the next, I get the Gauss layer. Notice also that each layer shows detail at a particular scale --- these are, basically, bandpass filtered versions of the image. Laplacian Pyramid (note top image is from Gaussian) CS223, Jana Kosecka

Add more oriented filters (Malik & Perona, 1990) Spots and bars at a fine scale for the butterfly; images show squared response for corresponding filter CS223, Jana Kosecka

Alternative: Gabor filters Gabor filters: Product of a Gaussian with sine or cosine Top row shows anti-symmetric (or odd) filters, bottom row the symmetric (or even) filters. No obvious advantage to any one type of oriented filters. Gabor filters are just another way to generate spots and bars. CS223, Jana Kosecka

An edge is not a line... How can we detect lines ? CS223, Jana Kosecka