CS 691B Computational Photography Instructor: Gianfranco Doretto Image Filtering

What is an image? We can think of an image as a function, f, from R2 to R: f( x, y ) gives the intensity at position ( x, y ) Realistically, we expect the image only to be defined over a rectangle, with a finite range: f: [a,b]x[c,d]  [0,1] A color image is just three functions pasted together. We can write this as a “vector-valued” function: As opposed to [0..255]

Images as functions Render with scanalyze????

Image Processing image warping: change domain of image g(x) = f(h(x)) image filtering: change range of image g(x) = h(f(x)) f x f x h image warping: change domain of image g(x) = f(h(x)) f x f x h

Image Processing image warping: change domain of image g(x) = f(h(x)) image filtering: change range of image g(x) = h(f(x)) h image warping: change domain of image g(x) = f(h(x)) h

Point Processing The simplest kind of range transformations are these independent of position x,y: g = t(f) This is called point processing. Important: every pixel for himself – spatial information completely lost!

Negative

Contrast Stretching

Image Histograms Cumulative Histograms s = T(r)

Histogram Equalization

Image filtering Image filtering: compute function of local neighborhood at each position Really important! Enhance images Denoise, resize, increase contrast, etc. Extract information from images Texture, edges, distinctive points, etc. Detect patterns Template matching

1D Smoothing examples Pixel offset coefficient original 8 impulse 2.4 Pixel offset coefficient original 8 impulse 2.4 0.3 filtered

1D Smoothing examples Pixel offset coefficient 8 8 edge 4 4 0.3 Pixel offset coefficient 8 8 edge 4 4 0.3 original filtered

Example: Box filter 1

Image filtering 1 90 90 ones, divide by 9

Image filtering 1 90 90 10 ones, divide by 9

Image filtering 1 90 90 10 20 ones, divide by 9

Image filtering 1 90 90 10 20 30 ones, divide by 9

Image filtering 1 90 10 20 30 ones, divide by 9

Image filtering 1 90 10 20 30 ? ones, divide by 9

Image filtering 1 90 10 20 30 50 ? ones, divide by 9

Image filtering 1 90 10 20 30 40 60 90 50 80

Box Filter What does it do? Replaces each pixel with an average of its neighborhood Achieve smoothing effect (remove sharp features) 1

Smoothing with box filter

Cross-correlation filtering Let’s write the box filter down as an equation. Assume the averaging window is (2k+1)x(2k+1): We can generalize this idea by allowing different weights for different neighboring pixels: This is called a cross-correlation operation and written: H is called the “filter,” “kernel,” or “mask.”

Convolution Cross-correlation: A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image: It is written: Suppose h is the mean kernel (or box filter). How does convolution differ from cross-correlation?

Convolution is nice! Notation: Convolution is a multiplication-like operation commutative associative distributes over addition scalars factor out identity: unit impulse e = […, 0, 0, 1, 0, 0, …] Conceptually no distinction between filter and signal Usefulness of associativity often apply several filters one after another: (((a * b1) * b2) * b3) this is equivalent to applying one filter: a * (b1 * b2 * b3)

Practice with linear filters 1 ? Original

Practice with linear filters 1 Original Filtered (no change)

Practice with linear filters 1 ? Original

Practice with linear filters 1 Original Shifted left By 1 pixel

Practice with linear filters - 2 1 ? (Note that filter sums to 1) Original

Practice with linear filters - 2 1 Original Sharpening filter Accentuates differences with local average

1D sharpening example 1.7 11.2 8 8 coefficient -0.25 -0.3 original Sharpened (differences are accentuated; constant areas are left untouched).

Sharpening

Other filters -1 1 -2 2 Sobel Vertical Edge (absolute value)

Other filters -1 -2 1 2 Horizontal Edge (absolute value) Q? Sobel 1 2 Questions at this point? Sobel Horizontal Edge (absolute value)

Important filter: Gaussian Weight contributions of neighboring pixels by nearness 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 5 x 5

Smoothing with Gaussian filter

Smoothing with box filter

Gaussian filters Remove “high-frequency” components from the image (low-pass filter) Images become more smooth Convolution with self is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2 Separable kernel Factors into product of two 1D Gaussians Linear vs. quadratic in mask size

Separability of the Gaussian filter The 2D Gaussian can be expressed as the product of two functions, one a function of x and the other a function of y In this case, the two functions are the (identical) 1D Gaussian

Separability example * 2D convolution (center location only) The filter factors into a product of 1D filters: * = Perform convolution along rows: Followed by convolution along the remaining column:

Separability Why is separability useful in practice?

Some practical matters

Practical matters How big should the filter be? Values at edges should be near zero Rule of thumb for Gaussian: set filter half-width to about 3 σ

Practical matters What is the size of the output? MATLAB: filter2(h, f, shape) shape = ‘full’: output size is sum of sizes of f and h shape = ‘same’: output size is same as f shape = ‘valid’: output size is difference of sizes of f and h full same valid h h h h f f f h h h h h h h h

Practical matters What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge

Practical matters methods (MATLAB): clip filter (black): imfilter(f, h, 0) wrap around: imfilter(f, h, ‘circular’) copy edge: imfilter(f, h, ‘replicate’) reflect across edge: imfilter(f, h, ‘symmetric’)

Questions?

Assignment 1: Hybrid Images A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006 Gaussian Filter! Gaussian unit impulse Laplacian of Gaussian Laplacian Filter! Assignment Instructions: http://www.csee.wvu.edu/~gidoretto/courses/2012-fall-cp/assignments/assignment1/index.html

Slide Credits This set of sides also contains contributions kindly made available by the following authors Alexei Efros Svetlana Lazebnik Frédo Durand Steve Seitz Derek Hoiem David Lowe Steve Marschner