Linear Filtering CS 678 Spring 2018

Outline Linear filtering Box filter vs. Gaussian filter Median filter Gaussian and Laplacian pyramids Sharpening images Some slides from Lazebnik

Motivation: Noise reduction Given a camera and a still scene, how can you reduce noise? Answer: take lots of images, average them Take lots of images and average them! What’s the next best thing? Source: S. Seitz

Moving average Let’s replace each pixel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for a 3x3 moving average? 1 “box filter” Source: D. Lowe

Defining convolution f Let f be the image and g be the kernel. The output of convolving f with g is denoted f * g. f Convention: kernel is “flipped” MATLAB: conv2 vs. filter2 (also imfilter) Source: F. Durand

Key properties Linearity: filter(f1 + f2 ) = filter(f1) + filter(f2) Shift invariance: same behavior regardless of pixel location: filter(shift(f)) = shift(filter(f)) Theoretical result: any linear shift-invariant operator can be represented as a convolution

Properties in more detail Commutative: a * b = b * a Conceptually no difference between filter and signal Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b1) * b2) * b3) This is equivalent to applying one filter: a * (b1 * b2 * b3) Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = […, 0, 0, 1, 0, 0, …], a * e = a

Annoying details What is the size of the output? MATLAB: filter2(g, f, shape) shape = ‘full’: output size is sum of sizes of f and g shape = ‘same’: output size is same as f shape = ‘valid’: output size is difference of sizes of f and g full same valid g g g g f f f g g g g g g g g

Annoying details What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner

Annoying details What about near the edge? the filter window falls off the edge of the image need to extrapolate methods (MATLAB): clip filter (black): imfilter(f, g, 0) wrap around: imfilter(f, g, ‘circular’) copy edge: imfilter(f, g, ‘replicate’) reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner

Practice with linear filters 1 ? Original Source: D. Lowe

Practice with linear filters 1 Original Filtered (no change) Source: D. Lowe

Practice with linear filters 1 ? Original Source: D. Lowe

Practice with linear filters 1 Original Shifted left By 1 pixel Source: D. Lowe

Practice with linear filters ? 1 Original Source: D. Lowe

Practice with linear filters 1 Original Blur (with a box filter) Source: D. Lowe

Practice with linear filters - 2 1 ? (Note that filter sums to 1) Original Source: D. Lowe

Practice with linear filters - 2 1 Original Sharpening filter Accentuates differences with local average Source: D. Lowe

Sharpening Source: D. Lowe

Smoothing with box filter revisited Smoothing with an average actually doesn’t compare at all well with a defocused lens Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square I always walk through the argument on the left rather carefully; it gives some insight into the significance of impulse responses or point spread functions. Source: D. Forsyth

Smoothing with box filter revisited Smoothing with an average actually doesn’t compare at all well with a defocused lens Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square Better idea: to eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the center, like so: I always walk through the argument on the left rather carefully; it gives some insight into the significance of impulse responses or point spread functions. “fuzzy blob”

Gaussian Kernel 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 5 x 5,  = 1 Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case) Source: C. Rasmussen

Choosing kernel width Gaussian filters have infinite support, but discrete filters use finite kernels Source: K. Grauman

Choosing kernel width Rule of thumb: set filter half-width to about 3 σ

Example: Smoothing with a Gaussian

Mean vs. Gaussian filtering

Gaussian filters Remove “high-frequency” components from the image (low-pass filter) Convolution with self is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2 Separable kernel Factors into product of two 1D Gaussians Linear vs. quadratic in mask size Source: K. Grauman

Separability of the Gaussian filter Source: D. Lowe

Separability example * 2D convolution (center location only) The filter factors into a product of 1D filters: * = Perform convolution along rows: Followed by convolution along the remaining column: Source: K. Grauman

Separability Why is separability useful in practice?

Noise Salt and pepper noise: contains random occurrences of black and white pixels Impulse noise: contains random occurrences of white pixels Gaussian noise: variations in intensity drawn from a Gaussian normal distribution Salt and pepper and impulse noise can be due to transmission errors (e.g., from deep space probe), dead CCD pixels, specks on lens Source: S. Seitz

Gaussian noise Mathematical model: sum of many independent factors Good for small standard deviations Assumption: independent, zero-mean noise Source: M. Hebert

Reducing Gaussian noise Smoothing with larger standard deviations suppresses noise, but also blurs the image

Reducing salt-and-pepper noise 3x3 5x5 7x7 What’s wrong with the results?

Alternative idea: Median filtering A median filter operates over a window by selecting the median intensity in the window Better at salt’n’pepper noise Not convolution: try a region with 1’s and a 2, and then 1’s and a 3 Is median filtering linear? Source: K. Grauman

Median filter What advantage does median filtering have over Gaussian filtering? Robustness to outliers Source: K. Grauman

Salt-and-pepper noise Median filter Salt-and-pepper noise Median filtered MATLAB: medfilt2(image, [h w]) Source: M. Hebert

Median vs. Gaussian filtering 3x3 5x5 7x7 Gaussian Median

Image Pyramids Multi-resolution of images Gaussian pyramid Laplacian pyramid

Scaled representations Big bars (resp. spots, hands, etc.) and little bars are both interesting Stripes and hairs, say Inefficient to detect big bars with big filters And there is superfluous detail in the filter kernel Alternative: Apply filters of fixed size to images of different sizes Typically, a collection of images whose edge length changes by a factor of 2 (or root 2) This is a pyramid (or Gaussian pyramid) by visual analogy

Gaussian Pyramids Very useful for representing images Image Pyramid is built by using multiple copies of image at different scales. Each level in the pyramid is ¼ of the size of previous level The highest level is of the lowest resolution The lowest level is of the highest resolution

Gaussian Pyramids

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Aliasing Can’t shrink an image by taking every second pixel If we do, characteristic errors appear Common phenomenon Wagon wheels rolling the wrong way in movies Checkerboards misrepresented in ray tracing Striped shirts look funny on colour television

Resample the checkerboard by taking one sample at each circle Resample the checkerboard by taking one sample at each circle. In the case of the top left board, new representation is reasonable. Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

The Gaussian pyramid Smooth with gaussians, because Synthesis a gaussian*gaussian=another gaussian Synthesis smooth and sample gaussians are low pass filters

Reduce (1D)

Reduce (1D)

Convolution Kernel Symmetric Sum of mask should be 1

Convolution Kernel All nodes at a given level must contribute the same total weight to the nodes at the next higher level c c b b a

Convolution Kernel a=0.4 GAUSSIAN, a=0.5 TRIANGULAR

What about 2D? Separability of Gaussian Requires n2 k2 multiplications for n by n image and k by k kernel. Requires 2kn2 multiplications for n by n image and k by k kernel.

Algorithm Apply 1D mask to alternate pixels along each row of image. Apply 1D mask to alternate pixels along each column of resultant image from previous step. Down sample

The Laplacian Pyramid Similar to edge detected images Most pixels are zero Can be used in image compression

Constructing Laplacian Pyramid Compute Gaussian pyramid Compute Laplacian pyramid as follows:

Reconstructing Image

The Laplacian Pyramid

This is a Gaussian pyramid

And this a Laplacian pyramid And this a Laplacian pyramid. Notice that the lowest layer is the same in each case, and that if I upsample the lowest layer and add to the next, I get the Gauss layer. Notice also that each layer shows detail at a particular scale --- these are, basically, bandpass filtered versions of the image.

The Laplacian Pyramid Analysis Synthesis preserve difference between upsampled Gaussian pyramid level and Gaussian pyramid level band pass filter - each level represents spatial frequencies (largely) unrepresented at other levels Synthesis reconstruct Gaussian pyramid, take the finest layer

Sharpening revisited What does blurring take away? Let’s add it back: original smoothed (5x5) detail = – Let’s add it back: original detail + α sharpened =

unit impulse (identity) Unsharp mask filter Gaussian unit impulse Laplacian of Gaussian image blurred image unit impulse (identity) f + a(f - f * g) = (1+a)f-af*g = f*((1+a)e-g)

Application: Hybrid Images A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006