Multiscale Analysis of Images Gilad Lerman Math 5467 (stealing slides from Gonzalez & Woods, and Efros)

The Multiscale Nature of Images

Recall: Gaussian pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction.

Subsampling with Gaussian pre-filtering G 1/4G 1/8Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction.

Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose Figure from David Forsyth

Gaussian pyramid construction filter mask Repeat Filter Subsample Until minimum resolution reached can specify desired number of levels (e.g., 3-level pyramid) The whole pyramid is only 4/3 the size of the original image!

What does blurring take away? (recall) original

What does blurring take away? (recall) smoothed (5x5 Gaussian)

High-Pass filter smoothed – original

Band-pass filtering Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction Gaussian Pyramid (low-pass images)

Laplacian Pyramid How can we reconstruct (collapse) this pyramid into the original image? Need this! Original image

Image resampling (interpolation) So far, we considered only power-of-two subsampling What about arbitrary scale reduction? How can we increase the size of the image? Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale d = 1 in this example

Image resampling So far, we considered only power-of-two subsampling What about arbitrary scale reduction? How can we increase the size of the image? Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale d = 1 in this example

Image resampling So what to do if we don’t know d = 1 in this example Answer: guess an approximation Can be done in a principled way: filtering Image reconstruction Convert to a continuous function Reconstruct by cross-correlation:

Resampling filters What does the 2D version of this hat function look like? Better filters give better resampled images Bicubic is common choice Why not use a Gaussian? What if we don’t want whole f, but just one sample? performs linear interpolation (tent function) performs bilinear interpolation

Bilinear interpolation Smapling at f(x,y):

What happens in the frequency domain? Laplacian Pyramid gives rise to different frequency bands Alternatively can obtain similar pyramids according to frequency bands…

Example of Frequency-bands “pyramid” Taken from Rajashekar and Simoncelli (

Applications (more efficient with wavelets) Coding High frequency coefficients need fewer bits, so one can collapse a compressed Laplacian pyramid Denoising/Restoration Setting “most” coefficients in Laplacian pyramid to zero Image Blending

Denoising demonstration (motivation) Taken from Rajashekar and Simoncelli ( Observation: spatial correlation in noise-free image (and not the noisy image) Conclusion: Noise is more evident in the high-frequency components, than the low- frequency components (smooth components)

Denoising demonstration (idea) Taken from Rajashekar and Simoncelli ( Partition into sub-bands of frequencies (top: high freq., bottom: low) y-given image (vectorized), - estimated denoised image (vectorized) The middle graph – intensity thresholding. No thresholding for smoothest component and keeping only higher intensities for higher components (the filter is learned empirically) In practice, can have blurring and not competitive with NLM, BM3d, dictionary learning

Application: Pyramid Blending Left pyramidRight pyramidblend

Image Blending

Feathering = Encoding transparency I(x,y) = (  R,  G,  B,  ) I blend =  left I left +  right I right I left I right  left  right

Affect of Window Size 0 1 left right 0 1

Affect of Window Size

Good Window Size 0 1 “Optimal” Window: smooth but not ghosted Burt and Adelson (83): Choose by pyramids…

Pyramid Blending

Pyramid Blending (Color)

Laplacian Pyramid: Blending General Approach: 1.Build Laplacian pyramids LA and LB from images A and B 2.Build a Gaussian pyramid GR from selected region R (black white corresponding images) 3.Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j) 4.Collapse the LS pyramid to get the final blended image

laplacian level 4 laplacian level 2 laplacian level 0 left pyramidright pyramidblended pyramid

Blending Regions

Simplification: Two-band Blending Brown & Lowe, 2003 Only use two bands: high freq. and low freq. Blends low freq. smoothly Blend high freq. with no smoothing: use binary mask Can be explored in a project…