1. Multiscale Analysis of Images Gilad Lerman Math 5467 (stealing slides from Gonzalez & Woods, and Efros)

2. The Multiscale Nature of Images

3. Recall: Gaussian pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction.

4. Subsampling with Gaussian pre-filtering G 1/4G 1/8Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction.

5. Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform

6. A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose Figure from David Forsyth

7. Gaussian pyramid construction filter mask Repeat Filter Subsample Until minimum resolution reached can specify desired number of levels (e.g., 3-level pyramid) The whole pyramid is only 4/3 the size of the original image!

8. What does blurring take away? (recall) original

9. What does blurring take away? (recall) smoothed (5x5 Gaussian)

10. High-Pass filter smoothed – original

11. Band-pass filtering Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction Gaussian Pyramid (low-pass images)

12. Laplacian Pyramid How can we reconstruct (collapse) this pyramid into the original image? Need this! Original image

15. Image resampling (interpolation) So far, we considered only power-of-two subsampling What about arbitrary scale reduction? How can we increase the size of the image? Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale 12345 d = 1 in this example

16. Image resampling So far, we considered only power-of-two subsampling What about arbitrary scale reduction? How can we increase the size of the image? Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale 12345 d = 1 in this example

17. Image resampling So what to do if we don’t know 123452.5 1 d = 1 in this example Answer: guess an approximation Can be done in a principled way: filtering Image reconstruction Convert to a continuous function Reconstruct by cross-correlation:

18. Resampling filters What does the 2D version of this hat function look like? Better filters give better resampled images Bicubic is common choice Why not use a Gaussian? What if we don’t want whole f, but just one sample? performs linear interpolation (tent function) performs bilinear interpolation

19. Bilinear interpolation Smapling at f(x,y):

20. What happens in the frequency domain? Laplacian Pyramid gives rise to different frequency bands Alternatively can obtain similar pyramids according to frequency bands…

21. Example of Frequency-bands “pyramid” Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf)

22. Applications (more efficient with wavelets) Coding High frequency coefficients need fewer bits, so one can collapse a compressed Laplacian pyramid Denoising/Restoration Setting “most” coefficients in Laplacian pyramid to zero Image Blending

23. Denoising demonstration (motivation) Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf) Observation: spatial correlation in noise-free image (and not the noisy image) Conclusion: Noise is more evident in the high-frequency components, than the low- frequency components (smooth components)

24. Denoising demonstration (idea) Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf) Partition into sub-bands of frequencies (top: high freq., bottom: low) y-given image (vectorized), - estimated denoised image (vectorized) The middle graph – intensity thresholding. No thresholding for smoothest component and keeping only higher intensities for higher components (the filter is learned empirically) In practice, can have blurring and not competitive with NLM, BM3d, dictionary learning

25. Application: Pyramid Blending 0 1 0 1 0 1 Left pyramidRight pyramidblend

26. Image Blending

27. Feathering 0 1 0 1 + = Encoding transparency I(x,y) = (  R,  G,  B,  ) I blend =  left I left +  right I right I left I right  left  right

28. Affect of Window Size 0 1 left right 0 1

29. Affect of Window Size 0 1 0 1

30. Good Window Size 0 1 “Optimal” Window: smooth but not ghosted Burt and Adelson (83): Choose by pyramids…

31. Pyramid Blending

32. Pyramid Blending (Color)

33. Laplacian Pyramid: Blending General Approach: 1.Build Laplacian pyramids LA and LB from images A and B 2.Build a Gaussian pyramid GR from selected region R (black white corresponding images) 3.Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j) 4.Collapse the LS pyramid to get the final blended image

34. laplacian level 4 laplacian level 2 laplacian level 0 left pyramidright pyramidblended pyramid

35. Blending Regions

36. Simplification: Two-band Blending Brown & Lowe, 2003 Only use two bands: high freq. and low freq. Blends low freq. smoothly Blend high freq. with no smoothing: use binary mask Can be explored in a project…