1. Computational Photography CSE 590 Tamara Berg Filtering & Pyramids

2. Image filtering Image filtering: compute function of local neighborhood at each position Really important! – Enhance images Denoise, resize, increase contrast, etc. – Extract information from images Texture, edges, distinctive points, etc. – Detect patterns Template matching

3. 111 111 111 Slide credit: David Lowe (UBC) Example: box filter

4. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Credit: S. Seitz Image filtering 111 111 111

5. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

6. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

7. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

8. 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

9. 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz ?

10. 0102030 50 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz ?

11. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000 Image filtering 111 111 111 Credit: S. Seitz

12. What does it do? Replaces each pixel with an average of its neighborhood Achieve smoothing effect (remove sharp features) 111 111 111 Slide credit: David Lowe (UBC) Box Filter

13. Smoothing with box filter

14. Practice with linear filters 000 010 000 Original ? Source: D. Lowe

15. Practice with linear filters 000 010 000 Original Filtered (no change) Source: D. Lowe

16. Practice with linear filters 000 100 000 Original ? Source: D. Lowe

17. Practice with linear filters 000 100 000 Original Shifted left By 1 pixel Source: D. Lowe

18. Practice with linear filters Original 111 111 111 000 020 000 - ? Source: D. Lowe

19. Practice with linear filters Original 111 111 111 000 020 000 - Sharpening filter - Accentuates differences with local average Source: D. Lowe

20. Sharpening Source: D. Lowe

21. Characterizing edges An edge is a place of rapid change in the image intensity function image intensity function (along horizontal scanline) first derivative edges correspond to extrema of derivative source: Svetlana Lazebnik

22. Edge filters Approximations of derivative filters: Source: K. Grauman Convolve filter with image to get edge map

23. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to vertical edges

24. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to horizontal edges

25. Edges: example source: Svetlana Lazebnik

26. Gaussian Kernel Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case) 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5,  = 1 Source: C. Rasmussen

27. Example: Smoothing with a Gaussian source: Svetlana Lazebnik

28. Gaussian filters Remove “high-frequency” components from the image (low-pass filter) – Images become more smooth Convolution with self is another Gaussian – So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Separable kernel – Factors into product of two 1D Gaussians Source: K. Grauman

29. Separability of the Gaussian filter Source: D. Lowe The filter factors into a product of 1D filters:

30. Practical matters What is the size of the output? MATLAB: imfilter(g, f, shape) – shape = ‘full’: output size is sum of sizes of f and g – shape = ‘same’: output size is same as f – shape = ‘valid’: output size is difference of sizes of f and g f gg gg f gg g g f gg gg full samevalid Source: S. Lazebnik

31. Practical matters What about near the edge? – the filter window falls off the edge of the image – need to extrapolate – methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner

32. Practical matters – methods (MATLAB): clip filter (black): imfilter(f, g, 0) wrap around:imfilter(f, g, ‘circular’) copy edge: imfilter(f, g, ‘replicate’) reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner Q?

33. Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches? –Correlation –Zero-mean correlation –Sum Square Difference –Normalized Cross Correlation Side by Derek Hoiem

34. Matching with filters Goal: find in image Method 0: filter the image with eye patch Input Filtered Image What went wrong? f = image g = filter Side by Derek Hoiem

35. Matching with filters Goal: find in image Method 1: filter the image with zero-mean eye Input Filtered Image (scaled) Thresholded Image True detections False detections mean of f

36. Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) Thresholded Image True detections

37. Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) What’s the potential downside of SSD? Side by Derek Hoiem

38. Matching with filters Goal: find in image Method 3: Normalized cross-correlation mean image patch mean template Side by Derek Hoiem

39. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections

40. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections

41. Q: What is the best method to use? A: Depends Zero-mean filter: fastest but not a great matcher SSD: next fastest, sensitive to overall intensity Normalized cross-correlation: slowest, invariant to local average intensity and contrast Side by Derek Hoiem

42. Image half-sizing This image is too big to fit on the screen. How can we reduce it? How to generate a half- sized version?

43. Image sub-sampling Throw away every other row and column to create a 1/2 size image - called image sub-sampling 1/4 1/8 Slide by Steve Seitz

44. Image sub-sampling 1/4 (2x zoom) 1/8 (4x zoom) Aliasing! What do we do? 1/2 Slide by Steve Seitz

45. Anti-aliasing Get rid of all frequencies that are greater than half the new sampling frequency – Will lose information – But it’s better than aliasing – Apply a smoothing filter Slide by Derek Hoiem

46. Algorithm for downsampling by factor of 2 1.Start with image(h, w) 2.Apply low-pass filter im_blur = imfilter(image, fspecial(‘gaussian’, 7, 1)) 3.Sample every other pixel im_small = im_blur(1:2:end, 1:2:end); Slide by Derek Hoiem

47. Gaussian (lowpass) pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Slide by Steve Seitz

48. Subsampling with Gaussian pre-filtering G 1/4G 1/8Gaussian 1/2 Slide by Steve Seitz

49. Compare with... 1/4 (2x zoom) 1/8 (4x zoom) 1/2 Slide by Steve Seitz

50. Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform Slide by Steve Seitz

51. A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose Figure from David Forsyth

52. Coarse-to-fine Image Registration 1.Compute Gaussian pyramid 2.Align with coarse pyramid 3.Successively align with finer pyramids – Search smaller range Why is this faster?

53. What are pyramids good for? Improve Search –Search over translations Like HW1 Classic coarse-to-fine strategy –Search over scale Template matching E.g. find a face at different scales Pre-computation –Need to access image at different blur levels –Useful for texture mapping at different resolutions (called mip-mapping) –Useful for compositing/blending – coming soon!