1. CSCE 643 Computer Vision: Template Matching, Image Pyramids and Denoising Jinxiang Chai

2. Today’s class Template matching Gaussian Pyramids Laplacian Pyramids Image denoising

3. Template matching Goal: find in image

4. Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches?

5. Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches? – Correlation – Zero-mean correlation – Sum Square Difference – Normalized Cross Correlation Slide: Hoiem

6. Matching with filters Goal: find in image Method 0: filter the image with eye patch Input Filtered Image What went wrong? f = image g = filter Slide: Hoiem

7. Problem with Correlation

9. Solution

10. Slide: Hoiem Matching with filters Goal: find in image Method 1: filter the image with zero-mean eye Input Filtered Image (scaled) Thresholded Image True detections False detections mean of f

11. Slide: Hoiem Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) Thresholded Image True detections

12. Relationship SSD and Correlation

13. Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) What’s the potential downside of SSD? Slide: Hoiem

14. Matching with filters Goal: find in image Method 3: Normalized cross-correlation - subtracting the mean - dividing by the standard deviationstandard deviation

15. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Matlab: normxcorr2(template, im) mean image patch mean template

16. Slide: Hoiem Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections

17. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections Slide: Hoiem

18. Q: What is the best method to use? A: Depends SSD: faster, sensitive to overall intensity Normalized cross-correlation: slower, invariant to local average intensity and contrast

19. Q: What if we want to find larger or smaller eyes? A: Image Pyramid

20. Gaussian Pyramid Low-Pass Filtered Image Image Gaussian Filter Sample Low-Res Image

21. Gaussian Pyramid filter mask Repeat –Filter –Subsample Until minimum resolution reached –can specify desired number of levels (e.g., 3-level pyramid )

22. Gaussian pyramid

23. Template Matching with Image Pyramids Input: Image, Template 1.Match template at current scale 2.Downsample image 3.Repeat 1-2 until image is very small 4.Take responses above some threshold, perhaps with non-maxima suppression

24. Laplacian pyramid

25. Laplacian Filter The filtering kernel for Laplacian is Finite difference allows us to approximate the Laplacian using the following filtering kernel.

26. Laplacian Pyramid Laplacian Pyramid (subband images) - Created from Gaussian pyramid by subtraction

27. The Laplacian of Gaussian Kernel is the Laplacian operator:

28. The Laplacian of Gaussian Filter Kernel

29. Laplacian Pyramid What happens in frequency domain?

30. Laplacian Pyramid What happens in frequency domain? Bandpass filters Can we reconstruct the original from the laplacian pyramid?

31. Laplacian Pyramid What happens in frequency domain? Bandpass filters Can we reconstruct the original from the laplacian pyramid? - Yes, laplacian pyramid is a complete image representation which encodes fine to course structure in a different level

32. Image representation Pixels: great for spatial resolution, poor access to frequency Fourier transform: great for frequency, not for spatial info such as locations. Pyramids/filter banks: balance between spatial and frequency information

33. Major uses of image pyramids Compression Object detection – Scale search – Features Detecting stable interest points Image blending/mosaicing Registration – Course-to-fine

34. Multi-resolution SIFT Feature Detection - Object recognition from local scale-invariant features [pdf link], ICCV 09pdf link - David G. Lowe, "Distinctive image features from scale- invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110

35. Pyramid Blending

36. laplacian level 4 laplacian level 2 laplacian level 0 left pyramidright pyramidblended pyramid

37. Hierarchical Image Registration 1.Compute Gaussian pyramid 2.Align with coarse pyramid 3.Successively align with finer pyramids – Search smaller range Why is this faster? - Hierarchical Model-Based Motion Estimation- Hierarchical Model-Based Motion Estimation, ECCV 92

38. Denoising Additive Gaussian Noise Gaussian Filter

39. Gaussian inputGaussian filter

40. Median Filter For each neighbor in image, sliding the window Sort pixel values Set the center pixel to the median

41. Median Filter input Gaussian filterMedian filter

42. Median Filter Examples inputMedian 7X7

43. Median Filter Examples Median 11X11 Median 3X3

44. Median Filter Examples Median 11X11 Median 3X3 Straight edges kept Sharp features lost

45. Median Filter Properties Can remove outliers (peppers and salts) Window size controls size of structure Preserve some details but sharp corners and edges might get lost

46. Comparison of Mean, Gaussian, and Median originalMean with 6 pixels

47. Comparison of Mean, Gaussian, and Median originalGaussian with 6 pixels

48. Comparison of Mean, Gaussian, and Median originalMedian with 6 pixels

49. Common Problems Mean: blurs image, removes simple noise, no details are preserved

50. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ.

51. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise

52. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves sharp features such as edges?

53. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves sharp features such as edges? -yes, Tomasi's Bilateral Filter!Tomasi's Bilateral Filter

54. Bilateral Filter Example Gaussian filter Bilateral filter

55. Bilateral Filter Example Gaussian filter Bilateral filter

56. 1D Graphical Example Center Sample u It is clear that in weighting this neighborhood, we would like to preserve the step Neighborhood I(p) p

57. The Weights p I(p)

58. Filtered Values p I(p) Filtered value

59. Edges Are Smoothed p I(p) Filtered value

60. What Causes the Problem? p I(p) Filtered value

61. What Causes the Problem? p I(p) Filtered value Same weights for these two pixels!!

62. The Weights p I(p)

63. Bilateral filter Denoise Feature preserving Normalization Bilateral Filtering

64. Filter Parameters As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters: The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing N(u) – The neighbor size of the filter support,  c – The variance of the spatial distances,  s – The variance of the value distances,

65. Bilateral Filter Results Original

66. Bilateral Filter Results σ c = 3, σ s = 3

67. Bilateral Filter Results σ c = 6, σ s = 3

68. Bilateral Filter Results σ c = 12, σ s = 3

69. Bilateral Filter Results σ c = 12, σ s = 6

70. Bilateral Filter Results σ c = 15, σ s = 8

71. Summary – Template matching (SSD or Normxcorr2) SSD can be done with linear filters, is sensitive to overall intensity – Gaussian pyramid Coarse-to-fine search, multi-scale detection – Laplacian pyramid More compact image representation Can be used for compositing in graphics – Image denoising