1. CSCE 641 Computer Graphics: Image Filtering & Feature Detection Jinxiang Chai

2. Review 2D Fourier Transform Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal Antialiasing - prefiltering Gaussian pyramid - Mipmap

3. Review 2D Fourier Transform Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal Antialiasing - prefiltering Gaussian pyramid - Mipmap

4. Review: 2D Fourier Transform Fourier Transform: Inverse Fourier transform:

5. Review: Nyquist Sampling Rate T2T … -2T-T … 0 x f s (x) f(x) x F s (u) u -f max f max F(u) u -f max f max Inverse Fourier transform Fourier transform Sampling Rate ≥ 2 * max frequency in the signal!

6. Review: Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max

7. Review: Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max - Sinc filter - Mean filter - Triangular filter - Gaussian filter

8. Known as a Gaussian Pyramid –MipMap (Williams, 1983) Review: Image Pyramids

9. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

10. Filtering inputGaussian filter

11. Median Filter For each neighbor in image, sliding the window Sort pixel values Set the center pixel to the median

12. Median Filter input Gaussian filterMedian filter

13. Median Filter Examples inputMedian 7X7

14. Median Filter Examples Median 11X11 Median 3X3

15. Median Filter Examples Median 11X11 Median 3X3 Straight edges kept Sharp features lost

16. Median Filter Properties Can remove outliers (peppers and salts) Window size controls size of structure Preserve some details but sharp corners and edges might get lost

17. Comparison of Mean, Gaussian, and Median originalMean with 6 pixels

18. Comparison of Mean, Gaussian, and Median originalGaussian with 6 pixels

19. Comparison of Mean, Gaussian, and Median originalMedian with 6 pixels

20. Common Problems Mean: blurs image, removes simple noise, no details are preserved

21. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ.

22. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise

23. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves sharp features such as edges?

24. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves sharp features such as edges? - yes, bilateral filter

25. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

26. What Is Bilateral Filter? Bilateral - Affecting or undertaken by two sides equally Property: - Convolution filter - Smooth image but preserve edges - Operates in the domain and the range of image

27. Bilateral Filter Example Gaussian filter Bilateral filter

28. Bilateral Filter Example Gaussian filter Bilateral filter

29. 1D Graphical Example Center Sample u It is clear that in weighting this neighborhood, we would like to preserve the step Neighborhood I(p) p

30. The Weights p I(p)

31. Filtered Values p I(p) Filtered value

32. Edges Are Smoothed p I(p) Filtered value

33. What Causes the Problem? p I(p) Filtered value

34. What Causes the Problem? p I(p) Filtered value Same weights for these two pixels!!

35. The Weights p I(p)

36. Bilateral filter Denoise Feature preserving Normalization Bilateral Filtering

37. Kernel Properties Per each sample, we can define a ‘Kernel’ that averages its neighborhood This kernel changes from sample to sample! The sum of the kernel entries is 1 due to the normalization, The center entry in the kernel is the largest, Subject to the above, the kernel can take any form (as opposed to filters which are monotonically decreasing).

38. Filter Parameters As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters: The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing N(u) – The neighbor size of the filter support,  c – The variance of the spatial distances,  s – The variance of the value distances,

39. Bilateral Filter input

40. Bilateral Filter input

41. Bilateral Filter input WcWc

42. Bilateral Filter input WcWc WsWs

43. Bilateral Filter input WcWc WsWs W s *W c

44. Bilateral Filter input WcWc WsWs W s *W c Output

45. Bilateral Filter Results Original

46. Bilateral Filter Results σ c = 3, σ s = 3

47. Bilateral Filter Results σ c = 6, σ s = 3

48. Bilateral Filter Results σ c = 12, σ s = 3

49. Bilateral Filter Results σ c = 12, σ s = 6

50. Bilateral Filter Results σ c = 15, σ s = 8

51. Additional Comments The bilateral filter is a powerful filter: - Can work with any reasonable distance function W s and W c, - Easily extended to higher dimension signals, e.g. Images, video, mesh, animation data etc. - Easily extended to vectored-signals, e.g. RGB images or RGBD images etc. [Fleishman et al, siggraph 03] Bilateral Mesh Denoising

52. Readings on Image Filtering Section 3.3.1 in Szeliski book Tomasi's Bilateral Filter more about bilateral filter, click herehere

53. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

54. Edge Detection What are edges in this image?

55. Edge Detection What are edges in this image?

56. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

57. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

58. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

59. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge? - Related to the first-derivative. - How to compute the first-derivatives of an image?

60. Gradient A vector variable - Direction of the maximum growth of the function - Magnitude of the growth - Perpendicular to the edge direction

61. Gradients

62. How to Calculate Gradient? How to compute ? Intensity Pixel column

63. The Good ole’ Taylor Series Subtracting the second from the first we obtain or…

64. Discrete Gradient Estimation For discrete functions, we can use the first order approximation of the gradient where h corresponds to the step size

65. Discrete Gradient Estimation For discrete functions, we can use the first order approximation of the gradient where h corresponds to the step size For our purposes, h corresponds to the width of 1 pixel =>

66. Discrete Gradient Estimation So how can we compute the image gradient efficiently? - Using our good old friend convolution! - Dropped off the “divide by 2” for speed considerations. - This only scales the gradient.

67. Taking the Discrete Derivative abs()

68. Basic Edge Detection Step 1 INPUT IMAGE 1) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T But isn’t edge detection susceptible to noise?

69. Basic Edge Detection Steps 1-2 INPUT IMAGE 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T 1) Noise Smoothing

70. Discrete Gradient Estimation Remember that the gradient is a vector and we have calculated the coefficients in the x and y directions at each point in the image After convolving, we get the magnitude of the gradient from at each point (pixel) from In practice, we often sum the absolute values of the components for computational efficiency

71. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE

72. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE What does the gradient image mean?

73. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE What does the gradient image mean? - Magnitude of intensity changes around each pixel

74. Discrete Gradient Estimation So how do we segment the edges from the rest of the scene? - Thresholding!

75. The Effects of Thresholding While edge features are independent of illumination, the edge strength is not! Results from threshold values of 50 and 100

76. Basic Edge Detection Summary INPUT IMAGE 1) Noise Smoothing EDGE IMAGE 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE 3)Threshold

77. The effects of Filtering Noise Threshold 20 Gaussian SmoothingUnsmoothed Edges Threshold 50

78. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators Convolving each of these with the original image generates horizontal and vertical gradient images that are combined as before

79. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives

80. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives Why?

81. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives - In practice we may still need to smooth for noise

82. Robert and Prewitt Edge Detectors The Prewitt is similar to the Sobel, but uses a different kernel Roberts was an early edge detector kernel

83. Sobel Edge Detector Java Applet

84. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries

85. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries An alternative approach is to look for local extrema in the first derivative A peak in the first derivative corresponds what in the second derivative?

86. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries An alternative approach is to look for local extrema in the first derivative A peak in the first derivative corresponds what in the second derivative?

87. Localization with Laplacian An equivalent measure of the second derivative in 2D is the Laplacian: Numerically, we approximate the Laplacian using the following filtering kernel: Zeros crossing of the filter corresponds to positions of maximum gradient. - can be used for detecting edges - might be sensitive to noise - need to filter the image

88. The Laplacian of Gaussian Kernel

89. Review: Properties Linearity: Time shift: Derivative: Integration: Convolution:

90. The Laplacian of Gaussian Kernel

91. The Laplacian of Gaussian Filter Kernel

92. Laplacian Edge Detector Java applet: Click here here

93. Readings Section 4.2 in Szeliski book

94. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

95. Corner Detection Why do we need to detect corners? - Corners correspond to points in both the world and image spaces - Very important for image based rendering and modeling - Very useful in computer vision

96. What are Corners? Point features

97. What are Corners? Point features Where two edges come together Where the image gradient has significant components in the x and y direction We will establish corners from the gradient rather than the edge images

98. Basic Ideas What are gradients along x and y directions?

99. Basic Ideas What are gradients along x and y directions?

100. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images?

101. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images? - two major axes in the local window!

102. Basic Idea Corners: two major axes of gradients in the local window How to evaluate this?

103. Basic Idea Corners: two major axes of gradients in the local window How to evaluate this? Gradient vector

104. Basic Idea IxIx IyIy

105. IxIx IyIy IxIx IyIy IxIx IyIy

106. Finding Two Major Axes Compute covariance matrix Compute two eigen values: λ1 and λ2, which indicate magnitude of two dominant axes If both λ1 and λ2 are large, then we have two dominant axes (corners).

107. 1. Compute the image gradients 2. Define a neighborhood size as an area of interest around each pixel Corner Detection Algorithm 3x3 neighborhood

108. 3.For each image pixel (i,j), construct the following matrix from it and its neighborhood values e.g. Corner Detection Algorithm (cont’d) Similar to covariance matrix (I x,I y ) T !

109. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. - Eigenvalues indicate magnitude of two dominant axes - If either λ1 or λ2 is close to zero, then this is not a corner. Simple case:

110. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. Simple case: Isolated pixelsInterior Region Edge Corner λ 1, λ 2 =0 Large λ 1 and small λ 2 Large λ 1 and large λ 2 small λ 1 and small λ 2

111. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. - This is just a rotated version of the one on last slide - If either λ1 or λ2 is close to zero, then this is not a corner. General case:

112. Eigen-values and Corner - λ1 is large - λ2 is large

113. Eigen-values and Corner - λ1 is large - λ2 is small

114. Eigen-values and Corner - λ1 is small - λ2 is small

115. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. 5. If both λ 1 and λ 2 are big, we have a corner (Harris also checks the ratio of λs is not too high) ISSUE: The corners obtained will be a function of the threshold !

116. Image Gradients

117. Closeup of image orientation at each pixel

118. The Orientation Field Corners are detected where both λ1 and λ2 are big

119. The Orientation Field Corners are detected where both λ1 and λ2 are big

120. Corner Detection Sample Results Threshold=25,000Threshold=10,000 Threshold=5,000

121. Results

122. Required Readings Section 4.1.1 in Szeliski book Harris corner detector

123. Homework #1 Due by 9/23, midnight Submit it via csnet. If you do not have the csnet account, submit it to me via email.

124. Summary Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

125. Next Lecture SIFT features Image composition and blending Image warping Image morphing