1. Computational Photography: Image Processing Jinxiang Chai

2. Review 2D Fourier Transform Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal Antialiasing - prefiltering Gaussian pyramid - Mipmap

3. Review 2D Fourier Transform Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal Antialiasing - prefiltering Gaussian pyramid - Mipmap

4. Review: 2D Fourier Transform Fourier Transform: Inverse Fourier transform:

5. Review: Nyquist Sampling Rate T2T … -2T-T … 0 x f s (x) f(x) x F s (u) u -f max f max F(u) u -f max f max Inverse Fourier transform Fourier transform Sampling Rate ≥ 2 * max frequency in the signal!

6. Review: Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max

7. Review: Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max - Sinc filter - Mean filter - Triangular filter - Gaussian filter

8. Known as a Gaussian Pyramid –MipMap (Williams, 1983) Review: Image Pyramids

9. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

10. Filtering inputGaussian filter

11. Median Filter For each neighbor in image, sliding the window Sort pixel values Set the center pixel to the median

12. Median Filter input Gaussian filterMedian filter

13. Median Filter Examples inputMedian 7X7

14. Median Filter Examples Median 11X11 Median 3X3

15. Median Filter Examples Median 11X11 Median 3X3 Straight edges kept Sharp features lost

16. Median Filter Properties Can remove outliers (peppers and salts) Window size controls size of structure Preserve some details but sharp corners and edges might get lost

17. Comparison of Mean, Gaussian, and Median originalMean with 6 pixels

18. Comparison of Mean, Gaussian, and Median originalGaussian with 6 pixels

19. Comparison of Mean, Gaussian, and Median originalMedian with 6 pixels

20. Common Problems Mean: blurs image, removes simple noise, no details are preserved

21. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ.

22. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise

23. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves edges?

24. Common Problems Mean: blurs image, removes simple noise, no details are preserved Gaussian: blurs image, preserves details only for small σ. Median: preserves some details, good at removing strong noise Can we find a filter that not only smooths regions but preserves edges? - yes, bilateral filter

25. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

26. What Is Bilateral Filter? Bilateral - Affecting or undertaken by two sides equally Property: - Convolution filter - Smooth image but preserve edges - Operates in the domain and the range of image

27. Bilateral Filter Example Gaussian filter Bilateral filter

28. Bilateral Filter Example Gaussian filter Bilateral filter

29. 1D Graphical Example Center Sample u It is clear that in weighting this neighborhood, we would like to preserve the step Neighborhood I(p) p

30. The Weights p I(p)

31. Filtered Values p I(p) Filtered value

32. Edges Are Smoothed p I(p) Filtered value

33. What Causes the Problem? p I(p) Filtered value

34. What Causes the Problem? p I(p) Filtered value Same weights for these two pixels!!

35. The Weights p I(p)

36. Bilateral filter Denoise Feature preserving Normalization Bilateral Filtering

37. Kernel Properties Per each sample, we can define a ‘Kernel’ that averages its neighborhood This kernel changes from sample to sample! The sum of the kernel entries is 1 due to the normalization, The center entry in the kernel is the largest, Subject to the above, the kernel can take any form (as opposed to filters which are monotonically decreasing).

38. Filter Parameters As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters: The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing N(u) – The neighbor size of the filter support,  c – The variance of the spatial distances,  s – The variance of the value distances,

39. Bilateral Filter input

40. Bilateral Filter input

41. Bilateral Filter input WcWc

42. Bilateral Filter input WcWc WsWs

43. Bilateral Filter input WcWc WsWs W s *W c

44. Bilateral Filter input WcWc WsWs W s *W c Output

45. Bilateral Filter Results Original

46. Bilateral Filter Results σ c = 3, σ s = 3

47. Bilateral Filter Results σ c = 6, σ s = 3

48. Bilateral Filter Results σ c = 12, σ s = 3

49. Bilateral Filter Results σ c = 12, σ s = 6

50. Bilateral Filter Results σ c = 15, σ s = 8

51. Additional Comments The bilateral filter is a powerful filter: - Can work with any reasonable distances function W s and W c, - Easily extended to higher dimension signals, e.g. Images, video, mesh, animation data etc. - Easily extended to vectored-signals, e.g. Color images, etc. [Fleishman et al, siggraph 03] Bilateral Mesh Denoising

52. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

53. Edge Detection What are edges in this image?

54. Edge Detection What are edges in this image?

55. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

56. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

57. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge?

58. Edge Detection One of the most important image processing problems - really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications How to tell a pixel is on the edge? - Related to the first-derivative. - How to compute the first-derivatives of an image?

59. Gradient A vector variable - Direction of the maximum growth of the function - Magnitude of the growth - Perpendicular to the edge direction

60. How to Calculate Gradient? How to compute ? Intensity Pixel column

61. The Good ole’ Taylor Series Subtracting the second from the first we obtain or…

62. Discrete Gradient Estimation For discrete functions, we can use the first order approximation of the gradient where h corresponds to the step size

63. Discrete Gradient Estimation For discrete functions, we can use the first order approximation of the gradient where h corresponds to the step size For our purposes, h corresponds to the width of 1 pixel =>

64. Discrete Gradient Estimation So how can we compute the image gradient efficiently? - Using our good old friend convolution! - Dropped off the “divide by 2” for speed considerations. - This only scales the gradient.

65. Taking the Discrete Derivative abs()

66. Basic Edge Detection Step 1 INPUT IMAGE 1) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T But isn’t edge detection susceptible to noise?

67. Basic Edge Detection Steps 1-2 INPUT IMAGE 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T 1) Noise Smoothing

68. Discrete Gradient Estimation Remember that the gradient is a vector and we have calculated the coefficients in the x and y directions at each point in the image After convolving, we get the magnitude of the gradient from at each point (pixel) from In practice, we often sum the absolute values of the components for computational efficiency

69. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE

70. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE What does the gradient image mean?

71. Basic Edge Detection (cont’d) INPUT IMAGE 1) Noise Smoothing 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE What does the gradient image mean? - Magnitude of intensity changes around each pixel

72. Discrete Gradient Estimation So how do we segment the edges from the rest of the scene? - Thresholding!

73. The Effects of Thresholding While edge features are independent of illumination, the edge strength is not! Results from threshold values of 50 and 100

74. Basic Edge Detection Summary INPUT IMAGE 1) Noise Smoothing EDGE IMAGE 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE 3)Threshold

75. The effects of Filtering Noise Threshold 20 Gaussian SmoothingUnsmoothed Edges Threshold 50

76. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators Convolving each of these with the original image generates horizontal and vertical gradient images that are combined as before

77. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives

78. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives Why?

79. Sobel Edge Detection One of the dominant edge detection schemes uses the Sobel operators - Can be approximated as a derivative of Gaussian - First Gaussian smoothing and then compute derivatives - In practice we may still need to smooth for noise

80. Robert and Prewitt Edge Detectors The Prewitt is similar to the Sobel, but uses a different kernel Roberts was an early edge detector kernel

81. Sobel Edge Detector Java Applet

82. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries

83. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries An alternative approach is to look for local extrema in the first derivative A peak in the first derivative corresponds what in the second derivative?

84. Second Derivative Edge Detector Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries An alternative approach is to look for local extrema in the first derivative A peak in the first derivative corresponds what in the second derivative?

85. Localization with Laplacian An equivalent measure of the second derivative in 2D is the Laplacian: Numerically, we approximate the Laplacian using the following filtering kernel: Zeros crossing of the filter corresponds to positions of maximum gradient. - can be used for detecting edges - might be sensitive to noise - need to filter the image

86. The Laplacian of Gaussian Kernel

87. The Laplacian of Gaussian Filter Kernel

88. Laplacian Edge Detector Java applet: Click here here

89. Outline Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

90. Corner Detection Why do we need to detect corners? - Corners correspond to points in both the world and image spaces - Very important for image based rendering and modeling - Very useful in computer vision

91. What are Corners? Point features Where two edges come together Where the image gradient has significant components in the x and y direction We will establish corners from the gradient rather than the edge images

92. Basic Ideas What are gradients along x and y directions?

93. Basic Ideas What are gradients along x and y directions?

94. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images?

95. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images? - two major axes in the local window!

96. Corner Detection Algorithm 1. Compute the image gradients 2. Define a neighborhood size as an area of interest around each pixel 3x3 neighborhood

97. 3.For each image pixel (i,j), construct the following matrix from it and its neighborhood values e.g. Corner Detection Algorithm (cont’d) Similar to covariance matrix (I x,I y ) T !

98. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. - This means dominant gradient direction aligns with x or y axis. - If either λ1 or λ2 is close to zero, then this is not a corner. Simple case:

99. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. Simple case: Isolated pixelsInterior Region Edge Corner λ 1, λ 2 =0 Large λ 1 and small λ 2 Large λ 1 and large λ 2 small λ 1 and small λ 2

100. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. - This is just a rotated version of the one on last slide - If either λ1 or λ2 is close to zero, then this is not a corner. General case:

101. Eigen-values and Corner - λ1 is large - λ2 is large

102. Eigen-values and Corner - λ1 is large - λ2 is small

103. Eigen-values and Corner - λ1 is small - λ2 is small

104. Corner Detection Algorithm (cont’d) 4.For each matrix C (i,j), determine the 2 eigenvalues λ (i.j) = [λ 1, λ 2 ]. 5. If both λ 1 and λ 2 are big, we have a corner (Harris also checks the ratio of λs is not too high) ISSUE: The corners obtained will be a function of the threshold !

105. Image Gradients

106. Closeup of image orientation at each pixel

107. The Orientation Field Corners are detected where both λ1 and λ2 are big

108. The Orientation Field Corners are detected where both λ1 and λ2 are big

109. Corner Detection Sample Results Threshold=25,000Threshold=10,000 Threshold=5,000

110. Summary Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

111. Next Lecture Image composition and blending Image warping Image morphing