1. CSCE 641 Computer Graphics: Image Sampling and Reconstruction Jinxiang Chai

2. Review: 1D Fourier Transform A function f(x) can be represented as a weighted combination of phase-shifted sine waves How to compute F(u)? Inverse Fourier Transform Fourier Transform

3. Review: Box Function x u f(x) |F(u)| If f(x) is bounded, F(u) is unbounded

4. Review: Cosine 1  If f(x) is even, so is F(u)

5. Review: Gaussian If f(x) is gaussian, F(u) is also guassian.

6. Review: Properties Linearity: Time shift: Derivative: Integration: Convolution:

7. Review: Properties Linearity: Time shift: Derivative: Integration: Convolution:

8. Review: Properties Linearity: Time shift: Derivative: Integration: Convolution:

9. Outline 2D Fourier Transform Nyquist sampling theory Antialiasing Gaussian pyramid

10. Extension to 2D Fourier Transform: Inverse Fourier transform:

11. Building Block for 2D Transform Building block: Frequency: Orientation: Oriented wave fields

12. Building Block for 2D Transform Higher frequency Building block: Frequency: Orientation: Oriented wave fields A function f(x,y) can be represented as a weighted combination of phase-shifted oriented wave fields.

13. Some 2D Transforms From Lehar

14. Some 2D Transforms From Lehar

15. Some 2D Transforms From Lehar

16. Some 2D Transforms From Lehar

17. Some 2D Transforms From Lehar

18. Some 2D Transforms From Lehar Why we have a DC component?

19. Some 2D Transforms Why we have a DC component? - the sum of all pixel values From Lehar

20. Some 2D Transforms Why we have a DC component? - the sum of all pixel values Oriented stripe in spatial domain = an oriented line in spatial domain From Lehar

21. 2D Fourier Transform Why? - Any relationship between two slopes?

22. 2D Fourier Transform Linearity Why? - Any relationship between two slopes?

23. 2D Fourier Transform The spectrum is bounded by two slopes. Linearity Why? - Any relationship between two slopes?

24. Online Java Applet http://www.brainflux.org/java/classes/FFT2D Applet.html

25. 2D Fourier Transform Pairs Gaussian

26. 2D Image Filtering Inverse transform Fourier transform From Lehar

27. 2D Image Filtering Inverse transform Fourier transform Low-pass filter From Lehar

28. 2D Image Filtering Inverse transform Fourier transform Low-pass filter high-pass filter From Lehar

29. 2D Image Filtering Inverse transform Fourier transform Low-pass filter high-pass filter band-pass filter From Lehar

30. Aliasing Why does this happen?

31. Aliasing How to reduce it?

32. Sampling Analysis f(x) x T2T … -2T-T … 0 f s (x) x Sampling

33. Sampling Analysis f(x) x T2T … -2T-T … 0 f s (x) x Sampling Reconstruction

34. Sampling Analysis What sampling rate (T) is sufficient to reconstruct the continuous version of the sampled signal? f(x) x T2T … -2T-T … 0 f s (x) x Sampling Reconstruction

35. Sampling Theory How many samples are required to represent a given signal without loss of information? What signals can be reconstructed without loss for a given sampling rate?

36. Sampling Analysis: Spatial Domain f(x) T2T … -2T-T … 0 f s (x) x xT2T … -2T-T … 0x X ? continuous signal discrete signal (samples) comb function

37. Sampling Analysis: Spatial Domain f(x) T2T … -2T-T … 0 f s (x) x xT2T … -2T-T … 0x X ?What happens in Frequency domain? continuous signal discrete signal (samples) comb function

38. Sampling Analysis: Spatial Domain f(x) T2T … -2T-T … 0 f s (x) x xT2T … -2T-T … 0x X ?What happens in Frequency domain? continuous signal discrete signal (samples) comb function

39. Fourier Transform of Dirac Comb T

40. Review: Dirac Delta and its Transform x 1 u f(x) |F(u)| Fourier transform and inverse Fourier transform are qualitatively the same, so knowing one direction gives you the other

41. Review: Fourier Transform Properties Linearity: Time shift: Derivative: Integration: Convolution:

42. Fourier Transform of Dirac Comb T

44. T1/T1/T - Fourier transform of a Dirac comb is a Dirac comb as well. - Moving the spikes closer together in the spatial domain moves them farther apart in the frequency domain!

45. Sampling Analysis: Spatial Domain f(x) T2T … -2T-T … 0 f s (x) x xT2T … -2T-T … 0x X ? What happens in Frequency domain? continuous signal discrete signal (samples) comb function

46. Review: Properties Linearity: Time shift: Derivative: Integration: Convolution:

47. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max

48. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max How does the convolution result look like?

49. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max

50. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max

51. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max G(0)? G(f max )? G(u)?

52. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max G(0) = F(0) G(f max ) = F(f max ) G(u) = F(u)

53. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max F s (u) u -f max f max -1/T1/T How about

54. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max F s (u) u -f max f max -1/T1/T How about

55. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max F s (u) u -f max f max -1/T1/T

56. Sampling Theory How many samples are required to represent a given signal without loss of information? What signals can be reconstructed without loss for a given sampling rate?

57. Sampling Analysis: Freq. Domain 1/T…-1/T…0u F(u) u -f max f max F s (u) u -f max f max -1/T1/T How can we reconstruct the original signal?

58. Reconstruction in Freq. Domain u -f max f max F s (u) u -f max f max box(u) u -f max f max

59. Signal Reconstruction in Freq. Domain T2T … -2T-T … 0 x f s (x) f(x) x F s (u) u -f max f max F(u) u -f max f max Inverse Fourier transform Fourier transform

60. Signal Reconstruction in Spatial Domain T2T … -2T-T … 0 x x f s (x) sinc(x)

61. Aliasing Why does this happen?

62. Sampling Analysis T … -T … 0 f(x)F(u) f s (x) x ux -f max f max F s (u) u -f max f max When does aliasing happen?

63. Sampling Analysis T … -T … 0 f(x)F(u) f s (x) x ux -f max f max F s (u) u -f max f max When does aliasing happen?

64. Sampling Analysis T … -T … 0 f(x)F(u) f s (x) x ux -f max f max F s (u) u -f max f max When does aliasing happen?

65. Sampling Analysis T … -T … 0 f(x)F(u) f s (x) x ux -f max f max F s (u) u -f max f max When does aliasing happen?

66. Sampling Analysis What sampling rate (T) is sufficient to reconstruct the continuous version of the sampled signal? f(x) x T2T … -2T-T … 0 f s (x) x Sampling Reconstruction

67. Sampling Analysis What sampling rate (T) is sufficient to reconstruct the continuous version of the sampled signal? f(x) x T2T … -2T-T … 0 f s (x) x Sampling Reconstruction Sampling Rate ≥ 2 * max frequency in the signal this is known as the Nyquist Rate

68. Antialiasing

69. Increase the sampling rate to above twice the highest frequency u -f max f max

70. Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max

71. Antialiasing Increase the sampling rate to above twice the highest frequency Introduce an anti-aliasing filter to reduce f max u -f max f max F s (u) u -f max f max

72. Sinc Filter F(u) u -f max f max Is this a good filter?

73. Sinc Filter F(u) u -f max f max x Inverse Fourier transform Is this a good filter?

74. Sinc Filter Is this a good filter? F(u) u -f max f max x Multiplying with a box function in frequency domain Convolution with a sinc function in spatial domain Inverse Fourier transform

75. Sinc Filter Is this a good filter? F(u) u -f max f max x Multiplying with a box function in frequency domain Convolution with a sinc function in spatial domain Good: removes all frequency components above a given bandwidth Bad: - an infinite support in spatial domain and hard to do in the spatial domain - negative weights (not good for filtering pixels) - fluctuation causes ripples Inverse Fourier transform

76. Good Pre-filtering filters Finite support in frequency domain - cut-off frequency Finite support in spatial domain - implementation in spatial domain Positive weights - good for filtering pixels

77. Other Filter Choices Mean filter Triangular filter Gaussian filter x f(x) u |F(u)| sinc(u) Sin 2 c(u) Cannot have a filter with finite support in both spatial and frequency domain (box, mean, triangular) - box filter: finite in freq. and infinite in spatial domain - mean and triangular: infinite in freq. and finite in spatial domain Gaussian filter provides a good tradeoff between two domains

78. Other Filter Choices Mean filter Triangular filter Gaussian filter x f(x) u |F(u)| sinc(u) Sin 2 c(u) Cannot have a filter with finite support in both spatial and frequency domain (box, mean, triangular) - box filter: finite in freq. and infinite in spatial domain - mean and triangular: infinite in freq. and finite in spatial domain Gaussian filter provides a good tradeoff between two domains

79. Gaussian Filters Cannot have a filter with finite support in both spatial and frequency domain (box, mean, triangular) - box filter: finite in freq. and infinite in spatial domain - mean and triangular: infinite in freq. and finite in spatial domain Gaussian filter provides a good tradeoff between two domains

80. Gaussian Filters Cannot have a filter with finite support in both spatial and frequency domain (box, mean, triangular) - box filter: finite in freq. and infinite in spatial domain - mean and triangular: infinite in freq. and finite in spatial domain Gaussian filter provides a good tradeoff between two domains

81. Gaussian Filter F s (u) u -f max f max

82. Gaussian Filter F s (u) u -f max f max -0.02 0.03 0.08 0.13 0.18 Inverse Fourier transform

83. Gaussian Filter F s (u) u -f max f max Multiplying with gaussian function in frequency domain Convolution with a gaussian function in spatial domain -0.02 0.03 0.08 0.13 0.18 Inverse Fourier transform

84. Gaussian Filter F s (u) u -f max f max Multiplying with gaussian function in frequency domain Convolution with a gaussian function in spatial domain - Since the Gaussian function decays rapidly, it is reasonable to truncate the filter window -Easy to implement it in the spatial domain - Its standard deviation controls the smoothness of filtered signal: large σ in spatial domain = lower cutoff frequency -0.02 0.03 0.08 0.13 0.18 Inverse Fourier transform

85. Review: Gaussian If f(x) is gaussian, F(u) is also guassian.

86. Filtering in Spatial Domain = Filter functionInput imageFiltered image

87. Gaussian Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 filtering window: 5X5 ? Step 1: evaluate the filtering function at each grid Step 2: Normalize each weight to ensure the sum of all weights is one

88. Gaussian Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 filtering window: 5X5 G(0,0) Step 1: evaluate the filtering function at each grid Step 2: Normalize each weight to ensure the sum of all weights is one

89. Gaussian Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 filtering window: 5X5 G(0,0) ? Step 1: evaluate the filtering function at each grid Step 2: Normalize each weight to ensure the sum of all weights is one

90. Gaussian Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 filtering window: 5X5 G(0,0) Step 1: evaluate the filtering function at each grid Step 2: Normalize each weight to ensure the sum of all weights is one G(-2,-2)

91. Gaussian Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 Step 1: evaluate the filtering function at each grid Step 2: Normalize each weight to ensure the sum of all weights is one

92. Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0

93. Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0

94. Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0

95. Filtering in Spatial Domain Discrete approximation to Gaussian function with σ =1.0 Filtered_I 45 = X

96. Gaussian Pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample –Filter size should double for each ½ size reduction. Why?

97. Gaussian Prefiltering G 1/4G 1/8Gaussian 1/2 Solution: filter the image, then subsample –Filter size should double for each ½ size reduction. Why?

98. Without Prefiltering 1/4 (2x zoom) 1/8 (4x zoom) 1/2

99. Known as a Gaussian Pyramid –MipMap (Williams, 1983) Image Pyramids

100. MipMap Mip (latin phase): many things in a small place Used for texture mapping Small-scale texture used for rendering distant objects

101. Gaussian Pyramid filter mask Repeat –Filter –Subsample Until minimum resolution reached –can specify desired number of levels (e.g., 3-level pyramid ) The whole pyramid is only 4/3 the size of the original image!

102. Gaussian Pyramid

103. Laplacian Pyramid Laplacian Pyramid (subband images) - Created from Gaussian pyramid by subtraction - Encode fine to course structure in a different level

104. Laplacian Pyramid What happens in frequency domain?

105. Where Are They Used? Texture mapping - Mipmap Image processing - Image composition/blending - Image compression Computer vision - Image matching - Motion tracking - Optical flow estimation

106. Summary 2D Fourier Transform Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal Antialiasing - prefiltering Gaussian pyramid - Mipmap

107. Next Lecture Image Processing - Filtering noise - Bilateral filter - Feature extraction