1. Wavelets (Chapter 7) CS474/674 – Prof. Bebis

2. STFT - revisited Time - Frequency localization depends on window size. –Wide window  good frequency localization, poor time localization. –Narrow window  good time localization, poor frequency localization.

3. Wavelet Transform Uses a variable length window, e.g.: –Narrower windows are more appropriate at high frequencies –Wider windows are more appropriate at low frequencies

4. What is a wavelet? A function that “waves” above and below the x-axis with the following properties: –Varying frequency –Limited duration –Zero average value This is in contrast to sinusoids, used by FT, which have infinite duration and constant frequency. Sinusoid Wavelet

5. Types of Wavelets There are many different wavelets, for example: Morlet Haar Daubechies

6. Basis Functions Using Wavelets Like sin( ) and cos( ) functions in the Fourier Transform, wavelets can define a set of basis functions ψ k (t): Span of ψ k (t): vector space S containing all functions f(t) that can be represented by ψ k (t).

7. Basis Construction – “Mother” Wavelet The basis can be constructed by applying translations and scalings (stretch/compress) on the “mother” wavelet ψ(t): scale translate Example: ψ(t)

8. (dyadic/octave grid) Basis Construction - Mother Wavelet j k scale =1/2 j (1/frequency)

9. Continuous Wavelet Transform (CWT) translation parameter (measure of time) scale parameter (measure of frequency) mother wavelet (i.e., window function) normalization constant Forward CWT: scale =1/2 j (1/frequency)

10. Illustrating CWT 1.Take a wavelet and compare it to a section at the start of the original signal. 2.Calculate a number, C, that represents how closely correlated the wavelet is with this section of the signal. The higher C is, the more the similarity.

11. Illustrating CWT (cont’d) 3. Shift the wavelet to the right and repeat step 2 until you've covered the whole signal.

12. Illustrating CWT (cont’d) 4. Scale the wavelet and go to step 1. 5. Repeat steps 1 through 4 for all scales.

13. Visualize CTW Transform Wavelet analysis produces a time-scale view of the input signal or image.

14. Continuous Wavelet Transform (cont’d) Note the double integral! Inverse CWT: Forward CWT:

15. Fourier Transform vs Wavelet Transform weighted by F(u)

16. Fourier Transform vs Wavelet Transform weighted by C(τ,s)

17. Properties of Wavelets Simultaneous localization in time and scale - The location of the wavelet allows to explicitly represent the location of events in time. - The shape of the wavelet allows to represent different detail or resolution.

18. Properties of Wavelets (cont’d) Sparsity: for functions typically found in practice, many of the coefficients in a wavelet representation are either zero or very small.

19. Properties of Wavelets (cont’d) Adaptability: Can represent functions with discontinuities or corners more efficiently. Linear-time complexity: many wavelet transformations can be accomplished in O(N) time.

20. Discrete Wavelet Transform (DWT) (inverse DWT) (forward DWT) where

21. DFT vs DWT DFT expansion: DWT expansion or one parameter basis two parameter basis

22. Multiresolution Representation Using Wavelets j fine details coarse details wider, large translations

23. j fine details coarse details Multiresolution Representation Using Wavelets

24. j fine details coarse details narrower, small translations Multiresolution Representation Using Wavelets

25. high resolution (more details) low resolution (less details) … j Multiresolution Representation Using Wavelets

26. Pyramidal Coding - Revisited Approximation Pyramid (with sub-sampling)

27. Pyramidal Coding - Revisited Prediction Residual Pyramid (details) reconstruct (with sub-sampling) Approximation Pyramid

28. Efficient Representation Using “Details” details D 2 L0L0 details D 3 details D 1 (without sub-sampling)

29. Efficient Representation Using Details (cont’d) representation: L 0 D 1 D 2 D 3 A wavelet representation of a function consists of (1)a coarse overall approximation (2)detail coefficients that influence the function at various scales in general : L 0 D 1 D 2 D 3 …D J

30. Reconstruction (synthesis) H 3 =H 2 & D 3 details D 2 L0L0 details D 3 H 2 =H 1 & D 2 H 1 =L 0 & D 1 details D 1 (without sub-sampling)

31. Example - Haar Wavelets Suppose we are given a 1D "image" with a resolution of 4 pixels: [9 7 3 5] The Haar wavelet transform is the following: L 0 D 1 D 2 D 3 (with sub-sampling)

32. Example - Haar Wavelets (cont’d) Start by averaging and subsampling the pixels together (pairwise) to get a new lower resolution image: To recover the original four pixels from the two averaged pixels, store some detail coefficients. 1 [9 7 3 5]

33. Example - Haar Wavelets (cont’d) Repeating this process on the averages (i.e., low resolution image) gives the full decomposition: 1 Harr decomposition:

34. Example - Haar Wavelets (cont’d) The original image can be reconstructed by adding or subtracting the detail coefficients from the lower- resolution representations. 2 1 -1 [6] L 0 D 1 D 2 D 3

35. Example - Haar Wavelets (cont’d) Detail coefficients become smaller and smaller scale decreases. DjDj D j-1 D1D1 L0L0 How should we compute the detail coefficients D j ?

36. Multiresolution Conditions larger setIf a set of functions V can be represented by a weighted sum of ψ (2 j t - k), then a larger set, including V, can be represented by a weighted sum of ψ (2 j+1 t - k). low resolution high resolution j ψ(2 j t - k) ψ(2 j+1 t - k)

37. Multiresolution Conditions (cont’d) V j : span of ψ(2 j t - k): V j+1 : span of ψ(2 j+1 t - k):

38. ψ(t - k) ψ(2t - k) ψ(2 j t - k) … V0V0 V1V1 VjVj if f(t) V j then f(t) V j+1 j=0 j=1 j Nested Spaces Multiresolution Conditions (cont’d)

39. How to compute D j ? Define a set of basis functions that span the difference between V j+1 and V j IDEA:

40. Let W j be the orthogonal complement of V j in V j+1 V j+1 = V j + W j How to compute D j ? (cont’d)

41. If f(t) V j+1, then f(t) can be represented using basis functions φ(t) from V j+1 : V j+1 = V j + W j Alternatively, f(t) can be represented using two sets of basis functions, φ(t) from V j and ψ(t) from W j : V j+1

42. Think of W j as a means to represent the parts of a function in V j+1 that cannot be represented in V j V j W j How to compute D j ? (cont’d) differences between V j and V j+1 V j+1

43. How to compute D j ? (cont’d)  using recursion on V j : V 0 W 0, W 1, W 2, … basis functions V j+1 = V j-1 +W j-1 +W j = …= V 0 + W 0 + W 1 + W 2 + … + W j if f(t) V j+1, then: V j+1 = V j + W j

44. Summary: wavelet expansion (Section 7.2) Wavelet decompositions involve a pair of waveforms: φ(t) ψ(t) encodes low resolution info encodes details or high resolution info scaling function wavelet functionTerminology:

45. 1D Haar Wavelets Haar scaling and wavelet functions: computes average (low pass) computes details (high pass) φ(t) ψ(t)

46. Let’s consider the spaces corresponding to different resolution 1D images: 1D Haar Wavelets (cont’d) etc.... …. 1-pixel 2-pixel 4-pixel (j=0) (j=1) (j=2) V0V0 V1V1 V2V2

47. 1D Haar Wavelets (cont’d) V 0 represents the space of 1-pixel (2 0 -pixel) images Think of a 1-pixel image as a function that is constant over [0,1) Example: 0 1 width: 1 j=0

48. 1D Haar Wavelets (cont’d) V 1 represents the space of all 2-pixel (2 1 -pixel) images Think of a 2-pixel image as a function having 2 1 equal- sized constant pieces over the interval [0, 1). Example: 0 ½ 1 = + width: 1/2 e.g., j=1 Note that:

49. 1D Haar Wavelets (cont’d) V j represents all the 2 j -pixel images Functions having 2 j equal-sized constant pieces over interval [0,1). V j-1 Example: width: 1/2 j V j Note that: width: 1/2 j-1 width: 1/2 V1V1

50. 1D Haar Wavelets (cont’d) V 0, V 1,..., V j are nested i.e., Vj…V1V0Vj…V1V0 coarse info fine details

51. Define a basis for V j Scaling function: Let’s define a basis for V j : Note new notation:

52. Define a basis for V j (cont’d) width: 1/2 0 width: 1/2width: 1/2 2 width: 1/2 3

53. Define a basis for W j Wavelet function: Let’s define a basis ψ j i for W j : Note new notation:

54. Define basis for W j (cont’d) Note that the dot product between basis functions in V j and W j is zero!

55. Basis for V j+1 Basis functions ψ j i of W j Basis functions φ j i of V j form a basis in V j+1

56. Define a basis for W j (cont’d) V 3 = V 2 + W 2

57. Define a basis for W j (cont’d) V 2 = V 1 + W 1

58. Define a basis for W j (cont’d) V 1 = V 0 + W 0

59. Example - Revisited f(x)= V2V2

60. φ 2,0 (x) φ 2,1 (x) φ 2,2 (x) φ 2,3 (x) Example (cont’d) V2V2 f(x)=

61. Example (cont’d) V 1 and W 1 V 2 =V 1 +W 1 φ 1,0 (x) φ 1,1 (x) ψ 1,0 (x) ψ 1,1 (x) (divide by 2 for normalization)

62. Example (cont’d)

63. V 2 =V 1 +W 1 =V 0 +W 0 +W 1 V 0,W 0 and W 1 φ 0,0 (x) ψ 0,0 (x) ψ 1,0 (x) ψ 1,1 (x) (divide by 2 for normalization)

64. Example

65. Example (cont’d)

66. Filter banks The lower resolution coefficients can be calculated from the higher resolution coefficients by a tree- structured algorithm (i.e., filter bank). h 0 (-n) is a lowpass filter and h 1 (-n) is a highpass filter Subband encoding (analysis)

67. Example (revisited) [9 7 3 5] low-pass, down-sampling high-pass, down-sampling (9+7)/2 (3+5)/2 (9-7)/2 (3-5)/2 LP HP

68. Filter banks (cont’d) Next level: h 0 (-n) is a lowpass filter and h 1 (-n) is a highpass filter

69. Example (revisited) [9 7 3 5] high-pass, down-sampling low-pass, down-sampling (8+4)/2 (8-4)/2 LP HP

70. Convention for illustrating 1D Haar wavelet decomposition x x x x x x … x x detail (HP) average (LP) … re-arrange: LP HP … … … …

71. Examples of lowpass/highpass (analysis) filters Daubechies Haar h0h0 h1h1 h0h0 h1h1

72. Filter banks (cont’d) The higher resolution coefficients can be calculated from the lower resolution coefficients using a similar structure. Subband encoding (synthesis) g 0 (n) is a lowpass filter and g 1 (n) is a highpass filter

73. Filter banks (cont’d) Next level: g 0 (n) is a lowpass filter and g 1 (n) is a highpass filter

74. Examples of lowpass/highpass (synthesis) filters Daubechies: Haar (same as for analysis): + g0g0 g1g1 g0g0 g1g1

75. 2D Haar Wavelet Transform The 2D Haar wavelet decomposition can be computed using 1D Haar wavelet decompositions. –i.e., 2D Haar wavelet basis is separable We’ll discuss two different decompositions (i.e., correspond to different basis functions): –Standard decomposition –Non-standard decomposition

76. Standard Haar wavelet decomposition Steps: (1) Compute 1D Haar wavelet decomposition of each row of the original pixel values. (2) Compute 1D Haar wavelet decomposition of each column of the row-transformed pixels.

77. Standard Haar wavelet decomposition (cont’d) x x x … x … …. x x x... x (1) row-wise Haar decomposition: … detail average … … …. … … re-arrange terms

78. Standard Haar wavelet decomposition (cont’d) (1) row-wise Haar decomposition: … detail average … … … …. … row-transformed result … … ….

79. Standard Haar wavelet decomposition (cont’d) (2) column-wise Haar decomposition: … detail average … … … …. … … … … row-transformed result column-transformed result

80. Example … … … … …. row-transformed result … … …. re-arrange terms

81. Example (cont’d) … … … … …. column-transformed result

82. Standard Haar wavelet decomposition (cont’d)

83. What is the 2D Haar basis for the standard decomposition? To construct the standard 2D Haar wavelet basis, consider all possible outer products of the1D basis functions. φ 0,0 (x) ψ 0,0 (x) ψ 1,0 (x) ψ 1,1 (x) V 2 =V 0 +W 0 +W 1 Example:

84. To construct the standard 2D Haar wavelet basis, consider all possible outer products of the1D basis functions. φ 00 (x), φ 00 (x) ψ 00 (x), φ 00 (x) ψ 10 (x), φ 00 (x) What is the 2D Haar basis for the standard decomposition? Notation:

85. V2V2 What is the 2D Haar basis for the standard decomposition? Notation:

86. Non-standard Haar wavelet decomposition Alternates between operations on rows and columns. (1) Perform one level decomposition in each row (i.e., one step of horizontal pairwise averaging and differencing). (2) Perform one level decomposition in each column from step 1 (i.e., one step of vertical pairwise averaging and differencing). (3) Rearrange terms and repeat the process on the quadrant containing the averages only.

87. Non-standard Haar wavelet decomposition (cont’d) x x x … x … …. x x x... x one level, horizontal Haar decomposition: … … … …. … … … one level, vertical Haar decomposition: …

88. Non-standard Haar wavelet decomposition (cont’d) one level, horizontal Haar decomposition on “green” quadrant one level, vertical Haar decomposition on “green” quadrant … … … …. … … re-arrange terms … … … … …. …

89. Example … … … …. … … re-arrange terms

90. Example (cont’d) … … … … ….

91. Non-standard Haar wavelet decomposition (cont’d)

92. What is the 2D Haar basis for the non- standard decomposition? Define 2D scaling and wavelet functions: Apply translations and scaling:

93. V2V2 What is the 2D Haar basis for the non- standard decomposition? Notation:

94. LL LH: intensity variations along columns (horizontal edges) HL: intensity variations along rows (vertical edges) HH: intensity variations along diagonal LL: average Detail coefficients Three sets of detail coefficients (i.e., subband coding) 2D Subband Coding

95. LL LH HLHH

96. Wavelets Applications Noise filtering Image compression –Special case: fingerprint compression Image fusion Recognition G. Bebis, A. Gyaourova, S. Singh, and I. Pavlidis, "Face Recognition by Fusing Thermal Infrared and Visible Imagery", Image and Vision Computing, vol. 24, no. 7, pp. 727-742, 2006. Image matching and retrieval Charles E. Jacobs Adam Finkelstein David H. Salesin, "Fast Multiresolution Image Querying", SIGRAPH, 1995.