1. Presents

2. The Story of Wavelets
Robi Polikar Dept. of Electrical & Computer Engineering Rowan University

3. The Story of Wavelets Technical Overview Application Overview
But…We cannot do that with Fourier Transform….
Time - frequency representation and the STFT
Continuous wavelet transform
Multiresolution analysis and discrete wavelet transform (DWT)
Application Overview
Conventional Applications: Data compression, denoising, solution of PDEs, biomedical signal analysis.
Unconventional applications
Yes…We can do that with wavelets too…
Historical Overview
1807 ~ 1940s: The reign of the Fourier Transform
1940s ~ 1970s: STFT and Subband Coding
1980s & 1990s: The Wavelet Transform and MRA

4. What is a Transform and Why Do we Need One ?
Transform: A mathematical operation that takes a function or sequence and maps it into another one
Transforms are good things because…
The transform of a function may give additional /hidden information about the original function, which may not be available /obvious otherwise
The transform of an equation may be easier to solve than the original equation (recall your fond memories of Laplace transforms in DFQs)
The transform of a function/sequence may require less storage, hence provide data compression / reduction
An operation may be easier to apply on the transformed function, rather than the original function (recall other fond memories on convolution).

5. December, 21, 1807
Jean B. Joseph Fourier
( )
“An arbitrary function, continuous or with discontinuities, defined in a finite interval by an arbitrarily capricious graph can always be expressed as a sum of sinusoids”
J.B.J. Fourier
He announced these results in a prize paper on the theory of heat.
The judges: Laplace, Lagrange and Legendre
Three of the judges found it incredible that sum of sines and cosines could add up to anything but an infinitely differential function, but...
Lagrange: Lack of mathematical rigor and generality
Denied publication….
After 15 years, following several attempts and disappointments and frustration, he published his results in Theorie Analytique de la Chaleur in 1822 (Analytical Theory of Heat).
In 1829, Dirichlet proved Fourier’s claim with very few and non-restricting conditions.
Next 150 years: His ideas expanded and generalized. 1965: Cooley and Tukey--> Fast Fourier Transform==> Computational simplicity==>King of all transforms…Many applications engineering, finance, etc.

6. Complex function representation through simple building blocks
Basis functions
Using only a few blocks  Compressed representation
Using sinusoids as building blocks  Fourier transform
Frequency domain representation of the function

7. How Does FT Work Anyway?
Recall that FT uses complex exponentials (sinusoids) as building blocks.
For each frequency of complex exponential, the sinusoid at that frequency is compared to the signal.
If the signal consists of that frequency, the correlation is high  large FT coefficients.
If the signal does not have any spectral component at a frequency, the correlation at that frequency is low / zero,  small / zero FT coefficient.

8. FT At Work

9. FT At Work F F F

10. FT At Work F

11. Complex exponentials (sinusoids) as basis functions:
FT At Work
Complex exponentials (sinusoids) as basis functions: F
An ultrasonic A-scan using 1.5 MHz transducer, sampled at 10 MHz

12. Stationary and Non-stationary Signals
FT identifies all spectral components present in the signal, however it does not provide any information regarding the temporal (time) localization of these components. Why?
Stationary signals consist of spectral components that do not change in time
all spectral components exist at all times
no need to know any time information
FT works well for stationary signals
However, non-stationary signals consists of time varying spectral components
How do we find out which spectral component appears when?
FT only provides what spectral components exist , not where in time they are located.
Need some other ways to determine time localization of spectral components

13. Stationary and Non-stationary Signals
Stationary signals’ spectral characteristics do not change with time
Non-stationary signals have time varying spectra
Concatenation

14. Stationary vs. Non-Stationary
X4(ω)
Perfect knowledge of what
frequencies exist, but no
information about where
these frequencies are
located in time
X5(ω)

15. Shortcomings of the FT Sinusoids and exponentials
Stretch into infinity in time, no time localization
Instantaneous in frequency, perfect spectral localization
Global analysis does not allow analysis of non-stationary signals
Need a local analysis scheme for a time-frequency representation (TFR) of nonstationary signals
Windowed F.T. or Short Time F.T. (STFT) : Segmenting the signal into narrow time intervals, narrow enough to be considered stationary, and then take the Fourier transform of each segment, Gabor 1946.
Followed by other TFRs, which differed from each other by the selection of the windowing function
The F.T. was obviously inadequate for analyzing non-stationary signals, with time varying spectra
Dennis Gabor who was interested in representing a communication signal using oscillatory basis functions, was the first one to modify FT in the form of STFT. The basic idea was segmenting the signal by using a time localized window, and performing the Fourier analysis on each segment.
If each segment was taken short enough, then they could be considered as stationary, and hence obtained the first TFR.
Shortly after in 1947 Ville and Wigner developed an alternative, quadratic TFR, Wigner-Ville transform
Late 1940s to mid 1970s saw many new TFR, each differing from each other only by the selection of windowing function.
One major drawback: all use the same window for the entire signal…!

16. Short Time Fourier Transform (STFT)
Choose a window function of finite length
Place the window on top of the signal at t=0
Truncate the signal using this window
Compute the FT of the truncated signal, save.
Incrementally slide the window to the right
Go to step 3, until window reaches the end of the signal
For each time location where the window is centered, we obtain a different FT
Hence, each FT provides the spectral information of a separate time-slice of the signal, providing simultaneous time and frequency information

17. STFT Time parameter Frequency parameter Signal to be analyzed
FT Kernel
(basis function)
STFT of signal x(t):
Computed for each
window centered at t=t’
Windowing
function
Windowing function
centered at t=t’

18. STFT
t’= t’=-2
t’=4 t’=8

19. STFT at Work

20. STFT At Work

21. STFT At Work

22. STFT
STFT provides the time information by computing a different FTs for consecutive time intervals, and then putting them together
Time-Frequency Representation (TFR)
Maps 1-D time domain signals to 2-D time-frequency signals
Consecutive time intervals of the signal are obtained by truncating the signal using a sliding windowing function
How to choose the windowing function?
What shape? Rectangular, Gaussian, Elliptic…?
How wide?
Wider window require less time steps  low time resolution
Also, window should be narrow enough to make sure that the portion of the signal falling within the window is stationary
Can we choose an arbitrarily narrow window…?

23. Selection of STFT Window
Two extreme cases:
W(t) infinitely long:  STFT turns into FT, providing excellent frequency information (good frequency resolution), but no time information
W(t) infinitely short:
 STFT then gives the time signal back, with a phase factor. Excellent time information (good time resolution), but no frequency information
Wide analysis window poor time resolution, good frequency resolution
Narrow analysis windowgood time resolution, poor frequency resolution
Once the window is chosen, the resolution is set for both time and frequency.

24. Heisenberg Principle
Time resolution: How well two spikes in time can be separated from each other in the transform domain
Frequency resolution: How well two spectral components can be separated from each other in the transform domain
Both time and frequency resolutions cannot be arbitrarily high!!!  We cannot precisely know at what time instance a frequency component is located. We can only know what interval of frequencies are present in which time intervals

25. The Wavelet Transform
Overcomes the preset resolution problem of the STFT by using a variable length window
Analysis windows of different lengths are used for different frequencies:
Analysis of high frequencies Use narrower windows for better time resolution
Analysis of low frequencies  Use wider windows for better frequency resolution
This works well, if the signal to be analyzed mainly consists of slowly varying characteristics with occasional short high frequency bursts.
Heisenberg principle still holds!!!
The function used to window the signal is called the wavelet

26. The Wavelet Transform Scale = 1/frequency A normalization
Translation parameter, measure of time
Scale parameter, measure of frequency
A normalization
constant
Signal to be analyzed
Continuous wavelet transform of the signal x(t) using the analysis wavelet (.)
The mother wavelet. All kernels are obtained by translating (shifting) and/or scaling the mother wavelet
Scale = 1/frequency

27. WT at Work
Low frequency (large scale)
High frequency (small scale)

28. WT at Work

29. WT at Work

30. WT at Work

31. Matlab Demos on CWT

32. Discrete Wavelet Transform
CWT computed by computers is really not CWT, it is a discretized version of the CWT.
The resolution of the time-frequency grid can be controlled (within Heisenberg’s inequality), can be controlled by time and scale step sizes.
Often this results in a very redundant representation
How to discretize the continuous time-frequency plane, so that the representation is non-redundant?
Sample the time-frequency plane on a dyadic (octave) grid

33. Discrete Wavelet Transform
Dyadic sampling of the time –frequency plane results in a very efficient algorithm for computing DWT:
Subband coding using multiresolution analysis
Dyadic sampling and multiresolution is achieved through a series of filtering and up/down sampling operations
x[n] H
y[n]

34. Discrete Wavelet Transform ImplementationG H 2 +
x[n]
Decomposition
Reconstruction ~
Half band high pass filter
Half band low pass filter 2
Down-sampling
Up-sampling G H 2
2-level DWT decomposition. The decomposition can be continues as long as there are enough samples for down-sampling.

35. DWT - Demystified g[n] h[n] g[n] h[n] g[n] h[n] |H(jw)| w /2 -/2 a1
Length: 512
B: 0 ~ 
|H(jw)| w
/2
-/2
g[n]
h[n]
Length: 256
B: 0 ~ /2 Hz
Length: 256
B: /2 ~  Hz 2 2 a1
|G(jw)|
d1: Level 1 DWT
Coeff.
g[n]
h[n]
Length: 128
B: 0 ~  /4 Hz w
Length: 128
B: /4 ~ /2 Hz 2 2 -
-/2
/2  a2
d2: Level 2 DWT
Coeff.
g[n]
h[n] 2 2
Length: 64
B: 0 ~ /8 Hz
Length: 64
B: /8 ~ /4 Hz
…a3….
d3: Level 3 DWT
Coeff.
Level 3 approximation
Coefficients

36. Implementation of DWT on MATLAB
Choose wavelet
and number
of levels
Load signal
Hit Analyze
button
s=a5+d5+…+d1
Approx. coef.
at level 5
Level 1 coeff.
Highest freq.
(Wavedemo_signal1)

37. Applications of
Wavelets

39. Applications of Wavelets
Compression
De-noising
Feature Extraction
Discontinuity Detection
Distribution Estimation
Data analysis
Biological data
NDE data
Financial data

40. Compression
DWT is commonly used for compression, since most DWT are very small, can be zeroed-out!

41. Compression

42. Compression

43. ECG- Compression

44. Denoising Implementation in Matlab
First, analyze the signal with appropriate wavelets
Hit
Denoise
(Noisy Doppler)

45. Denoising Using Matlab
Choose thresholding
method
Choose noise type
Choose thrsholds
Hit
Denoise

46. Denosing Using Matlab

47. Discontinuity Detection
(microdisc.mat)

48. Discontinuity Detection with CWT
(microdisc.mat)

49. Application Overview Data Compression Wavelet Shrinkage Denoising
Source and Channel Coding
Biomedical Engineering
EEG, ECG, EMG, etc analysis
MRI
Nondestructive Evaluation
Ultrasonic data analysis for nuclear power plant pipe inspections
Eddy current analysis for gas pipeline inspections
Numerical Solution of PDEs
Study of Distant Universes
Galaxies form hierarchical structures at different scales
DATA COMPRESSION: Quest for wavelets better suited to specific application, yield better compression ratios. If a perfectly matching wavelet can be found for the exact time and frequency characteristics of the signal, the signal can be greatly compressed. Recall that wavelets are basis functions to which we decompose the signal into. If we choose these building blocks efficiently, we can reconstruct the signal with only a few of them.
DENOISING: Noise has a bandwidth of certain frequency band, typically at finer scales. Discarding or reducing the amplitude of the coefficients corresponding to those scales will remove / reduce the noise
SOURCE & CHANNEL CODING: Source coding requires a compact representation of the information, and channel coding requires incorporating controlled amounts of redundancy into the representation to reduce the ill effects of the channel noise.
BIOMEDICAL ENG: Almost all biological signals are of non-stationary nature. Wavelets have been used for analyzing EEG for Alzheimer detection, of ECG for analyzing heart rate variablity, MRI analysis etc.
NDE: UT signals for nuclear power plant inspection, and eddy current signals for gas pipeline inspection have been analyzed using wavelets
PDE: Discretized by using wavelets as basis functions, and then soled numerically
DISTANT UNIVERSES: Distribution of galaxies forms hierarchical structures at various scales. Albert Bijaoui.

50. Application Overview Wavelet Networks
Real time learning of unknown functions
Learning from sparse data
Turbulence Analysis
Analysis of turbulent flow of low viscosity fluids flowing at high speeds
Topographic Data Analysis
Analysis of geo-topographic data for reconnaissance / object identification
Fractals
Daubechies wavelets: Perfect fit for analyzing fractals
Financial Analysis
Time series analysis for stock market predictions
WAVELET NETWORKS: Excellent time and frequency localization properties of wavelets as basis functions replace the Gaussian functions of RBF networks.
W.networks are particularly useful in analyzing sparse data: use a higher resolution when the data space is dense, and use lower resolution when the space is sparse. Bakshi and Stephanopoulos)
Bernard, Mallat and Slotine: Wavelet interpolation network for real time learning of unknown functions.
TURBULENT ANALYSIS: Marie Farge-> solve Navier Stokes equations using wavelet based numerical techniques.
FRACTALS: Certain wavelets, such as Daubechies, have a fractal (self similar) structure, and hence provide a natural tool for analyzing fractals when combined by multiresolution formulation (Farge, A. Oppenheim)

51. History Repeats Itself…
1807, J.B. Fourier:
All periodic functions can be expressed as a weighted sum of trigonometric function
Denied publication by Lagrange, Legendre and Laplace
1822: Fourier’s work is finally published …
1965, Cooley & Tukey: Fast Fourier Transform
143 years

52. History Repeats Itself: Morlet’s Story
1946, Gabor: STFT analysis:
high frequency components using a narrow window, or
low frequency components using a wide window, but not both
Late 1970s, Morlet’s (geophysical engineer) problem:
Time - frequency analysis of signals with high frequency components for short time spans and low frequency components with long time spans
STFT can do one or the other, but not both Solution: Use different windowing functions for sections of the signal with different frequency content
Windows to be generated from dilation / compression of prototype small, oscillatory signals  wavelets
Criticism for lack of mathematical rigor !!!
Early 1980s, Grossman (theoretical physicist): Formalize the transform and devise the inverse transformation  First wavelet transform !
Rediscovery of Alberto Calderon’s 1964 work on harmonic analysis
Late 1970s, geophysical engineer J. Morlet was faced by the following problem:****
Morlet came up with the idea of using a different windowing functions for different sections of the signal corresponding to different frequencies. He proposed using a narrow window for high frequencies and a wide window for low frequencies. Furthermore, all these windows were generated by compression or dilation of the same prototype function, the Gaussian.
Gaussian shaped window was ideal since Gaussians have compact support in both time and frequency. Due to compact support (small) and oscillatory (wave) nature of these windows, Morlet called these basis functions as wavelets of constant shape.
Due to criticism for lack of mathematical rigor, he looked for help from the Math community. Met A. Grossman, a theoretical physicist of quantum mechanics. Grossman helped him to formalize the transformation, and devised an inverse formula.
Little did they know, however, their wavelet transform was merely a rediscovery, perhaps slightly different interpretation of Alberto Calderon’s work on harmonic analysis. Plagiarism? No. Ignorance? Maybe.

53. 1980s
1984, Yeves Meyer :
Similarity between Morlet’s and Colderon’s work, 1984
Redundancy in Morlet’s choice of basis functions
1985, Orthogonal wavelet basis functions with better time and frequency localization
Rediscovery of J.O. Stromberg’s 1980 work the same basis functions (also a harmonic analyst)
Yet re-rediscovery of Alfred Haar’s work on orthogonal basis functions, 1909 (!).
Simplest known orthonormal wavelets
Yeves Meyer, a French mathematician was fascinated by the elegant nonstationary analysis scheme of Morlet and Colderon.
Started working on developing better wavelets with improved time-frequency localization properties.
Constructed orth. Wavelet functions in 1985, only to realize that the same functions were already discovered 5 years ago by J.O. Stromberg.
Yet, neither of them were the first to discover of orthogonal wavelet basis functions. That honor goes all the way back to 1909, to the German mathematician, Alfred Haar.
Though, Haar’s wavelets were of little practical use, due to their poor frequency localization
AS a twist off history, it was later discovered that Haar’s work on developing orthonormal basis functions was independently elaborated in 1930s by Paul Levey studying random signals of Brownian motion, and by Littlewood, working on localizing the contributing energies of a function.

54. Transition to the Discrete Signal Analysis
Ingrid Daubechies:
Discretization of time and scale parameters of the wavelet transform
Wavelet frames, 1986
Orthonormal bases of compactly supported wavelets (Daubechies wavelets), 1988
Liberty in the choice of basis functions at the expense of redundancy
Stephane Mallat:
Multiresolution analysis w/ Meyer, 1986
Ph.D. dissertation, 1988
Discrete wavelet transform
Cascade algorithm for computing DWT
In the mean time, I. Daubechies, a former graduate student of Grossman at the Free University of Brussels, developed the wavelet frames for discretization of time and scale parameters of the WT.
Along with Mallat, she is credited for developing the transition from continuous to discrete signal analysis.
In particular, Stephane Mallat, then a graduate student at UPenn in 1986, developed the idea of multiresolution analysis for DWTR with Meyer, which later became his Ph.D. dissertation in 1988.

55. …However…
Decomposition of a discrete into dyadic frequencies (MRA) , known to EEs under the name of “Quadrature Mirror Filters”, Croisier, Esteban and Galand, 1976 (!)

56. Transition to the Discrete Signal Analysis
Martin Vetterli & Jelena Kovacevic
Wavelets and filter banks, 1986
Perfect reconstruction of signals using FIR filter banks, 1988
Subband coding
Multidimensional filter banks, 1992
Mallat’s idea was decomposing a discrete signal into its dyadic frequency bands by a series of lowpass and highpass filters to compute its DWT from the approximations at various scales.
Martin Vetterli and Jelena Kovacevic, elaborated on this idea to compute DWT from filter banks, and devised the idea of multiresolution filter banks. They also showed perfect reconstruction of the signal using filter banks.
But once again, a twist of history showed itself: The idea of using filter banks to approximate discrete signals was all too familiar to electrical engineers for a long time, under the name Quadrature Mirror Filtering. IN fact , these ideas have been around since 1976, when Croiser, Esteban and Galand first constructed filter banks.
Mallat’s work constituted a natural extension of time localization to the well established idea of frequency localization of QMFs and subband coding.
With the development of Daubechies’s orthonormal bases of compactly supported wavelts, the foundations of the modern wavelet theory were (re) laid.

57. 1990s Equivalence of QMF and MRA, Albert Cohen, 1990
Compactly supported biorthogonal wavelets, Cohen, Daubechies, J. Feauveau, 1993
Wavelet packets, Coifman, Meyer, and Wickerhauser, 1996
Zero Tree Coding, Schapiro 1993 ~ 1999
Search for new wavelets with better time and frequency localization properties.
Super-wavelets
Matching Pursuit, Mallat, 1993 ~ 1999

58. New & Noteworthy Zero crossing representation signal classification
computer vision
data compression
denoising
Super wavelet
Linear combination of known basic wavelets
Zero Tree Coding, Schapiro
Matching Pursuit , Mallat
Using a library of basis functions for decomposition
New MPEG standard
ZERO CROSSING: The signals can be represented by the multiscale zero crossings. The signal is then uniquely characterized by its zero crossings, and this representation is insensitive to shifting. Signal reconstruction is very efficient and fast. Mallat, Afzal (99)
SUPER WAVELETS: Two wavelets are better than one wavelet!!!. Linear combinations of known wavelets have been shown to possess better localization properties then each of the wavelet…Synergy !
ZERO TREE CODING:Used for image compression, taking the relative importance of frequencies into account.
MATCHING PURSUIT: Many wavelets are better than one wavelet !!! Mallat proposed using a different wavelet for different levels of the decomposition to make better use of the localization properties of individual wavelets. This allows very efficient compression of particularly images.

59. The Story of Wavelets