Wavelet – An Introduction 2018/11/17 Wavelet – An Introduction Multi-resolution decomposition Wavelet basis Dilation equations Wavelet-based image coding Embedded zero-tree Wavelet-based video coding Motion compensated temporal filtering 2018/11/17Wavelet, MC 2009 MC 2007

Wavelet Transformation 2018/11/17 Wavelet Transformation The wavelet transform describes a multi-resolution decomposition process in terms of expansion of an image onto a set of wavelet basis  functions. Wavelet transform vs. Short-term Fourier transform Wavelet basis function: well-localized in both space and frequency Dilation equations  : Two multi-resolution decomposition schemes: pyramid  and subband. Wavelet compression process Take wavelet transform of a digitized image. Allocate bits among the wavelet coefficients. 2018/11/17Wavelet, MC 2009 MC 2007

Wavelet-Based Image Coding 2018/11/17 Wavelet-Based Image Coding Embedded Zero-tree Wavelet coding (EZW) by J. M. Shapiro (1993) Set Partitioning in Hierarchical Trees (SPIHT) by A. Said and W. A. Pearlman (1996) http://www.cipr.rpi.edu/research/SPIHT/ Layered Zero Coding (LZC) by D. Taubman and A. Zakhor Rate-Distortion optimized Embedded coding (RDE) by J. Li and S.-M. Lei Embedded Block Coding with Optimized Truncation (EBCOT) by D. Taubman An important part of JPEG2000 Part 1 2018/11/17Wavelet, MC 2009 MC 2007

1-D Pyramidal Decomposition 2018/11/17 1-D Pyramidal Decomposition 2018/11/17Wavelet, MC 2009 MC 2007

Multi-resolution Decomposition 2018/11/17 Multi-resolution Decomposition A multiresolution analysis of the space L2 (R) consists of a sequence of nested subspaces … V0  V1  …  Vn  … that satisfies certain self-similarity relations in spatial and frequency, as well as completeness and regularity relations. Self-similarity in spatial demands that each subspace Vk is invariant under shifts by integer multiples of 2-k. That is, for each f  Vk, m   there is a g  Vk with f (x) = g(x+m2-k). Self-similarity in frequency demands that all subspaces Vk  Vl , k < l, are spatial-scaled versions of each other, with scaling respectively dilation factor 2l-k. That is, for each f  Vk there is a g  Vl with f (x) = g(x2l-k). LH1 HL1 HH1 HL2 LL1 LL2  2018/11/17Wavelet, MC 2009 MC 2007

Multi-resolution Analysis and the Scaling Functions 2018/11/17 Multi-resolution Analysis and the Scaling Functions Approximated by scaling functions Harr (1907) scaling functions Dilation version of the “mother” scaling function: In fact, we can obtain scaling functions at different resolutions in a manner similar to the procedure used for wavelets, In Harr 2018/11/17Wavelet, MC 2009 MC 2007

Wavelet Approximations by Harr (1) 2018/11/17 Wavelet Approximations by Harr (1) 2018/11/17Wavelet, MC 2009 MC 2007

Wavelet Approximations (2) 2018/11/17 Wavelet Approximations (2) 2018/11/17Wavelet, MC 2009 MC 2007

Short-term Fourier Transform (STFT) 2018/11/17 Short-term Fourier Transform (STFT) If we have a very non-stationary signal, we would like to know not only the frequency components but when in time the particular frequency components occurred. Short-term Fourier Transform We break the time signal f(t) into pieces of length T and apply Fourier analysis to each pieces. Thus, we obtain an analysis that is a function of both time and frequency. Boundary effect, (Problem #1) To reduce the boundary effect, we window each piece before we take the FT. If the window shape is given by g(t), the STFT is given by If the window function g(t) is Gaussian, the STFT is called the Gabor transform. The problem with the STFT is the fixed window size. The window size should contain at least one cycle of the low-frequency component, however, it can not accurately localize the high frequency spurt. Uncertainty principle: If we wish to have finer resolution in time, we end up with a lower resolution in the frequency domain. 2018/11/17Wavelet, MC 2009 MC 2007

2018/11/17 2018/11/17Wavelet, MC 2009 MC 2007

STFT basis functions WT basis functions 2018/11/17 STFT basis functions WT basis functions Small frequency changes in the FT will produce changes everywhere in the time domain. Wavelets are local in both frequency/scale (via dilations) and in spatial/time (via translations). 2018/11/17Wavelet, MC 2009 MC 2007

Wavelets A wavelet of the mother function (t), 2018/11/17 Wavelets A wavelet of the mother function (t), We scale a function f(t) by replacing t with t/a and translate a function to the right or left by an amount b by replacing t by t-b or t+b. If we want the scaled function to have the norm as the original function, we need to multiply it by Wavelet coefficients of a function f(t), We can recover the function f(t) by Ondelette in French, by Morlet 2018/11/17Wavelet, MC 2009 MC 2007

2018/11/17 Properties of Wavelet C should be finite, so (0) = 0. Since (0) is the average value of (t); therefore a requirement on the mother wavelet is that it has zero mean. We would also like the wavelets to have finite energy. Using Parseval’s relationship, we can write this requirement as For this to happen, | ()|2 have to decay as  goes to infinite. This requirements mean the energy in () is concentrated in a narrow frequency band. Discrete version: 2018/11/17Wavelet, MC 2009 MC 2007

2018/11/17 Dilation Equations The scaling function itself can be represented by its dilations at a high resolution: Multi-resolution analysis (MRA) equations: by substituting Harr scaling function: Orthonormal transforms 2018/11/17Wavelet, MC 2009 MC 2007

(x) = ½  (2x) +  (2x-1) + ½  (2x-2) 2018/11/17 (x) = ½  (2x) +  (2x-1) + ½  (2x-2) 2 1 2018/11/17Wavelet, MC 2009 MC 2007

Daubechies Wavelet DAUB4 2018/11/17 Daubechies Wavelet DAUB4 4 coefficients 2018/11/17Wavelet, MC 2009 MC 2007

Daubechies Wavelet DAUBx (2) 2018/11/17 Daubechies Wavelet DAUBx (2) Its derivative exists only almost everywhere (fails on points p/2n) Left differentiable, but not right differential. The 130th largest wavelet coefficient has an amplitude less than 10-5 of the largest coefficient among 1024 coefficients. Compact wavelets are better for lower accuracy approximation and for functions with discontinuities (such as edges), while small wavelets are better for achieving high numerical accuracy. BAUB6, p=3 moment vanishing 2018/11/17Wavelet, MC 2009 MC 2007

Daubechies 9-tap/7-tap Filter Cohen-Daubechies-Feauveau (CDF) 2018/11/17 Daubechies 9-tap/7-tap Filter Cohen-Daubechies-Feauveau (CDF) Biorthogonal wavelet: is a wavelet where the associated wavelet transform is invertible but not necessarily orthogonal. allows more degrees of freedoms than orthogonal wavelets. decomposed into a set of low-pass samples and a set of high-pass samples. Biorthogonal Daubechies 9-tap/7-tap filter (irreversible transform) Le Gall 5-tap/3-tap filter (reversible transform) it introduces quantization noise that depends on the precision of the decoder analysis filter 5+4 4+3 2018/11/17Wavelet, MC 2009 MC 2007

Subband Decomposition of an NM Images 2018/11/17 Subband Decomposition of an NM Images 2018/11/17Wavelet, MC 2009 MC 2007

Embedded Zerotree Wavelet (1) 2018/11/17 Embedded Zerotree Wavelet (1) Embedded coding The bits are ordered in their importance The encoding improves as more bits are transmitted. The codec can terminate at any point thereby allowing a target rate or distortion metric to be met exactly. Assumption : most energy is compacted into lower bands The coefficients closer to the root have higher magnitudes than coefficients further from the root. LH1 HL1 HH1 HL2 LL1 LL2  2018/11/17Wavelet, MC 2009 MC 2007

2018/11/17 EZW (2) Embedded Image Coding using Zerotrees of wavelet coefficients: DWT, prediction of the absence of significant information, entropy-coded successive-approximate quantization and lossless data compression. Zerotree structure If a coefficient has a magnitude less than a given threshold, all its descendants might have magnitudes less than that threshold. Significant coefficient at level T If |x| < T Symbols Zerotree root (ZR) all descenfents (including itself) are insignificant Isolated zero (IZ) the coefficient is insignificant but has some significant descendant Significant positive (SP) Significant negative (SN) 2018/11/17Wavelet, MC 2009 MC 2007

EZW (3) EZW is a multiple-pass algorithm 2018/11/17 EZW (3) EZW is a multiple-pass algorithm Each pass consists of 2 steps Significance map encoding (dominant pass) Refinement (subordinate pass) Two separate lists are maintained Dominant list coordinates of the coefficients that have not been found to be significant in the relative order Subordinate list magnitudes of the coefficients that have been found to be significant Successive Approximation During a subordinate pass the width of the effective quantizer step size (uncertainty interval) is cut in half 1 : the value falls in the upper half of the uncertainty interval 0 : the value falls in the lower half The threshold is halved before each dominant pass Scanning order 2018/11/17Wavelet, MC 2009 MC 2007

Flow Chart for Encoding a Coefficient 2018/11/17 Flow Chart for Encoding a Coefficient 2018/11/17Wavelet, MC 2009 MC 2007

EZW (4) Example 1 Initial threshold cmax will be in [T0, 2T0) 2018/11/17 EZW (4) Example 1 Initial threshold cmax will be in [T0, 2T0) 26 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 Dominant list Subordinate list Subband Coeff. Symbol LL2 26 SP HL2 6 ZR LH2 -7 ZR HH2 7 ZR “0” : [16, 24) “1” : [24, 32) Coef Symbol Recon 26 1 28 2018/11/17Wavelet, MC 2009 MC 2007

EZW (5) Example 1 (cont.) Pass 1 Pass 2 T1 = T0/2 = 8 T2 = T1/2 = 4 2018/11/17 EZW (5) Example 1 (cont.) Pass 1 T1 = T0/2 = 8 Pass 2 T2 = T1/2 = 4 Dominant list Subordinate list Subband Coeff. Symbol HL2 6 IZ LH2 -7 ZR HH2 7 ZR HL1 13 SP HL1 10 SP HL1 6 IZ HL1 4 IZ [8, 16) [16, 24) [24, 32) “0” : [8, 12) [16,20) [24,28) “1” : [12, 16) [20,24) [28,32) X 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 Coef Symbol Recon 26 10 2826 13 1 14 10 0 10 Subband Coeff. Symbol HL2 6 SP LH2 -7 SN HH2 7 SP HL1 6 SP HL1 4 SP LH1 4 SP LH1 -4 SN LH1 2 IZ LH1 -2 IZ HH1 4 SP HH1 -3 IZ HH1 -2 IZ HH1 0 IZ Coef Symbol Recon 26 10 0 2625 13 1 0 1413 10 0 0 109 6 0 5 -7 1 -7 7 1 7 4 0 5 -4 0 -5 X 6 X X -7 7 6 4 4 -4 4 -3 2 -2 -2 0 2018/11/17Wavelet, MC 2009 MC 2007

EZW (6) Example 2 63 -34 49 10 7 13 -12 7 Initial threshold 63 1 56 2018/11/17 EZW (6) Example 2 Initial threshold Dominant list Subband Coeff. Symbol LL3 63 SP HL3 -34 SN LH3 -31 IZ HH3 23 ZR HL2 49 SP HL2 10 ZR HL2 14 ZR HL2 -13 ZR LH2 15 ZR LH2 14 IZ LH2 -9 ZR LH2 -7 ZR HL1 7 ZR HL1 13 ZR HL1 3 ZR HL1 4 ZR LH1 -1 ZR LH1 47 SP LH1 -3 ZR LH1 -2 ZR Subordinate list “0” : [32, 48) “1” : [48, 64) 63 -34 49 10 7 13 -12 7 -31 23 14 -13 3 4 6 -1 15 14 3 -12 5 -7 3 9 9 -7 -14 8 4 -2 3 2 -5 9 -1 47 4 6 -2 2 3 0 -3 2 3 -2 0 4 2 -3 6 -4 3 6 3 6 5 11 5 6 0 3 -4 4 Coef Symbol Recon 63 1 56 34 0 40 49 1 56 47 0 40 2018/11/17Wavelet, MC 2009 MC 2007

EZW (7) Example 2 (2nd pass): 63 -34 49 10 7 13 -12 7 2018/11/17 EZW (7) Example 2 (2nd pass): Threshold T1 = 16 Dominant list Subordinate list 63 -34 49 10 7 13 -12 7 -31 23 14 -13 3 4 6 -1 15 14 3 -12 5 -7 3 9 9 -7 -14 8 4 -2 3 2 -5 9 -1 47 4 6 -2 2 3 0 -3 2 3 -2 0 4 2 -3 6 -4 3 6 3 6 5 11 5 6 0 3 -4 4 [16, 32) [32, 48) [48,64) “0”: [16, 24) [32,40) [48,56) “1”: [24, 32) [40,48) [56,64) Subband Coeff. Symbol LH3 -31 SN HH3 23 SP HL2 10 ZR HL2 14 ZR HL2 -13 ZR LH2 15 ZR LH2 14 ZR LH2 -9 ZR LH2 -7 ZR HH2 3 ZR HH2 -12 ZR HH2 -14 ZR HH2 8 ZR Coef Symbol Recon 63 11 5660 34 00 4036 49 10 5652 47 01 4044 -31 1 28 23 0 20 2018/11/17Wavelet, MC 2009 MC 2007

SPIHT (1) Set Partitioning in Hierarchical Trees Partitioning types 2018/11/17 SPIHT (1) Set Partitioning in Hierarchical Trees Use a partitioning of the spatial orientation trees in a manner that tends to keep insignificant coefficients together in a larger subsets Partitioning decision -> binary decision Provide a significance map encoding that is more efficient than EZW Each pass consists of 2 steps Significance map encoding (set partitioning and ordering step) Refinement Partitioning types O(i, j) Set of coordinates of the offsprings at (i, j) D(i, j) Set of all descendants of the coefficients at (i, j) H Set of all root nodes L(i,j) L(i,j) = D(i, j) - O(i, j) LIS Each node can either have 4 or none 2018/11/17Wavelet, MC 2009 MC 2007

SPIHT (2) Significant Three lists are maintained At each pass 2018/11/17 SPIHT (2) Significant A set D(i, j) or L(i,j) is significant if any coefficient in the set has a magnitude greater than the threshold Three lists are maintained LIP (list of insignificant pixels) Is Initialized with the set H LSP (list of significant pixels) Is initially empty LIS (list of insignificant sets) Contain the coordinates of the roots of types D or L At each pass Significant map encoding step Process the members of LIP If the coefficient is significant  transmit (1 + sign), move to LSP; Otherwise  transmit 0 Then process the members of LIS Insignificant set  0 Refinement step Process the elements of LSP 2018/11/17Wavelet, MC 2009 MC 2007

SPIHT (3) An example Initialize 1st pass (T = 16) T = 16 2018/11/17 SPIHT (3) An example Initialize T = 16 1st pass (T = 16) LIP {(0,0)  26, (0,1)  6, (1,0)  -7, (1,1)  7} LIS {(0,1)D, (1,0)D, (1,1)D} LSP { } LIP 26  1 + 0  move to LSP 6  0 -7  0 7  0 LIS (0,1)D  0 (1,0)D  0 (1,1)D  0 26 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 1 + sign 10 0 0 0 0 0 0 2018/11/17Wavelet, MC 2009 MC 2007

SPIHT(4) An example (cont.) 2nd pass (T = 8) 26 6 13 10 2018/11/17 SPIHT(4) An example (cont.) 2nd pass (T = 8) n--  n = 3, T = 8 LIP { (0,1)  6, (1,0)  -7, (1,1)  7} LIS {(0,1)D, (1,0)D, (1,1)D} LSP { (0,0)  26 } LIP 6  0 -7  0 7  0 LIS (0,1)D  1 13  1 + 0  move to LSP 10  1 + 0  move to LSP 6  0  move to LIP 4  0  move to LIP (1,0)D  0 (1,1)D  0 LSP 26 = (11010)2  1 26 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 0 0 0 10 10 0 0 0 0 1 2018/11/17Wavelet, MC 2009 MC 2007

SPIHT (5) An example (cont.) 26 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 2018/11/17 SPIHT (5) An example (cont.) 3rd pass (T = 4) n--  n = 2, T = 4 LIP {(0,1)  6, (1,0)  -7, (1,1)  7, (1,2)  6, (1,3)  4} LIS {(1,0)D, (1,1)D} LSP {(0,0)  26, (0,2)  13, (0,3)  10} LIP 6  1 + 0  move to LSP -7  1 + 1  move to LSP 7  1 + 0  move to LSP 4  1 + 0  move to LSP LIS (1,0)D  1 -4  1 + 1  move to LSP 2  0  move to LIP -2  0  move to LIP (1,1)D  1 -3  0  move to LIP 0  0  move to LIP LSP 26  0, 13  1, 10  0 26 6 13 10 -7 7 6 4 4 -4 4 -3 2 -2 -2 0 10 11 10 10 10 10 11 0 0 10 0 0 0 0 1 0 LIP {(3,0)2, (3,1)-2, (2,3)-3, (3,2)-2, (3,3)0} LIS { } LSP { (0,0)  26, (0,2)  13, (0,3)  10, (0,1)  6, (1,0)  -7, (1,1)  7, (1,2)  6, (1,3)  4, (2,0)  4, (2,1)  -4, (2,2)  4} 2018/11/17Wavelet, MC 2009 MC 2007

Interframe Wavelet Coding 2018/11/17 Interframe Wavelet Coding Motion-Compensated 3-D Subband Coding of Video IEEE TRANSACTIONS ON IMAGE PROCESSING VOL. 8, NO. 2, FEB 1999 Seung-Jong Choi and John W. Woods Adaptive Motion-Compensated Wavelet Filtering for Image Sequence Coding VOL. 6, NO. 6, JUNE 1997 Jean-Pierre Leduc, Jean-Marc Odobez, and Claude Labit Three-dimensional subband coding with motion compensation VOL. 3, SEPT 1993 J.-R. Ohm MC 2007

Outline Wavelet Transform-based Coding 2018/11/17 Outline Wavelet Transform-based Coding Motion Compensated Temporal Filtering (MCTF) Block Diagram of Interframe Wavelet Coding Experimental Results Conclusions 2018/11/17Wavelet, MC 2009 MC 2007

Wavelet Transform-based Coding 2018/11/17 Wavelet Transform-based Coding Discrete wavelet transform (DWT) decomposes a non-stationary signal into multi-scaled subbands Wavelet transform Quantization (Embedded) Entropy coding (Zero-tree) 2018/11/17Wavelet, MC 2009 MC 2007

Motion Compensated Temporal Filtering (MCTF) 2018/11/17 Motion Compensated Temporal Filtering (MCTF) The spatio-temporal scenes will be segmented according to motion build the trajectories by a spatio-temporal segmentation algorithm and a related trajectory construction Temporal filtering will be applied along the assumed motion trajectories 2018/11/17Wavelet, MC 2009 MC 2007

Motion Compensated Temporal Filtering (MCTF) 2018/11/17 Motion Compensated Temporal Filtering (MCTF) (Haar filter [1, 1] [1, -1]) (Haar filter [1, 1] [1, -1]) with motion compensation 2018/11/17Wavelet, MC 2009 MC 2007

Motion Compensated 3D Wavelet Coding by Lifting Filter Implementation 2018/11/17 Motion Compensated 3D Wavelet Coding by Lifting Filter Implementation The “highpass” pictures are very much the same as difference frame in a frame prediction can be achieved in combination with motion compensation A “lifting filter” structure is an efficient implementation of wavelet decomposition Consisting of a “prediction step” (highpass filter) and an “update step” (lowpass filter) 2018/11/17Wavelet, MC 2009 MC 2007

An MCTF Four-level Decomposition (GOP=16 frames) 2018/11/17 An MCTF Four-level Decomposition (GOP=16 frames) Divides consecutive frames into groups (GOP=16) Perform MC filtering on pairs of frames to produce temporal low and high frames. Temporal low frames decomposed again with the same MC filtering 2018/11/17Wavelet, MC 2009 MC 2007

A Three-level 3D motion Compensated Decomposition 2018/11/17 A Three-level 3D motion Compensated Decomposition Pictures at leaves of tree are subject to spatial wavelet decomposition 2018/11/17Wavelet, MC 2009 MC 2007

Block Diagram of Interframe Wavelet 2018/11/17 Block Diagram of Interframe Wavelet Motion Compensated Temporal Filtering + Zero-Tree Coding John Woods, Jens Ohm 2018/11/17Wavelet, MC 2009 MC 2007

Examples of comparison between MCTF and AVC (GOP=16 frames) 2018/11/17 Experimental Results Examples of comparison between MCTF and AVC (GOP=16 frames) 2018/11/17Wavelet, MC 2009 MC 2007

Conclusions Scalability: spatial, temporal, SNR and complexity 2018/11/17 Conclusions Scalability: spatial, temporal, SNR and complexity (Wavelet coding which inherently possess scalability) Providing flexible scalability of a single bitstream (encoding once, decoding multiple times) Error resilience (a non-recursive coder) 2018/11/17Wavelet, MC 2009 MC 2007