1. Fast Lossless Multi-Resolution Motion Estimation for Scalable Wavelet Video Coding
Yu Liu and King Ngi Ngan
Department of Electronic Engineering
The Chinese University of Hong Kong
ISCAS2006, May 21-24, Island of Kos, Greece

2. Outline Introduction Background Proposed Algorithm
Experimental Results
Conclusion

3. Introduction Motion Estimation in Critically-Sampled Wavelet Domain
Pro: basically free form the blocking effects
Con: inefficient in high bands
Motion Estimation in Shift-Invariant Wavelet Domain
Pro: perform ME more precisely and efficiently
Con: computational complexity
e.g. low-band-shift (LBS) and complete-to-overcomplete DWT (CODWT)
ME/MC in wavelet domain has received much attention due to its superior performance by comparing to conventional ME/MC in spatial domain.
ME/MC in wavelet domain is basically free form the blocking effects due to the global nature of wavelet transform.
However, ME/MC in critically sampled wavelet domain is very inefficient in high bands because of the shift-variant property of wavelet decomposition.
To overcome the shift-variant property of wavelet transform, LBS and CODWT are proposed.
These methods avoid the shift-variant property of the wavelet transform and perform ME more precisely and efficiently.
However, a major disadvantage of these methods is the computational complexity which mainly comes from full search algorithm.

4. Background Motion Estimation in Shift-Invariant Wavelet Domain (1)
Two Level Shift-Invariant Wavelet Decomposition by using Low-Band-Shift (LBS) or Complete-to-Overcomplete DWT (CODWT)
First, let’s look at some background. This is an example of two level SIDWT by using LBS or CODWT.
For LBS method, these other shifted subbands are obtained by shifting the LL Band of each level, then followed by normal wavelet transform.
For CODWT method, these other shifted subbands are obtained by making a direct link between the critically-sampled subbands and shift-invariant subbands using the complete-to-overcomplete prediction filters.

5. Background Motion Estimation in Shift-Invariant Wavelet Domain (2)
Generation of Wavelet Blocks
For ME in shift-invariant wavelet domain, the coefficients of each wavelet tree rooted in the lowest subband are rearranged to form a wavelet block. The purpose of the wavelet block is to provide a direct association between the wavelet coefficients and what they represent spatially in the image. Related coefficients at all scales and orientations are included in each wavelet block.
The v-pixel-shifted or {dx,dy}-pixel-shifted coefficient of the pth wavelet block of the reference frame t’ can be represented by
(dx%2l, dy%2l) means which shifted subband it is, (i+[dx/2l],j+[dy/2l]) means where the pixel is located for the shifted subband.
l denotes decomposition level, k denotes subband type, such as LL/HL/LH/HH subband type.
The coefficient of the pth wavelet block of the current frame t can be represented by
The difference between these two representations of the reference and current frames is that, for ME/MC, we just want to use the shift-invariant wavelet coefficients of the reference frame to predict the critically-sampled wavelet coefficients of the current frame.
The v-pixel-shifted or {dx,dy}-pixel-shifted coefficient of the pth wavelet block of reference frame t’ can be represented by
The coefficient of the pth wavelet block of current frame t can be represented by

6. Background Motion Estimation in Shift-Invariant Wavelet Domain (3)
The sum of absolute difference (SAD) of the pth wavelet block for the motion vector v is computed as follows:
The wavelet blocks in the search window in the reference frame are compared to current wavelet block, and a reference wavelet block that leads to the best match is selected.
The sum of absolute difference (SAD) of the pth wavelet block for the displacement vector v is computed as follows:
The optimum motion vector v∗ of the pth wavelet block, which has minimum displacement error, is given by:
However, computational complexity of full search ME in SIWD is so time consuming because of the extra shifted subbands that limits its practical applications. For this reason, alternative and faster techniques should be developed.
The optimum motion vector v∗ of the pth wavelet block, which has minimum displacement error, is given by:

7. Background Partial Distortion Elimination
Partial Distortion Elimination (PDE) is a fast algorithm which has identical quality as that of FSA.
The partial SAD (PSAD) is used to eliminate impossible candidates before the complete calculation of the SAD:
The partial distortion elimination (PDE) is a fast algorithm which has identical quality as that of the full search algorithm (FSA).
The partial sum of differences (PSAD), which is used to eliminate impossible candidates before the complete calculation of the SAD, is calculated as follows:
where m is the number of differences needed to reach the SADmin which has already been found so far, and O (n) represents a generic matching order in the wavelet block.

8. Proposed Algorithm Wavelet Matching Error Characteristic
To improve the computational saving of PDE, if the expected values of the matching error dp+v(i, j) in the search window w fulfills the following criterion:
First, let’s look at the proposed fast lossless ME algorithm in SIWD. Since the partial distortion elimination (PDE) is a good way to find a fast lossless ME algorithm, our proposed method is also based on PDE.
To improve the computational saving of PDE, if the expected values of the matching error dp+v(i, j) in the search window w fulfills the following criterion:
then we assume that this matching order will be generally effective for all of the candidate vectors.
In order to obtain the predicted matching error ep(i, j), we consider
then we assume that this matching order will be generally effective for all of the candidate vectors in the search window.
Therefore, to fulfill the above objective, we use ep(i, j) to predict the matching error.

9. Proposed Algorithm Wavelet Matching Error Characteristic
To obtain the predicted matching error ep(i, j), solve this equation
We finally can obtain an approximate solution of Eq. (6):
To solve this equation, we have
We finally can obtain an approximate solution of Eq. (6):
The solution of the predicted matching error consists of two parts: one is the wavelet coefficient magnitude in the current wavelet block, another is the expected value of wavelet coefficient magnitude in the reference wavelet block. Since the expected value of the wavelet coefficient magnitude in a search window varies smoothly, the second part can be considered as a constant for different ep(i,j).
The predicted matching error is proportional to the wavelet coefficient magnitude in the current wavelet block.
That is to say, larger wavelet magnitude in the current wavelet block tends to produce larger matching error.
Larger wavelet coefficient magnitude in the current wavelet block tends to produce larger matching error

10. Proposed Algorithm MR-WMEC-PDE
Three key factors which affect the performance of PDE
the Searching Order
in which the wavelet blocks are tested during the searching phase.
the Matching Order
in which the coefficients within a wavelet block are picked up to compute the SAD.
the Comparison Interval
in which comparison between PSAD and SADmin is performed.
Three new strategies for PDE by using wavelet matching error characteristic (WMEC) are proposed.
Observed from the PDE equation in previous slide, we can sum up three key factors which affect the performance of PDE.
the Searching Order, If a good SADmin is found early, then many more successive tests have a tighter distortion bound and may be skipped.
the Matching Order, If the highest contributions to SAD are found early, then the distortion bound may be reached after a small number of
differences and the PSAD can be stopped.
the comparison interval, If a good tradeoff between the cost of comparisons and the number of useless differences is achieved, then the number of comparisons and differences can be reduced.
In order to make use of these three factors, the wavelet matching error characteristic (WMEC) is employed to develop three new strategies for PDE.
Searching Order Strategy based on Wavelet Multi-Resolution Property
Matching Order Strategy based on Wavelet-tree Grouping Scheme
Comparison Interval Strategy based on Adaptive Sub-blocks Checking Unit

11. Proposed Algorithm MR-WMEC-PDE
Searching Order Strategy based on Wavelet Multi-Resolution Property
Instead of the spiral search order, the proposed searching order strategy uses the normalized partial SAD in LL subband level as the estimated SAD (ESAD)
Because the LL subband actually corresponds to the lowest resolution version of original frame, the motion information in LL subband of a wavelet block is highly correlated with that of the whole wavelet block. This property can be used for a new searching order strategy to get a good match sooner.
Therefore, Instead of the spiral search order, the proposed searching order strategy uses the normalized partial SAD in LL subband level as the estimated SAD (ESAD). Then, sort the ESAD using the counting sort algorithm in ascending order to obtain the searching order SO = {vn | n = 0, ...,w−1}.
Then, sort the ESAD using the counting sort algorithm in ascending order to obtain the searching order SO = {vn | n = 0, ...,w−1}.

12. Proposed Algorithm MR-WMEC-PDE
Matching Order Strategy based on Wavelet-tree Grouping Scheme
A wavelet-tree grouping scheme according to spatial self-similarity property and matching error clustering property of wavelet coefficient
Sort Ep(Bl,bl,m) using the quick sort algorithm in descending order to obtain the matching order of level l: MOl = {bl,m | m = 0, ...,M − 1}
Since the wavelet coefficient magnitude tends to be larger in the higher decomposition level, the matching error also tends to be larger in the higher decomposition level according to above reasoning. Therefore, the trend of matching order is from the higher decomposition level to the lower decomposition level.
In addition, the matching errors with similar magnitude in the same level tend to appear in clusters. Thus, we propose a wavelet tree grouping scheme which groups the coefficients into sub-blocks according to the spatial self-similarity property and the matching error clustering property of wavelet coefficients.
To determine the matching order of sub-blocks, the estimated matching error of each sub-block is calculated by
Sort Ep(Bl,bl,m) using the quick sort algorithm in descending order to obtain the matching order of level l: MOl
The figure shows (a) Wavelet-tree grouping scheme in a wavelet block, (b) an example of adaptive matching order based on wavelet-tree grouping scheme

13. Proposed Algorithm MR-WMEC-PDE
Comparison Interval Strategy based on Adaptive Sub-blocks Checking Unit
In conventional PDE methods, fixed comparison interval, such as eight-pixels or sixteen-pixels checking unit, is usually adopted.
Combined with the wavelet-tree grouping scheme, an adaptive comparison interval strategy is proposed.
In the proposed strategy, every 2l−1 sub-blocks in the decomposition level l are used as the checking unit.
In conventional PDE methods, fixed comparison interval, such as eight-pixels or sixteen-pixels checking unit, is usually adopted.
Combined with the wavelet-tree grouping scheme, an adaptive comparison interval strategy is proposed.
In the proposed scheme, every 2l−1 sub-blocks in the decomposition level l are used as the checking unit, that is, the comparison between PSAD and SADmin is performed every 2l−1 sub-blocks for the decomposition level l.
So that, the cost of comparisons remains at a reasonable level.

14. Experimental Results (1)
Simulation results are reported in the following ways:
operation number : used to compute the partial distortion
speed-up ratio : for motion estimation including the required overheads for comparison.
Simulation results are reported in the following ways:
operation number per block used to compute the partial distortion;
execution time per frame for motion estimation including the required overheads for comparison.
For the performance comparison, we tested the following algorithms: FSA, Spiral-PDE, CPME-PDS, MR-WMEC-PDE.
For performance comparison with other algorithms
Full Search Algorithm (FSA)
Spiral-PDE [5]
CPME-PDS [6]
proposed MR-WMEC-PDE

15. Experimental Results (2)
Average Operation Number per Block for Tested Algorithms
This table lists the Experimental Results of Average Operation Number per Block for Tested Algorithms
On average speed-up ratio in terms of operation number, MR-WMEC-PDE is better than Spiral-PDE and CPME-PDS by about 92% and 34%, respectively.
On average speed-up ratio in terms of operation number, MR-WMEC-PDE is better than Spiral-PDE and CPME-PDS by about 92% and 34%, respectively.

16. Experimental Results (3)
Average Execution Time per Frame for Tested Algorithms
This table lists the Experimental Results of Average Execution Time per Frame for Tested Algorithms
On average speed-up ratio in terms of execution time, MR-WMEC-PDE is better than Spiral-PDE and CPME-PDS by about 84% and 63%, respectively.
The overhead of MR-WMEC-PDE algorithm still can be tolerable since the loss of performance that comes from the overhead is only 10% on average.
On average speed-up ratio in terms of execution time, MR-WMEC-PDE is better than Spiral-PDE and CPME-PDS by about 84% and 63%, respectively.

17. Conclusion Fast Lossless Multi-Resolution Motion Estimation Algorithm
Wavelet Matching Error Characteristic (WMEC)
Three New Strategies for PDE
Searching Order Strategy based on Wavelet Multi-Resolution Property
Matching Order Strategy based on Wavelet-tree Grouping Scheme
Comparison Interval Strategy based on Adaptive Subblocks Checking Unit
In this paper, we proposed a fast lossless multi-resolution motion estimation algorithm, MR-WMEC-PDE.
We take advantage of the new discovery about wavelet matching error characteristic to develop the three new strategies for PDE:
Searching Order Strategy based on Wavelet Multi-Resolution Property
Matching Order Strategy based on Wavelet-tree Grouping Scheme
Comparison Interval Strategy based on Adaptive Subblocks Checking Unit