1. Jingming Xu Multimedia Communications Lab University of Waterloo
Rate-distortion Optimization for MP3 and AAC Audio Coding with Complete Decoder Compatibility
Welcome Ladies and Gentlemen. Thanks for coming to my master’s seminar.
I am Jingming Xu. I did my master’s with Dr. En-hui Yang at the Multimedia Communications Lab.
Today, I will talk about…
If you have any comments or questions during my talk, I will be happy to answer them at the end of the presentation.
Jingming Xu
Multimedia Communications Lab
University of Waterloo

2. Outline Introduction and motivation
MP3, AAC, and Two-nested-loop Search
Rate-distortion optimization for MP3
Rate-distortion optimization for AAC
Conclusions and Future Research
Introduction to audio coding and MPEG standards, and motivation of our research in …..
Then I will give a short review of
MP3 and AAC audio coding standards, with emphasis on quantization and entropy coding constraints.
State-of-the-art MP3 and AAC quantization and entropy coding scheme, and its problems.
Based on the standard constraints, we will develop …. , in each case, I will provide simulation results with comparison to state-of-the-art…
Rate-distortion optimization for MP3.
Rate-distortion optimization for AAC.
Concluding remarks.
September 16th, 2005 2

3. Introduction Audio coding - different from universal data compression
Long term correlations
Multi-channel correlations
Subject to natural noises
Subjective perceptual quality judgement
Audio coding methods - for both lossy and lossless
Linear prediction
Time-frequency mapping (DCT, FFT, MDCT, etc.)
Parameter coding ….
Audio signals have their characteristics such as …., which makes universal data compression schemes inefficient when applied directly on audio.
During the past 30 years, people have come up with many methods specifically for both lossy and lossless audio coding, such as …
Many of them have become international or industry standards, among which, Mpeg is certainly the most popular one.
September 16th, 2005 3

4. Introduction (2)
MPEG - the most successful audio coding standard series so far
MPEG-1 (1992) - T/F mapping based, 3 Layers with increased complexity
MPEG-2 BC (1994) - backward compatible with MPEG-1, with multi-channel and sampling frequency extensions
MPEG-2 AAC (1997) - introducing more coding tools and giving up backward compatibility to improve quality
MPEG-4 AAC (1999) - inherited from MPEG-2 AAC with TwinTQ and bitrate scalability extensions
The development of MPEG audio standard can be roughly divided into 2 phases ….
The first phase, …., MP3
MPEG-4 even supports a new vector quantization scheme TwinTQ, and bitrate scalability. However, since those extensions are beyond the scope of our research, we simply denote both MPEG-2 AAC and MPEG-4 AAC as AAC.
MPEG-1 Layer 3 and MPEG-2 BC Layer 3 define the popular “MP3”
September 16th, 2005 4

5. Introduction (3) Motivations
MP3 and AAC leave structured encoding blocks design open for performance enhancement.
The state-of-the-art MP3 and AAC quantization and entropy coding scheme, Two-nested-loop Search (TNLS), is essentially incapable to exploit the maximal standard-constrained flexibility for best rate-distortion tradeoff.
The huge success of MP3 and AAC in the digital audio industry.
Like many other multimedia compression standards, ….
As we will later, …. …. There is still room we can exploit.
The huge success of MP3 and AAC in the digital audio industry also motivates our research.
September 16th, 2005 5

6. Introduction (4) Quality evaluation of compressed audio
Most widely used objective measure - noise-to-mask ratio
Most widely used subjective measure - ITU listening test (ITU-R Recommendation BS.1116)
Triple sources A, B, C with hidden reference, double blind
5-grade impairment score scale
Two quality evaluation methods used in our research ….
NMR is the ratio of noise energy in this band to its perceptual masking threshold, here w_{i} is the inverse of the masking threshold.
During the test, listener is free to listen to sources A, B, or C.
Source A is known to be the reference signal.
However, source B and C may be either the reference signal or the test signal.
The assignment is determined randomly in that neither the listener nor the test administrator should know beforehand.
After listening, listener is asked to rate sources B and C relative to source A according to a continuous 5-grade impairment scale.
September 16th, 2005 6

7. MP3 and AAC audio coding standards
Encoding process
Window switching
Stereo coding
Pre-processing in AAC: gain control, prediction, noise shaping and substitution, etc.
A high-level block diagram of MP3 encoding process is shown in …
Time domain audio samples are first fed into a T/F mapping block which converts them into spectral coefficients.
They are also fed into a psychoacoustic model which generates control information for T/F mapping (window switching), quantization and entropy coding.
Under the psychoacoustic modeling control, spectral coefficients are quantized, entropy coded, and packed up with format information and control information.
T/F mapping option: window switching
Quantization and entropy coding option: separate/joint channel coding
September 16th, 2005 7

8. MP3 and AAC audio coding standards (2)
Quantization and entropy coding in MP3
Scale factor bands and non-uniform quantization
scale_factor values are encoded by fixed number of bits in the side information and variable number of bits in the main_data stream
The whole spectrum is divided into a fixed number of scale factor bands
The non-uniform quantizer, corresponding to the de-quantizer defined in MP3, can be formulated as,
Where global_gain is ….
After quantization, the scale factor values for one frame are broken down into four parts in the bitstream for efficient storage.
Occupy
September 16th, 2005 8

9. MP3 and AAC audio coding standards (3)
Quantization and entropy coding in MP3
Huffman coding
34 fixed Huffman codebooks
Huffman coding region division: Each region is coded with a different codebook that best matches the statistics of that region.
big_value, count_1, zero, ….
After quantization, the quantized spectrum is Huffman encoded. ….
The region division fashion is generally open to design.
Except for the big_value subdivision in short windows.
For AAC, there are lots of …
Those audio pre-processing tools are optionally applied before the quantization and entropy coding block.
The coding efficiency resulting from the adoption of these tools is signal dependent,
and thus they usually operate under the control of psychoacoustic model.
September 16th, 2005 9

10. MP3 and AAC audio coding standards (4)
Quantization and entropy coding in AAC
Non-uniform quantizer: same as in MP3
scale_factor values are differentially encoded relatively to the one of the preceding band by fixed Huffman codebook
Huffman coding
12 fixed Huffman codebooks
Huffman coding region division: Section boundaries can only be at the scale factor band boundaries
For each section, the length of the section in scale factor bands, and the index of the codebook used for that section, are transmitted with a fixed number of bits.
AAC uses the same ….
In nature, the codebook index of each band is also differentially encoded solely relative to the one of the preceding band except for band 0: if they are the same, no bit needs to be transmitted at all; otherwise, it costs a fixed number of bits.
September 16th, 2005 10

11. Two-nested-loop Search algorithm
Outer Loop
Inner Loop
Given a target data rate, the task of its outer loop is to amplify the scale factor for “distorted” band so that NMR is less than 1.
Since the amplified parts of the spectrum need more bits for encoding, but the number of available bits is constant, the inner loop changes the global quantizer step size until the given spectrum can be encoded by available bits.
In all, this mechanism shifts bits from spectral regions where they are not required to those where they are required.
September 16th, 2005 11

12. Two-nested-loop Search algorithm (2)
Problems in TNLS
Quantization, scale factor adaption and Huffman coding are considered separately.
Has no convergence guarantee
Does not target at minimizing the overall distortion
Disregards the inter-band correlations of scale factors and Huffman codebook selection in AAC
However, they actually work together to determine the rate-distortion performance.
Optimization on only one of these factors in one step may force sub-optimal selection of the other factors in the following steps
and degrade the performance of the whole system in the end.
The best parameters so far have to be stored during each iteration and restored as output after the final termination.
And the iteration process has to be terminated according to predefined conditions without knowing the optimality of the result.
In our research, we aim at directly attack these problems …
September 16th, 2005 12

13. Rate-distortion optimization for MP3
Problem formulation
Lagrangian RD cost minimization
- quantized coefficients
- scale factors
We formulate the rate-distortion optimization problem as the minimization of the actual Lagrangian RD cost …. By incorporating all coding factors in the …. stage
- Huffman coding region division
- Huffman codebook selection
- non-uniform de-quantizer defined in MP3
- noise-to-mask ratio
September 16th, 2005 13

14. Rate-distortion optimization for MP3 (2)
Problem formulation
Soft-decision quantization
In conventional hard-decision quantization, is solely determined by given , i.e.,
However, in the soft-decision quantization scenario, is considered as a flexible coding factor and selected such that the actual RD cost can be minimized.
Therefore,
The key point in our problem formulation is to further consider the quantized coefficients as optimization variable, leading to the so-called soft decision quantization.
September 16th, 2005 14

15. Rate-distortion optimization for MP3 (3)
Fixed-slope graph-based iterative RD optimization
Step 1: Initialize a set of scale factors from the given frame of spectrum with a HCB selection fashion . Set t=0, and specify a tolerance as the convergence criterion.
Step 2: Given and for any t 0, find the optimal quantized spectrum and HCB region division fashion throughout a standard-constrained graph, where and achieve the minimum
Denote by .
Based on the problem formulation, we propose a fixed-slope graph-based iterative algorithm for RD optimization in mp3
Since the maximal allowed coefficient amplitude is closely related to the Huffman codebook region in which that coefficient lies given fixed Huffman codebook selection, the coding gains from the quantized spectrum and the region division are exploited jointly in Step 2, by an efficient graph-based optimal path search algorithm.
September 16th, 2005 15

16. Rate-distortion optimization for MP3 (4)
The directed graph is constructed based on the MP3 quantization and entropy coding constraints for long window.
A simpler version exists for short window.
Each layer corresponds to a HCB region and each state in one layer stands for two neighboring coefficients to be encoded using the HCB selected for that region.
Two special states, frame_begin and frame_end, are used to take care of the start and the end of the frame, respectively.
Assign each transition a cost resulting from minimizing the decomposed cost on the state which that transition goes to by adapting the corresponding quantized coefficients,
the minimization then becomes the problem to search for the path with minimal accumulated cost through the graph.
This graph-based search is a full dynamic programming and always gives the optimal solution.
Graph Search for MP3 Quantized Spectrum and Region Division
September 16th, 2005 16

17. Rate-distortion optimization for MP3 (5)
Fixed-slope graph-based iterative RD optimization
Step 3: Given , and , update to , so that
achieves the minimum
Step 4: Given , and , update to , so that
Step 5: Repeat Steps 2, 3 and 4 for t = 0,1,2…. Until
, then output , , and
Step3 update scale factors,
Note that MP3 has adaptive storage for scale factors: R(q) is usually determined by a few largest scale factors in the frame, and there is no close-form formula to calculate optimal ones. Therefore, a full search of all scale factor storage fashions within the standard needs to be applied.
Step4 update Huffman codebook selection,
For each region, the Huffman codebook that gives the minimum codeword length is selected.
Step5 repeat …. Until convergence occurs.
September 16th, 2005 17

18. Rate-distortion optimization for MP3 (6)
Simulation results: ANMR
(implementation based on ISO MP3 reference codec)
We implement our optimization algorithm based on ISO reference codec and the most advanced state-of-the-art MP3 codec LAME respectively.
In each case, we use the original output as our initialization.
For ISO reference codec, we see that the joint optimization algorithm successfully improves the coding efficiency with at least 0.8dB distortion reduction for bitrates above 64 kbit/s.
violin.wav
spme50_1.wav
September 16th, 2005 18

19. Rate-distortion optimization for MP3 (7)
Simulation results: ANMR
(implementation based on LAME Best-quality mode)
At least 0.6dB distortion reduction for bitrates above 64 kbit/s for LAME.
In reference codec, the best parameters so far are not stored during each outer loop, which also leads to a great advantage for our joint optimization algorithm, especially in high bitrates.
In both cases, the proposed joint optimization algorithm plays a less important role at low bitrates than high bitraes, since there are less non-zero quantized coefficients that can be optimized on using soft-decision quantization.
violin.wav
spme50_1.wav
September 16th, 2005 19

20. Rate-distortion optimization for MP3 (8)
Simulation results: ITU listening test (80kb/s)
Furthermore, in listening tests, we notice that even though the proposed optimization algorithm yields roughly 0.7 gain for ISO reference codec in both music and speech cases, it still lags behind the LAME encoder alone.
We believe it’s mainly the initial parameters we directly derive from the original reference codec output to blame.
The iterative optimization progress most likely ends at inferior local optimality from this initialization.
September 16th, 2005 20

21. Rate-distortion optimization for MP3 (9)
Remarks
The iteration process may only achieve local optimality, thus a wisely chosen initial state is favored when one targets at achieving the best possible RD performance.
The fixed-slope graph-based iterative algorithm we proposed provides a feasible solution to the problems in TNLS.
One can adaptively adjust the value of , to meet rate or distortion constraints in real audio compression applications.
The iterative process, which can be viewed as a steepest descent minimization approach,
may only achieve local optimality
considering its discrete parameter space which is most likely non-convex.
Specifically, we incorporate all quantization and entropy coding variables in our optimization framework,
directly target at minimizing the overall perceptual distortion,
and guarantee convergence to global/local optimum.
September 16th, 2005 21

22. Rate-distortion optimization for AAC
Problem formulation
Lagrangian RD cost minimization
- scale factor sequence
- Huffman codebook index sequence
Suggest we use Viterbi algorithm to solve the optimization problem.
first-order inter-band dependency ->
Dynamic programming (Viterbi algorithm)
September 16th, 2005 22

23. Rate-distortion optimization for AAC (2)
Fixed-slope trellis-based RD optimization
Step 1: Build up trellis structure. For each state , = 0,1,…., -1, = 0,1,…., -1, = 0,1,…., -1, in the trellis, find the best to minimize its decomposed RD cost
Step 2: Find the optimal path throughout the Trellis by Viterbi algorithm
Step 3: Backtrack the optimal , and as final output
Trellis:
We have totally N stages, where N denotes the number of scale factor bands of one frame.
And each stage is represented by (si; hi). There are Ns*Nh possible representations or states for each (si; hi), corresponding to the combination of Ns possible values of si and Nh possible values of hi.
Step1:
…. Given fixed scale factor j and Huffman codebook k.
Based on state cost got in step 1 and state transition cost ….
September 16th, 2005 23

24. Rate-distortion optimization for AAC (3)
Trellis Structure for AAC Quantization and Entropy Coding
September 16th, 2005 24

25. Rate-distortion optimization for AAC (4)
Simulation results: ANMR
Implementation based on ISO AAC reference codec
Also compared with Aggarwal’s approach (Steps 2, 3 only)
We see that our proposed algorithm successfully improves the coding efficiency of ISO reference codec with at least 2.2dB distortion reduction for bitrates above 96 kbit/s, and more distortion reduction for bitrates below.
The side-information, including the scale factors and Huffman codebook indexs, counts for a larger portion of the total bitrate in low bitrates, which leads to a great advantage for Viterbi over traditional TNLS scheme used in ISO reference codec.
While, soft-decision quantization (Step1) plays a more important role at high bitrates. The distortion reduction from soft-decision quantization grows up to roughly the same level as that from Viterbi when bitrate reaches 192 kbit/s.
violin.wav
spme50_1.wav
September 16th, 2005 25

26. Rate-distortion optimization for AAC (5)
Simulation results: ITU listening test (64kb/s)
Even though pure Viterbi already yields roughly 1.25 improvement against TNLS in both music and speech cases, joint soft-decision quantization and Viterbi can still push for another 0.25 gain.
September 16th, 2005 26

27. Rate-distortion optimization for AAC (6)
Remarks
The fixed-slope trellis-based algorithm we proposed achieves the global optimum RD performance within the quantization and entropy coding stage under the AAC standard constraints.
Joint design of the pre-processing decisions with our proposed optimization can theoretically achieve the global optimum performance in the entire standard-constrained parameter space, however, with computational complexity exponential to the number of bands per frame.
Please refer to the proof of “global optimality” in the thesis.
Since AAC supports M/S stereo coding, TNS, Prediction, LTP and PNS on a band-to-band basis ….
September 16th, 2005 27

28. Conclusions and Future Research
Fixed-slope approach converts the encoding problem to a search problem through a constrained space and then permits the implementation of efficient sequential search algorithm.
Soft-decision quantization spirit completes our RD optimization frameworks, and introduces significant performance enhancement.
Substantial performance improvement against the state-of-the-art encoders is achieved with complete decoder compatibility in each case. ….
It’s also important to emphasize that …. The additional computation complexity due to the proposed optimization is only incurred at the encoder.
September 16th, 2005 28

29. Conclusions and Future Research (2)
Real-time implementations
Extension to scalable AAC
Joint pre-processing and optimization for AAC
Optimal lossy audio compression without syntax constraints
Optimal settings for transform (e.g. block lengths), quantization (e.g. stepsizes) and prediction
Joint design of quantization and entropy coding ….
The proposed MP3 RD optimization, implemented by using pure C code, runs 8 times slower than real-time on a 1.7Ghz CPU, and the proposed AAC rate-distortion optimization, also pure C code based, takes even more time.
Joint RD optimization schemes for each transmission layer so that the optimal RD performance for the entire system can be achieved.
To make the complexity affordable in real audio compression applications, an immediate challenge is then how to design certain joint perceptual modeling and signal analysis methods to sort all possible pre-processing decision candidates.
The successes of our RD optimization for lossy audio compression with complete MP3 and AAC decoder compatibility also give rise to the research proposal of developing optimal lossy audio compression algorithms without any standard constraints, where many fundamental design problems are open, such as,
September 16th, 2005 29

30. Questions?