1. Present document contains informations proprietary to France Telecom. Accepting this document means for its recipient he or she recognizes the confidential nature of its content and his or her engagement not to reproduce it, not to transmit it to a third party, not to reveal its content and not to use it for commercial purposes without previous FTR&D written consent. Method of Packet Errors Cancellation Suitable for any Speech and Sound Compression Scheme STQ Workshop, Sophia-Antipolis, February 11 th 2003 Balazs KÖVESI, Dominique MASSALOUX

2. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D2 Introduction  Context  Application like VoIP or audio streaming –Possible high packet loss rate (up to 10 %)  Proposition of a frame error concealment (FEC) method  Copes with high packet loss rate  Relies on CELP synthesis scheme  Independent from the codec type  Speech oriented but also suitable for music  Includes adaptive gain control  Avoids "robot" voice  Ensures the decoder memory update  Smoothing after an erased period

3. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D3 Plan  Basic principle of the new FEC method  Implementation in a MDCT codec  Generalization to other codec types  Conclusion

4. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D4 Basic principle of the new FEC Decoded signal decoder Valid data Indication of erased data Storage of decoded samples Synthesis of missing samples Decoder update reconstructed signal Smoothing with decoded signal

5. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D5  The MDCT transform  Analysis with 50 % overlap  Synthesis with overlap-add Implementation in a MDCT codec memory for the next frame overlap-add -windowing -TF -T  F transform -FT -F  T transform -windowing -overlap-add Analysis window n-1 20 ms Analysis window n t Synthesis window n-1 decoded frame n 20 ms t Synthesis window n

6. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D6  Effect of frame erasure  the loose of x bitstream frames affects x+1 output frames  these frames have to be synthesized in the decoder Implementation in a MDCT codec Disturbed: frames n-1 & n 20 ms Erased: frame n t

7. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D7  Memorizing part  After decoding a valid frame –The 40 ms output memory is updated –The energy of the frame is calculated –The energy memory buffer is updated ( 5 s ) Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal

8. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D8 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal LTP analysis & V/UV detection

9. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D9 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  LPC filter modelizes the spectral envelope  Coefficients not transmitted –LPC analysis order can be higher than in a usual CELP (32 @ 16kHZ)  better performance on music past decoded signal LPC coefficients filter A(z) classical method 20 ms

10. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D10 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  Precise pitch estimation is crucial for the good performance  Only integer pitch (P) values are examined [50 Hz, 600 Hz]  Normalized correlations on the last 2P samples  Pitch criteria: maximum correlation + multiple & fractional verifications  V/UV criteria: selected correlation value + energy value  5 s energy memory energy evolution of the last two pitch periods past decoded signal p1p1 p2p2 p3p3 p4p4 correlation calculations

11. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D11 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  LPC analysis filtering past decoded signal A(z) past excitation signal

12. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D12 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  Excitation signal generation for the LPC synthesis filtering  voiced excitation: –2 components –harmonic, lower frequency bands LTP filter combined with a low pass filter –less harmonic, higher frequency bands LTP filter combined with a high pass filter + randomly evolving pitch

13. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D13  Excitation signal generation for the LPC synthesis filtering  voiced excitation: –2 components –harmonic, lower frequency bands LTP filter combined with a low pass filter –less harmonic, higher frequency bands LTP filter combined with a high pass filter + randomly evolving pitch Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  unvoiced excitation –non harmonic, lower frequency bands –“randomized” LTP filtering + low pass filtering + sudden energy variations are suppressed

14. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D14 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  LPC synthesis filtering excitation signal synthesized signal A(z) 1

15. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D15 Implementation in a MDCT codec Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthe- sized signal LTP parameters (B(z)) LPC parameters (A(z)) LPC analysis LTP analysis & V/UV detection calc. past excitation signal LTP filtering 1/B(z) LPC synthesis 1/A(z) Adaptive gain control Memory of past decoded signal  Important in case of long erased periods (> 20 ms) Q 2 adaptation laws: –stationary –non-stationary – The adaptations also depend on the pitch value Q decision available from the LTP analysis background noise level 40 ms t t

16. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D16  Recoverable information Implementation in a MDCT codec Synthesized frames n-1, n, n+1 20 ms Erased: frame n-1 & n t decoded frame n+2

17. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D17  Recoverable information  for the first erased frame Implementation in a MDCT codec MDCT transform on the first 2 synthesized frames t Partly recovered frame n-1IMDCT transform

18. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D18  Decoder memory update Implementation in a MDCT codec t Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal MDCT transform on the last 2 synthesized frames IMDCT memory to update IMDCT transform (F  T + windowing) updated IMDCT memory

19. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D19  Recoverable information  for the last erased frame Implementation in a MDCT codec t Partly recovered frame n+1IMDCT transform

20. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D20  Smoothing part  without smoothing Implementation in a MDCT codec t Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthesized frames n-1, n, n+1 decoded frame n+2 discontinuity synthesized domain error-free domain

21. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D21  Smoothing part  A codec independent solution: Implementation in a MDCT codec t Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal Synthesized frames n-1, n, n+1 decoded frame n+2 synthesized domain error-free domain Extra synthesized samples crossfading  1 0

22. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D22  Smoothing part  with MDCT smoothing Implementation in a MDCT codec t smooth transition at frame n+1overlap-add like crossfading Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal synthesized domain error-free domain

23. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D23 s can be adapted to any coding scheme Q was successfully implemented in – temporal codecs (G.711, G.721, G722) – in a CELP codec (G.723.1) – in a hierarchical codec composed of a CELP and a transform codec  Memorizing and synthesis part are codec independent  Decoder memory update –very important for recursive codecs (CELP) –general solution: coding – decoding on the synthesized frames –too complex for CELP –less complex solution: backtracking  Smoothing –a general solution: crossfading –more efficient smoothing can be found for some coding schemes (ex.: MDCT) –the decoder memory update ensure the smoothing in CELP codecs Generalization to other codec types Valid data Decoded signal decoder Erased data indication Storage of decoded samples Synthesis of missing samples Decoder update synthesized signal Smoothing with decoded signal

24. France Telecom R&D Diffusion of this document is subject to France Telecom authorization D24  A general FEC method for any coding scheme  optimal for speech voice, good performances on music  avoids too synthetic sound for voiced frames  keeps the nature of the unvoiced frames  enhanced energy management Q careful update of the decoder memory  smoothing after an erased period s Informal subjective tests have shown its good behavior  Successfully implemented in group communication applications  Perspectives:  speech / music decision + enhanced music mode  … Conclusion