1. diffusion libre A state of the art of voice quality measurement techniques and of associated standards at ITU-T SG 12 Orange Labs Vincent BARRIAC, Recherche & Développement, TECH/OPERA 18th September 2009

2. 2 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality section 1 Introduction on Voice quality and associated metrics

3. 3 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality  Pre-requesites to service: what is not visible by users but mandatory for the service to work:  Service VoIP : what is directly visible by users: DNS (translation logical name -> IP address) FTP (firlware update download) NTP (synchronisation) DHCP (IP address resolution) FCPE (registration, association client/line) Voice quality Access to voice messages Call set up Registration on serviceplatform Extra services (message indication, number presentation, call signalling, secret, muti-com, et c.) QoS dimensions of VoIP : a network operator point of view Call retainability

4. 4 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Voice quality  Voice quality is a subjective notion, related with perception, and impacting the acceptability, and the success of services  It is an end-to-end concept, directly depending on technical characteristics of all involved elements : –Handsets (acoustics and signal processing) –Transmission equipments –Service platforms  Main parameters to consider : –Intrinsic quality of the signal (Mean Opinion Score, or MOS), depending on – Distortion (low bit-rate coding) – Packet loss – Signal level or attenuation –Transmission delay and its variation (jitter) –Echo –Noise

5. 5 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Principal components of voice quality  Distortion of the signal –Impact on the intelligibility of speech  End-to-end transmission time (also sometimes called delay) –Impact on the interactivity of the conversation  Echo: a two-dimensional phenomenon –Characterized by its delay and its level (or its attenuation) –Impact on the interactivity of the conversation and on transmission times  Noise –Impact on the listening comfort  Vocal level –Impact on the listening comfort

6. 6 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Inter-relationship of QoS Factors (one-way quality, VoIP) Decoding + signal processing Performance  Network Factors Network Delay Network Packet Loss Network Jitter Overall Delay  Application Factors Overall Packet Loss Jitter Buffers Perceived Quality QoS Service Level

7. 7 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Evolution of speech quality and customers' satisfaction key factors [source: ETSI, 2000]  80's and before: –user satisfaction correlated to speech quality key factor: speech quality  90's: –apparition of competition (users => customers) –apparition of mobile services key factor: accessibility  00's => today: –apparition of VOIP (new BM) key factor: price  Today, it's more complicated than ever –ungaranteed, time-varying speech quality –heterogeneity of contexts of usage, terminals, quality –relationship between customer satisfaction and quality? key factors: price & simplicity  Tomorrow?

8. 8 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Things have considerably changed  However, there is a strong demand from network and marketing units for producing speech quality data in order to: –monitor speech quality –market speech quality (essentially wideband)  Today, this demand is mainly answered by the realization of –listening tests – Evaluation techniques based on the judgement of a panel of users in listening or conversational situations – Techniques follow very strict standardized protocols, generally consigned in ITU-T Recommendations, of which most used is P.800 –the use of "objective" (or instrumental) models – Based on tools (hardware or software) able to analyze the contents of the signal (of speech or any other type used for measurement) – Determination of the quality with a maximum correlation with the judgement of the users (depends of the accuracy of the tool) – Less costly in terms of time and resources

9. 9 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality section 2 Voice quality objective methods and models

10. 10 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality  1) Non-intrusive measurements  Rather dedicated to supervision (network performance) on real communications  Ignores the contribution of terminal  Without additional cost, can supervise a big amount of communications – and thus obtain a good macroscopic view of transmission quality  On IP : protocol analysers dedicated for VoIP –IP parameters (packet loss, jitter, codec type ) –Overall transmission quality rating (ITU-T P.564 or E-model) –Requires a signal decoding and reconstruction for signal-based measurements  2) Intrusive measurements  For end-to-end assessment close to the client –Good correlation with subjective perception –Access to the signal and thus possible use of PESQ  Performed on test communications –Extra traffic, with potential cost and load –Point to point (microscopic) vision, difficult to extrapolate without a wide spread of probes  Analogue interface (PSTN, GSM or behind a HGW) –MOS with reference (PESQ) and many other possible measurements (levels, echo, delay, etc.)  IP interface –IP parameters (packet loss, jitter, codec type ) –Overall transmission quality rating (ITU-T P.564 or E-model) Two major ways to measure voice quality

11. 11 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality PESQ (Perceptual Evaluation of Speech Quality)  Measurement on speech signal (bandwidth: 300-3400 Hz for narrow-band or 50-7000 Hz for wide-band)  Principle: Comparison of the voice signal to assess with a reference signal  One-way measurement – does not take delay into account – applies only for listening quality (no echo, no double talk).  Independent of speaker gender and language  Estimates delay variations –applies for packet transmission (including VoIP), –measures these variations  Standard Model (ITU-T P.862 Recommendation) –validated against a large variety of subjective databases ( ρ =0.93 on 13 databases) –reliable and cheap alternative to subjective listening quality tests  P.862.1 and.2 give a mapping function from raw PESQ scores into MOS-LQON and MOS-LQOM respectively  Requires some precautions to get reliable MOS-LQO scores –original speech signal (level, voice activity rate, duration, etc.) –implementation of the model (+ IPR)

12. 12 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality The E-model  ITU-T G.107 Recommendation  Parametric approach (i.e. not based on signal analysis)  Officially, only a transmission planning tool –But often used as a passive monitoring tool (caution!)  Takes into account the subjective effects of all impairments in an additive way –Concept of "impairment factors" –Default values can be used when no measurement results are available  The E-Model output is a score (R) between 0 and 100 –Can be translated under certain conditions into MOS-LQEN or MOS-CQEN scores –A new version for wide-band audio has been standardised in 2006 – R wb can now reach a maximal value of 129  Takes into account : – Transmission delay and echo (speaker, listener and sidetone) – Levels (SNR, noise, loudness ratings) – Packet loss – Low bit rate coding (intrinsic quality and robustness against packet loss)

13. 13 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Limitations of existing methods  Not yet fully adapted to wide-band (50-7000 Hz) and further audio transmission –Even PESQ-WB (P.862.2) has not been enough validated –Developments ongoing to develop new subjective methods and replace PESQ (P.OLQA, 2010)  The most known tools are for one-way measurements –Do not take delay into account –Apply only for listening quality (no echo, no double talk) –Developments ongoing to develop new methods (P.CQO)  Requires some precautions to get reliable and comparable MOS-LQO (N or M) scores –Original speech signal (level, voice activity rate, duration, gender of speakers etc.) for PESQ –Implementation of the models (+ IPR) –Problem of different quality scales  They don't address quality from a real end-through-end perspective –Need for new tools for correlation and diagnostic

14. 14 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality section 3 Main results and ongoing works in ITU-T SG12

15. 15 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality ITU-T SG12 Responsible for Recommendations  on the end-to-end transmission performance of terminals and networks, in relation to the perceived quality and acceptance by users of text, data, speech (mostly), and multi-media applications.  Although this work includes the related transmission implications of all networks and all telecommunication terminals, a special focus is given to, interoperability and implications for NGN, and also includes work on performance and resource management.  Although this work includes the related transmission implications of all networks and all telecommunication terminals, a special focus is given to IP QoS, interoperability and implications for NGN, and also includes work on performance and resource management. Lead Study Group on QoS and Performance Note: free Access to ITU-T Recommendations http://www.itu.int/ITU-T/publications/recs.html

16. 16 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality Non-intrusive Speech Quality Modelling: P.564 An example of "framework" standard Conformance testing for voice over IP transmission quality assessment models (NB and WB)  minimum criteria for objective speech quality assessment models that predict the impact of observed IP network impairments on the one-way listening quality experienced by the end-user in IP/UDP/RTP-based telephony applications Main applications : monitoring of transmission quality for operations and maintenance purposes, measurements in support of service level agreements (SLAs) between service providers and their customers.

17. 17 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality P.564 – 3 modes of operation Mode A : Dynamic Operation In the network, but uses information from end-point (RTCP-XR packets) Mode B : Static Operation In the network, with a-priori knowledge of end-point behaviour Mode 3 : Embedded Operation Model co-located with the jitter buffer in the end ‑ point with access to the jitter buffer

18. 18 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality On-going work in ITU-T SG12  Wider bandwidth –From speech (Narrow and wide band) to super-wideband and full- band.  Consequences on coders, acoustic interfaces, test signal, analysis methodologies, quality models (including binaural)  Quality Modelling (speech and multimedia) The future "universal Model" for Speech (NB, WB, SWB), including acoustical interfaces, and diagnostic features: P. OLQA A Model to qualify noise cancellation performance: P.ONRA A Model for conversational quality: P.CQO  IPTV - Quality of experience requirements for IPTV (G.1080) - Performance monitoring for IPTV (P.NAMS and P.NBAMS)  IP QoS and network performance metrics

19. 19 diffusion libre Recherche & Développement - A state of the art of measurement of voice quality P.OLQA  P.OLQA is supposed to be the replacement of PESQ –Extension to a bigger bandwidth (up to 14 kHz instead of 7) –Extension to acoustical interfaces –Further conditions taken into account (incl. time warping) –Addition of a degradation decomposition (4 dimensions : frequency content, continuity, noisiness, loudness) to enhance the core model and give diagnostic information  A competition has been launched to select the best model –FT is candidate in partnership with DT –Other competitors : TNO, Psytechnics, Ericsson, Opticom, SwissQual –The results of validation of models are awaited for next February… –The real application of P.OLQA in real measurement tools is not before mid- 2010

20. diffusion libre Thank you