Department of Communication and Electronic Engineering University of Plymouth, U.K. Lingfen Sun Emmanuel Ifeachor New Methods for Voice Quality Evaluation for IP Networks

4 September, 2003ITC-18, Berlin, Germany2 Outline Introduction Subjective MOS test from traditional to Internet based Intrusive voice quality measurement from listening-only to conversational Non-intrusive measurement from subjective based to objective based Conclusions

4 September, 2003ITC-18, Berlin, Germany3 Introduction Aims: to investigate new subjective and objective measurement methods for VoIP applications Subjective tests MOS (Mean Opinion Score), user perceived quality Benchmarking objective methods Slow, time consuming and expensive Objective measurements Intrusive methods (e.g. PESQ, only listening quality) Non-intrusive methods (e.g. E-model, only for limited codec/applications)

4 September, 2003ITC-18, Berlin, Germany4 Traditional MOS test Strict test requirement (e.g. sound proof room) Guarantee consistent testing environment Slow, time-consuming and expensive Test environment far to reality Originated from codec quality assessment (the quality difference is subtle).

4 September, 2003ITC-18, Berlin, Germany5 Internet-based MOS test Based on Internet, carried out at office, project room or classroom, …, close to reality Easier access to large number of subjects (e.g. 39 tests at the same time) Save time and money Lack of a controlled testing environment (e.g. background noise).

4 September, 2003ITC-18, Berlin, Germany6 Online MOS test

4 September, 2003ITC-18, Berlin, Germany7 Online MOS test

4 September, 2003ITC-18, Berlin, Germany8 Online MOS test results

4 September, 2003ITC-18, Berlin, Germany9 Voice Quality Evaluation Reference speech encoder Trace data Jitter buffer Degraded speech PESQ MOS (PESQ) E-model MOS (E-model) decoder Subjective test MOS (Web or Room)

4 September, 2003ITC-18, Berlin, Germany10 Performance Comparison

4 September, 2003ITC-18, Berlin, Germany11 Performance Comparison Results show that Internet-based MOS test compares well with traditional MOS test. Two objective test methods (PESQ and E-model) can both predict subjective MOS score well. Correlation coefficients for MOS comparison NamePESQ vs. RMOS PESQ vs. WMOS Emodel vs. RMOS Emodel vs. WMOS WMOS vs. RMOS Emodel vs. PESQ Corr. Coeff

4 September, 2003ITC-18, Berlin, Germany12 Intrusive conversational quality measurement PESQ is the latest ITU standard for intrusive voice quality measurement. It can only predict one-way listening voice quality. How to extend it to predict conversational quality? One solution: to combine with E-model

4 September, 2003ITC-18, Berlin, Germany13 Procedures: Obtain MOS from PESQ based on a comparison of reference and degraded speech Convert it back to R-factor, then to I e Obtain I d from end-to-end delay Calculate MOS c from I d and I e from E-model Intrusive conversational measurement PESQ Delay model MOS  R  I e IeIe End-to-end delay E-model MOS c IdId Reference speech Degraded speech MOS (PESQ)

4 September, 2003ITC-18, Berlin, Germany14 R-factor with MOS (1) (2) ( From R to MOS, from G.107) ( From MOS to R, by 3 rd order polynomial)

4 September, 2003ITC-18, Berlin, Germany15 Combined with E-model (3) (4) (5) Convert R back to MOS to obtain conversational voice quality

4 September, 2003ITC-18, Berlin, Germany16 Non-intrusive objective measurement E-model is the latest ITU standard for non- intrusive voice quality prediction. Its parameters/equations (e.g. I e ) are still based on subjective tests. Difficult to apply to new codecs/applications How to improve it ? Based on objective methods (e.g. PESQ) to derive parameters/equations for new codecs and applications.

4 September, 2003ITC-18, Berlin, Germany17 Procedure 1: obtain MOS (PESQ) Obtain MOS (PESQ) vs. packet loss rate (random) for the codec (e.g. AMR 12.2 Kb/s) MOS (PESQ) value was obtained by averaging over different speech samples (male and female) in ITU data base and different loss locations/patterns.

4 September, 2003ITC-18, Berlin, Germany18 Procedure 2: Obtain I e equation Convert MOS vs. packet loss to I e vs. packet loss using Equations (2) and (3). Curve fitting to obtain equation of I e vs. loss. (6)

4 September, 2003ITC-18, Berlin, Germany19 Procedure 3: Obtain MOS (loss, delay) From I e and I d, R can be derived, then MOS can be obtained directly from packet loss and end-to-end delay. This can be used for non-intrusive quality monitoring and perceptual buffer optimization and perceived QoS control.

4 September, 2003ITC-18, Berlin, Germany20 Conclusions/Future work Investigated novel subjective and objective voice quality evaluation methods. Future work More extensive Internet-based MOS test and comparison with P.800 MOS test New applications for predicted perceived voice quality.

4 September, 2003ITC-18, Berlin, Germany21 Contact MOS Test Website: Contact Lingfen Sun: Emmanuel Ifeachor: Thank you!