1. 2nd Workshop on Wideband Speech Quality - June 2005 1 Perceptual Wideband Audio Quality Assessments Using PEAQ Christian Schmidmer Opticom GmbH, Erlangen info@opticom.de

2. 2nd Workshop on Wideband Speech Quality - June 2005 2 Contents  Quality, definitions  User expectation  Subjective tests  Psychoacoustics  PEAQ  PESQ vs. PEAQ

3. 2nd Workshop on Wideband Speech Quality - June 2005 3 Aspects of Perceived Quality  Conversational Quality = ...

4. 2nd Workshop on Wideband Speech Quality - June 2005 4 What is “Quality”? “Quality is the difference between what we perceive and what we expect.” From habilitation thesis of Prof. Ute Jekosch “…they are used to phones that sound like a phone.” Frank Meier, Infineon Maybe more important: …is for free.

5. 2nd Workshop on Wideband Speech Quality - June 2005 5 Differences in Perception of Voice and Audio  Experience, a priori knowledge  Expectation  Cognitive effects  “Error correction”  Different subjective tests require different models

6. 2nd Workshop on Wideband Speech Quality - June 2005 6 The Problem of Subjective Scales BitrateMOS 256kBit/s5 128kBit/s4 … 64kBit/s1 BitrateMOS 128Bit/s5 64kBit/s4 … 16kBit/s1 MP3 @ High Quality: MP3 @ Intermediate Quality: The range of qualities in the subjective test defines the subjective scale!

7. 2nd Workshop on Wideband Speech Quality - June 2005 7 MOS acc. To P.800  Standardized Listening Test Procedure acc. to ITU-T P.800ff  Absolute Category Rating Test (ACR), no comparison to reference signal (original)  „How good does it sound?“  5-point grading scale ‚opinion scale‘  Averaging over test Subjects: MOS ‚Mean Opinion Score‘  Language dependent! Excellent Good Fair Poor Bad 5432154321 ImpairmentGrade

8. 2nd Workshop on Wideband Speech Quality - June 2005 8 Standardised assessment procedure for 'small impairments' in audio systems (ITU-R 1994) Comparison between reference and test signal Very sensitive to subtle distortions double-blind triple-stimulus with hidden reference Subjective Assessment in ITU-R BS.1116 OriginalAB original / coded coded / original

9. 2nd Workshop on Wideband Speech Quality - June 2005 9 Continuous grading scale with “anchors” “Subjective Difference Grade“ (SDG) Question: „How different do the files sound“ Subjective Assessment in ITU-R BS.1116

10. 2nd Workshop on Wideband Speech Quality - June 2005 10 Subjective Testing of Intermediate Audio Quality (IAQ) “MUSHRA” Multi Stimulus Test with Hidden Reference and Anchors  developed by EBU working group B/AIM  targets at IAQ  ITU-R BS.1534

11. 2nd Workshop on Wideband Speech Quality - June 2005 11 MUSHRA Test Training of Subjects subjects can randomly access all types of codecs at similar bitrate comparison with CD quality reference two low-pass 'anchors' (7kHz, 3.5kHz) incl.

12. 2nd Workshop on Wideband Speech Quality - June 2005 12 MUSHRA Test Scoring Phase comparison with CD reference, hidden reference inc.. two low-pass 'anchors' (7kHz, 3.5kHz) inc.. subjects can randomly assess all codecs under test of similar bitrate at the same time subjects adjust slider, no score involved slider mapped to 0..100

13. 2nd Workshop on Wideband Speech Quality - June 2005 13 Comparison of Subjective Test Methods

14. 2nd Workshop on Wideband Speech Quality - June 2005 14

15. 2nd Workshop on Wideband Speech Quality - June 2005 15 Temporal Masking 005010015050100200150-50 t [ms] 0 20 40 60 SL [dB] Pre-Simultaneous-Postmasking Premasking: 2-5ms Postmasking: 120ms Depending on the signal characteristics of the masker Masker

16. 2nd Workshop on Wideband Speech Quality - June 2005 16 Pitch Scale / Critical Bands A sine tone and a noise of critical bandwidth with the same center frequency and energy density are perceived equally loud.

17. 2nd Workshop on Wideband Speech Quality - June 2005 17 Threshold in Quiet - Masked Threshold Threshold in Quiet

18. 2nd Workshop on Wideband Speech Quality - June 2005 18 PEAQ is based on: –PAQM KPN Research, Netherlands / OPTICOM –NMR Fraunhofer, Germany / OPTICOM –DIX TU Berlin / Deutsche Telekom Berkom –POM CCETT, France –PERCEVAL CRC, Canada –"Tool box" IRT, Germany ITU-R TG 10/4: Call for proposals (1995) Jan. 1999 released as ITU-R Rec. BS.1387 PEAQ

19. 2nd Workshop on Wideband Speech Quality - June 2005 19 Intrusive Testing Network X A Network Y B Comparison with known stimulus: + Very high accuracy +Black box approach – no knowledge of DUT - Requires a reference signal -Generates traffic Alternatively both signals may be captured by the test system!

20. 2nd Workshop on Wideband Speech Quality - June 2005 20 Two Versions of PEAQ:  PEAQ „Basic“  computational efficiency  realtime performance  PEAQ „Advanced“  highest possible accuracy

21. 2nd Workshop on Wideband Speech Quality - June 2005 21 Structure of a perceptual measurement tool Reference (=sent file) Feature- Extractor Perceptual Model Test (=received file) Cognitive Model MOS (Quality Measure) Perceptual Model a b a b

22. 2nd Workshop on Wideband Speech Quality - June 2005 22 Excitation Listening Level (dB SPL) Input Signal 1 FFT & Scaling 2048 Punkte 42.6ms/23.4Hz Outer and Middle Ear Weighting Grouping into Critical Bands ¼ Bark “Pitch” Internal Noise Spreading Temporal Masking Forward masking 2 + fs=48kHz (fs=44.1kHz) a b Perceptual Model, PEAQ “Basic”

23. 2nd Workshop on Wideband Speech Quality - June 2005 23 MOVs used in PEAQ “Basic” Version

24. 2nd Workshop on Wideband Speech Quality - June 2005 24 Perceptual Model, PEAQ “Advanced”

25. 2nd Workshop on Wideband Speech Quality - June 2005 25

26. 2nd Workshop on Wideband Speech Quality - June 2005 26 PEAQ vs. MUSHRA Microsoft Windows Media 4 MPEG-4 AAC (Fraunhofer) MP3 (Fraunhofer) Quicktime 4, Music-Codec 2 (Qdesign) Real Audio 5.0 RealAudio G2 MPEG-4 TwinVQ (Yahama) EBU Tests of Internet Audio Codecs

27. 2nd Workshop on Wideband Speech Quality - June 2005 27 Constraints of MUSHRA Testing no absolute scores: -> scores depend on the test condition low-pass anchors are only one quality dimension -> disturbance of artefacts is another one spreading of the scale from best to worst -> what about adding new items to an existing test? In order to verify PEAQ performance we must adjust the best and worst item (not the anchors!)

28. 2nd Workshop on Wideband Speech Quality - June 2005 28 PEAQ vs. MUSHRA (EBU Test)

29. 2nd Workshop on Wideband Speech Quality - June 2005 29 Results

30. 2nd Workshop on Wideband Speech Quality - June 2005 30 Results

31. 2nd Workshop on Wideband Speech Quality - June 2005 31 Results

32. 2nd Workshop on Wideband Speech Quality - June 2005 32 Results

33. 2nd Workshop on Wideband Speech Quality - June 2005 33 Results

34. 2nd Workshop on Wideband Speech Quality - June 2005 34 When to use PEAQ or PESQ  Is it a BS.1116 or MUSHRA Experiment?  Use PEAQ!  Is the subjective test P.800?  Is it speech? Yes: –Is the bandwidth <= 8kHz? »Yes: Use PESQ! »No: Use PEAQ with care! No: »Use PEAQ with care!

35. 2nd Workshop on Wideband Speech Quality - June 2005 35 Final Question:  Can I use PESQ instead of PEAQ?  Perception of voice differs from perception of music  PESQ time alignment fails on music  PEAQ and PESQ are modelling different subjective tests No!

36. 2nd Workshop on Wideband Speech Quality - June 2005 36 www.opticom.de OPTICOM Germany More Information: info@opticom.de Thank you!