1. T.Gy. Intrernetes médiakommunikáció. 2009.05. 13. 1 Internetes médiakommunikáció Az Internetes Médiakommunikáció minőségének megítélése, mérése, a rendszer tervezési alapjai Takács György 12. előadás 2009. 05. 13.

2. What is QoE? The overall acceptability of an application or service, as perceived subjectively by the end user. NOTES 1 Quality of Experience includes the complete end- to-end system effects (client, terminal, network, services infrastructure, etc). 2 Overall acceptability may be influenced by user expectations and context. ~From ITU-T Recommendation P.10/G.100~

5. Factors affecting QoE of IPTV services

7. Why QoE evaluation? Recent survey indicates that…. 84% of IPTV providers think video quality monitoring is critical or very important 77% say video quality is a main reason for customer churn 78% say video quality problems are a main reason for high volumes of customer service calls ~From http://telephonyonline.com/iptv/news/iptv-video- quality-0219/~ There surely is a market need for QoE evaluation methods/tools!

8. Quality monitoring as well as quality design Conventional telephone services over PSTN are relatively stable after a call is connected. In PSTN services, service providers established telephone systems with static quality design, then, monitor/manage the “availability” and “functionality” IP networks (core/access/home-NWs) and IP-based terminals are time-variant systems. In IPTV services, media quality varies depending on coding performance at H/E, traffic conditions in networks, and terminal performance

9. What to measure? Pre-transmission domain Encode quality System failure (blackout, frame freezing etc.) Network domain Packet loss Delay Delay jitter Post-transmission domain Error recovery performance such as FEC (Forward Error Correction) Decode quality (including PLC: Packet Loss Concealment) D/A conversion performance Color management performance Display performance

10. When to measure? QoE measurement is necessary in the following phases: Before the service is developed: Individual quality factors (coding distortion etc.) Quality planning After the service is developed, but before the service is delivered: Off-service quality check After the service is delivered: In-service quality monitoring at user end In-service network mid-point monitoring On-site quality check (on claim)

11. Where to measure?

12. How to measure? Subjective quality assessment is time consuming and expensive. It cannot be applied to on-site and/or real-time measurement, which is essential in quality monitoring. Objective quality assessment, which estimates QoE by subjective testing, is the solution!

13. Overview of objective quality assessment models From the viewpoint of input information into a model, objective quality assessment models can be categorized as follows: Media-layer models Parametric packet-layer models Bitstream-layer models Parametric planning models Hybrid models

14. Media-layer models

15. Current standards of media-layer models (video) ITU-T standardized Rec. J.144, which recommends four fullreference models for Standard Definition TV, in 2004. – Coding distortion only (no transmission errors such as packet loss) VQEG finalized its evaluation on full-reference and reducedreference models applicable to QCIF ~ VGA. The draft Recommendation has been consented at ITU-T SG9 as Rec. J.247, which includes four models, and Rec. J.246, respectively in May 2008. – Packet-loss degradation as well as coding distortion is in the scope of Recommendation. ITU-T SG9 considered that current no-reference technologies seemed to be premature in terms of quality-estimation accuracy.

16. Parametric packet-layer models Only uses packet-header information. Light-weight implementation. Can be applied even if payload is encrypted.

17. Bitstream-layer models Use up to coded bitstream (payload), but not decoder output. Can take into account the content dependence of the impact of packet loss.

18. Standardization activities for packet-layer and bitstream-layer models Parametric packet-layer models ITU-T SG12 Q.14 & VQEG P.NAMS (non-intrusive parametric model for the assessment of performance of multimedia streaming) Bitstream-layer models ITU-T SG12 Q.14 & VQEG P.NBAMS (non-intrusive parametric bitstream model for the assessment of performance of multimedia streaming)

19. Parametric planning models Use quality planning parameters as input. ITU-T Recommendation G.1070 for videophone applications. ITU-T SG12 Q.13 is working on G.OMVAS (opinion model for video and audio streaming applications).

20. Present state of multimedia quality assessment methods Future trends of quality assessment research Multimodality Multiparty Wideband Examples of NTT’s studies on multimedia quality assessment for audiovisual communication services

21. Relevant ITU-T/R Recommendations on Multimedia Quality Assessment Multimedia quality assessment is at a reasonably advanced stage.

22. Existing Subjective Multimedia Quality Assessment Methods (P.911) Multimedia quality is assessed in a similar way to individual audio/video qualities. Absolute Category Rating (ACR) Degradation Category Rating (DCR) Pair Comparison Method (PC) Single Stimulus Continuous Quality Evaluation (SSCQE) Assessment paying attention to the cross-modal influences is important. Interactions between differing quality levels in different modalities

25. Existing Subjective Multimedia Quality Assessment Methods (P.920) Several category judgment scales are used to evaluate multimedia quality. Overall audiovisual quality Individual audio/video qualities Effort needed to interrupt Communication difficulty Acceptability of communication Communication quality depends on tasks used in conversational test. Assessment considering interactivity and usability is also important.

26. Existing Concept Model of Objective Multimedia Quality Evaluation (J.148) Basic components of an objective multimedia quality model are defined.

27. Examples of Multimedia Quality Evaluation Model

28. Framework for Conducting Quality Assessment Research

29. Key Words for Future Trends of Multimedia Quality Assessment (1/3) Multimodality Combination of multiple media such as audio, video, text, graphics, fax, and telephony in the communication of information Key points of quality assessment √ Intermedia synchronization √ Intermedia quality balance

30. Key Words for Future Trends of Multimedia Quality Assessment (2/3) Multiparty Communication style extending from 1:1 to N:N (e.g., instant messaging, teleconferencing, and distributed collaboration services) Key points of quality assessment √ Interdestination synchronization √ Interdestination quality balance

31. Key Words for Future Trends of Multimedia Quality Assessment (3/3) Wideband Telecommunications applications having more bandwidth available for higher-quality multimodal services Key points of quality assessment √ Psychological factors: We need to assess the richness of high-quality services not only on a one-dimensional scale, like MOS, but also on a multi-dimensional scale.

32. Examples of NTT’s Studies on Quality Assessment for “Multimodality” (1/3)

33. Examples of NTT’s Studies on Quality Assessment for “Multimodality” (2/3)

34. Examples of NTT’s Studies on Quality Assessment for “Multimodality” (3/3)

35. Examples of NTT’s Studies on Quality Assessment for “Multiparty” (1/2) Quality imbalance is one of the multiparty quality degradation factors.

36. Examples of NTT’s Studies on Quality Assessment for “Multiparty” (2/2) Overall quality is strongly affected by the inferior quality at another point. Other-point audiovisual quality depends on conversation task or roles.

37. Examples of NTT’s Studies on Quality Assessment for “Wideband” (1/3) Interactive multimodal quality can be evaluated using a multi- dimensional scale of psychological factors.

38. Examples of NTT’s Studies on Quality Assessment for “Wideband” (2/3) Psychological factors were extracted by using the semantic differential (SD) technique and factor analysis. Subject’s impression of an audiovisual communication service was evaluated on the basis of 25 pairs of bipolar adjectives on a seven-grade comparison scale.

39. Examples of NTT’s Studies on Quality Assessment for “Wideband” (3/3) Multimedia quality was formulated as a function of two psychological factors expressing an aesthetic feeling and a feeling of activity.