1 Presented by Jari Korhonen Centre for Quantifiable Quality of Service in Communication Systems (Q2S) Norwegian University of Science and Technology (NTNU) Research Activities for Quality of Experience in Networked Multimedia within Q2S

2 Outline  About Q2S  Research topics and vision at Q2S o Research directions and vision o Subjective / Objective assessment of Audio / Video / Audiovisual content  Research highlights: comparing apples and oranges o Subjective comparison of source and channel distortion in video streaming  Conclusions

3 Centre of Excellence in Quantifiable Quality of Service in Communication Systems – Q2S Funded by Norwegian Research Council Supported by Telenor R&I Hosted by Norwegian University of Science and Technology 52 people (7 profs., post-docs, PhD students and administration) Basic research and laboratory experimentation Main Goals: –Network Media Handling –QoS and QoE assessment and monitoring –QoS mechanisms for dynamic networks. Q2S

4 Q2S Research  Networked Media Handling o Technology to present, manipulate and evaluate multimedia content o Focus on media representation, error protection, perceived quality, functional placement  Quality assessment and monitoring o Measuring methods and models for QoS and QoE o Measurement of perceived QoS and QoE, measurement methods and architecture, and traffic performance  QoS mechanisms for dynamic networks o Mechanisms that affect QoS, emphasis on heterogeneous and dynamic environment o Focus on dependability, security, resource allocation

5 Research Directions (QoE) QoE Applications Virtual worlds Serious gaming Broadcasting Medical Tourism... Content User generated Interactivity Identity and security Technology Network Media Handling 3D Media Perception Cognitive technologies Markets Business Models Creativity Implementation Modeling and assessment Subjective test methods

6 Facilities

7 QoS and QoE modeling and assessment Development of Subjective Test Methods and Objective Models - Multimodal perception of audio and video - Content dependency - Perceived audio-visual quality assessment for future media (multimodal media with complex scenes (HDTV, UHDTV, NHX, 3D, etc.)) - Design of quantifiable metrics for perceived quality (audio, video, audiovisual, …) Use of Objective Models - Automated monitoring of end-user perceived quality - Using “Perceived QoS” to adapt and enhance system performance - Objective network measurements of end-to-end QoS

8 Research highlights: Comparing apples to oranges  In multimedia communications applications (such as video streaming), both source and channel distortion may appear o Source distortion is derived from lossy compression o Channel distortion is caused by transmission errors (packet losses and/or bit errors)

9 Motivation – comparison of source and channel distortion  Qualitative characteristics of source and channel distortion are different o Source distortion impacts the overall quality (see the upper image) o Channel distortion typically appear in spatially and temporally limited areas (see the lower image) o Classical example of ”comparing apples and oranges” applies Source distortion Channel distortion

10 Motivation – limitations of known quality assessment methods  It is challenging to compare the perceptual impact of source and channel distortion o Objective metrics, such as PSNR, typically give reliable results only when same type of distortions are compared o Traditional subjective metrics and methodologies require quite a lot of work and reliability may be questionable Different persons may understand subjective scales differently  There is a need for new subjective quality assessment method!

11 Proposed test method  Instead of rating the difference between test sequence and anchor sequence (traditional double- stimulus impairment scaling), we use adjustable stimulus o Named as double-stimulus adjustable quality fixed anchor (DSAQFA) o User adjusts one of the stimuli so that its quality is (as closely as possible) similar to the non-adjustable anchor sequence o Allows comparison between different types of distortions o Minimizes the need for training and eliminates the personal differences how quality labels are understood

12 Functionality of the test program  Anchor sequence and adjustable sequence played in sync  Adjustable sequence: different source distortion levels the user can choose between  When ready with adjustment, user presses ’ok’, result is recorded and system moves to the next test case

13 UI of the test program Adjustable sequence Anchor sequence Slider for adjusting quality, and ok-button

14 Generation of test sequences  Adjustable sequences: original video clip is encoded with H.264/AVC using 16 different quantization parameter (QP), varying from 24 (best quality) to 51 (worst quality)  Anchor sequences: bit errors inserted in encoded files using Gilbert-Elliot model, to create channel distortion

15 Voted and anchor PSNR in selected cases

16 Discussion of results  PSNR seems to overestimate the quality, when source distortion in anchor is low and there is some channel distortion  However, this difference seems to disappear when the source distortion in anchor gets lower  Content matters: channel distortion seems to be less annoying (compared to PSNR) for sequences with high temporal and spatial activity  Also a lot of individual variance: some test subjects give constantly higher or lower ratings than average  Ongoing research: other objective metrics than PSNR included

17 Concluding summary  General introduction of NTNU-Q2S  Research areas and vision o Emphasis of presentation on quality metrics and assessment for networked video  Research highlights o Comparing apples and oranges: novel subjective quality assessment method for comparing video sequences with different types of distortion (source and channel)

18 QoMEX ’10 The Second International Workshop on Quality of Multimedia Experience June 21-23, 2010 Trondheim, Norway  Topics of interest (not limited to): o User experience assessment and enhancement o Visual and auditory user experience o QoE for virtual, augmented and mixed realities o Link between QoS, QoE and acceptance o Psychological and social dimensions of QoE o Standardization acitivities in multimedia quality evaluation  Important dates: o Submission deadline: January 31, 2010 o Notification of acceptance: April 1, 2010  More information: qomex2010.org QoMEX