AIMS’99 Workshop Heidelberg, May 1999 Assessing Audio Visual Quality P905 - AQUAVIT Assessment of Quality for audio-visual signals over Internet and UMTS Geoff Morrison, BT Labs

AIMS’99 Workshop Heidelberg, May 1999 Contents Why we are doing this What we want to achieve How we are going about it Who is doing it –BT, CSELT, Berkom When are we doing it –February 1999 to December 2000 The Results

AIMS’99 Workshop Heidelberg, May 1999 Coding algorithms, such as MPEG-4 and H.263, can compress audio and video signals to bit rates <1 Mbit/s as available over mobile and IP networks. The impact of impairments, from compression and transmission, on users' opinion must be understood. Audiovisual communication must be evaluated under representative conditions and real tasks, such as conversational services and AV database retrieval. Network performance must be monitored to guarantee a minimum QoS. Need single-ended models to check audio and video transmission quality in real time. Rationale

AIMS’99 Workshop Heidelberg, May 1999 Is the resulting application usable for purpose? How best to assign a given total bit rate between video and audio? Compression Impairments Digital TV in studios is 216 Mbit/s Lossless compression typically only halves the bit rate To reach very low rates requires lossy audio and video compression, introducing significant distortions

AIMS’99 Workshop Heidelberg, May 1999 Transmission Impairments AV bits <> data Audiovisual content has a “timeliness” aspect –Critical in conversational applications –Important in database retrieval applications Error detection and retransmission protocols not ideal for audiovisual content –variable delay –variable throughput Compressed AV is very sensitive to transmission errors, packet loss –more compression  more sensitivity

AIMS’99 Workshop Heidelberg, May 1999 Error Handling Error resilience tools for encoder –structure compressed data so that damage is localised –optimise encoder options for the network performance Forward Error Correction –improve apparent network performance seen by decoder Error concealment at decoder –reconstruct approximations for damaged parts of pictures and sound Combinations of above How well do these work in practice?

AIMS’99 Workshop Heidelberg, May 1999 Project Objectives Understand the potential of audio, video and AV communication at bit rates < 1Mbit/s for UMTS and IP. Develop tools and techniques for simulation of complete audiovisual transmission systems. Develop quality evaluation methods for complete audiovisual transmission systems. Investigate the relationship between system parameters (network and terminal) and subjective quality. Provide guidelines for the system parameters to be used for new applications. Encourage the use of audiovisual communication on these dominant networks of the future

AIMS’99 Workshop Heidelberg, May 1999 Work Items Define the tests to be performed Develop/implement a UMTS test bed Develop/implement an IP test-bed Conduct subjective tests Make objective measurements and relate them to subjective results Disseminate the findings

AIMS’99 Workshop Heidelberg, May 1999 Definition of tests Literature survey for available test methods Select appropriate methods

AIMS’99 Workshop Heidelberg, May 1999 UMTS Test Bed Video codec (MPEG-4 and/or H.263) Audio codec (MPEG AAC, scalable and non-scalable) Error resilience tools (MPEG-4) MPEG-4 multiplex Transmission errors - simulated and perhaps actual Error recovery Demux Optimisation of parameters Tools to make objective measurements Production of material for subjective and objective tests

AIMS’99 Workshop Heidelberg, May 1999 UMTS Test Bed Baseline non real-time test bed –MPEG-4 baseline video codec, AAC and G audio, AL layer and MPEG DMIF Enhanced non real-time test bed –Error resilience tools, layered audio

AIMS’99 Workshop Heidelberg, May 1999 UMTS Test Bed Server ApplicationAudio decoderVideo decoder Network interface Network Server Client

AIMS’99 Workshop Heidelberg, May 1999 IP Test Bed Video codecs (H.263 and/or MPEG-4) Audio codecs (G.723.1, MPEG AAC scalable and n/s) Error resilience tools (ITU-T, MPEG, IETF) Packetisation Transmission errors - simulated, measured profiles and actual Error recovery Optimisation of parameters Tools to make objective data measurements Production of material for subjective and objective tests

AIMS’99 Workshop Heidelberg, May 1999 IP Test Bed Baseline non real-time test bed –G.723.1, H.263, simplified RTP, simple network impairment simulator Enhanced non real-time test bed –more codecs including MPEG, error resilience, complete RTP, error concealment, enhanced network simulator Enhanced real-time test bed –real time image capture, encoding and decoding

AIMS’99 Workshop Heidelberg, May 1999 IP Test Bed

AIMS’99 Workshop Heidelberg, May 1999 Subjective tests “Conventional” subjective tests covering a suitably wide range of variables –the codecs –bit rate, video frame rate, audio bandwidth –Error conditions (packet loss, delay variability, …) –Protocols (CBR, VBR, ABR, RSVP,..) –Application (one-way, conversational) Users may accept lower quality in return for mobility. Testing will seek an estimation of the threshold of acceptability of audio and video quality in mobile applications

AIMS’99 Workshop Heidelberg, May 1999 Objective measurements Objective assessment of audio quality –TOSQA, PAMS,... Objective assessment of video quality –model of human vision –other algorithms to objectively estimate user’s opinion Compare subjective and objective assessments Investigate relationship between parameter values and subjective audio, video and AV quality assessments

AIMS’99 Workshop Heidelberg, May 1999 Double Ended Model Measuring instrument needs both original input to encoder and delivered version at decoder output

AIMS’99 Workshop Heidelberg, May 1999 Single Ended Model Measuring instrument needs only decoder output –more suitable for live performance monitoring

AIMS’99 Workshop Heidelberg, May 1999 Project Resources 20 man-months

AIMS’99 Workshop Heidelberg, May 1999 Results Reports due in Q4 of The findings should be of benefit to at least 3 groups: –Network Providers. Results such as minimum desirable bit rates and error characteristics should influence the planning and operation of IP and UMTS networks intended to carry audiovisual services. –Application Providers. The results will provide reliable guidance about which applications and operating parameters will satisfactorily meet customers' expectations and those which will not. –Researchers in the field of "human factors". It is expected that the project will advance the state of the art in perceptual modelling and in the understanding of cross-modal effects.