QoS Measurement and Management for Multimedia Services Thesis Proposal Wenyu Jiang April 29, 2002

Topics Covered Objective QoS metrics for real-time multimedia Subjective/Perceived quality Objective perceptual quality estimation algorithms Quality enhancement for real-time multimedia IP telephony deployment VoIP quality in the current Internet

Backgrounds and Motivations The Internet is still best-effort –Needs QoS monitoring What to measure/monitor? –Loss, delay, jitter –Must map to perceived quality What to do if quality is not good? –End-to-End: FEC, LBR –Network provisioning: voice traffic aggregation IP telephony service deployment –Current ITSPs are not doing well –Lack of study on localized deployment What is the status of the current Internet?

How Real-time Multimedia Works A/D conversion; Encoding; Packet transmission; Decoding; Playout; D/A conversion Dominant QoS factors: –Loss  clipping/distortion in audio –Delay  lower interactivity –Jitter  late loss

Delay and Loss Measurement Solutions for clock synchronization –Telephone-based synchronization –RTT-based, assume symmetric delays –GPS-based Dealing with Clock drift –De-skewing by linear regression One-way vs. round-trip measurement –Internet load often asymmetric –One-way loss and delay are more relevant to real-time multimedia

Loss and Delay Models Loss Models –Gilbert model –Extended Gilbert model –Others Delay Models –More difficult to construct –No universal distribution function –Temporal correlation between delays

Perceived Quality Estimation Mean Opinion Score (MOS) –Requires human listeners –Labor and time intensive –Reflective of real quality Objective perceptual quality estimation algorithms –PESQ, PSQM/PSQM+, MNB, EMBSD –Speech recognition based (new) MOS GradeScore Excellent5 Good4 Fair3 Poor2 Bad1

Network Provisioning for VoIP Silence suppression –Saves bandwidth  statistical multiplexing –The on/off patterns in human voice depend on the voice codec or the silence detector Voice traffic aggregation –Multiplexing by token bucket filtering –The on/off patterns in human voice directly affects aggregation performance Past study assumes exponential distribution

IP Telephony Deployment Localized deployment –More practical than a grand-scale Internet deployment –Can still interoperate with an IP telephony carrier Issues –PSTN interoperability –Security –Scalability –Billing

Research Objectives Objective QoS metrics –Modeling –Their relationship to perceived quality Objective perceptual quality estimation algorithms vs. perceived quality (MOS) Quality improvement measures –End-to-End: FEC vs. LBR –Network-based: voice traffic aggregation IP telephony deployment issues VoIP quality measurement over the Internet

Completed Work: QoS Measurement Tools UDP packet trace generator Clock synchronization and de-skewing tool Loss and delay modeling tools –By examining a packet trace –Outputs Gilbert and extended Gilbert model parameters –Outputs conditional delay CCDF Playout simulator –Simulates several common playout algorithms –FEC is also supported

Completed Work: Comparison of Loss Models Loss burst distribution –Roughly, but not exactly exponential Inter-loss distance –Clustering between adjacent loss bursts

Loss Model Comparison, contd. Loss burstiness on FEC performance –FEC less efficient under bursty loss Final loss pattern (after playout, FEC) –Generally also bursty

Mapping from Loss Model to Perceived Quality Random vs. bursty loss –Bursty  lower MOS Effect of loss burstiness –Sometimes very bursty loss does not lead to lower quality

A New Delay Model Conditional CCDF (C 3 DF) Allows estimation of burstiness in the late losses introduced by (fixed) playout algorithm

Objective vs. Subjective MOS Algorithms: PESQ, PSQM, PSQM+, MNB, EMBSD Using Original Linear 16 samples as reference signal Using G.729 no loss clip as reference signal

Objective MOS Correlation, contd. Second test set Stronger “saturation” effect observed for MNB1 and MNB2, but not for PESQ Linear-16 reference signalG.729 reference signal

Auditory Distance vs. MOS EMBSD and PSQM+ appear to have the largest spread, i.e., least correlation w. MOS PSQM seems to be similar to MNB in terms of correlation

Auditory Distance vs. MOS, contd. Second test set Similar behaviors observed Linear-16 reference signalG.729 reference signal

Analysis of Objective MOS Correlation Quantitative metric –Correlation coefficient  –But it does not tell everything! AlgorithmTest Set 1Test Set 2  l16  g729  l16  g729 MNB MNB PESQ

Speech Recognition Performance as a MOS predictor Evaluation of automatic speech recognition (ASR) based MOS prediction –IBM ViaVoice Linux version –Codec used: G.729 –Performance metric absolute word recognition ratio relative word recognition ratio

Recognition Ratio vs. MOS Both MOS and R abs decrease w.r.t loss Then, eliminate middle variable p

Speaker Dependency Check Absolute performance is speaker-dependent But relative word recognition ratio is not

Speech Intelligibility Results Human listeners are asked to do transcription Human recognition result curves are less “smooth” than MOS curves.

Analysis of Voice On-Off Patterns Past study finds spurt & gap distributions to be exponential Modern voice codecs and silence detectors have different behaviors

Voice Traffic Aggregation Simulation environment –DiffServ token bucket filter –Exponential, CDF and trace- based model simulations –N voice sources –Token buffer size B (packets) –R: ratio of reserved vs. peak bandwidth Key performance figure –Probability of out-of-profile packet

Aggregation Simulation Results Results based on G.729 VAD –CDF model resembles trace model in most cases –Exponential (traditional) model Under-predicts out-of-profile packet probability; The under-prediction ratio increases as token buffer size B increases

Simulation Results, contd. Results based on NeVoT SD (default parameters: high threshold, long hangover) –Similar behavior, although the gap between exponential and CDF model is smaller for NeVoT case

Comparisons of FEC and LBR Forward error correction –Bit-exact recovery –No decoder state drift upon recovery Low bit-rate redundancy (LBR) –Just the opposite to FEC Design of an optimal LBR algorithm –State repair via redundant codec –Optimal packet alignment –MOS quality verified to be better than the rat LBR –Allows a more “fair” comparison with FEC

MOS Quality of FEC vs. LBR FEC shows a substantial and consistent advantage over LBR –This is true for all LBR configurations we tested Main codec is G.729 except for AMR LBR DoD-LPC LBR DoD-CELP LBR

MOS of FEC vs. LBR, contd. AMR LBR: narrowest gap with FEC (Not shown here) FEC out-performs LBR under random loss as well G LBRAMR LBR

Optimizing FEC Quality Packet interval   loss burstiness   FEC efficiency  Result: FEC MOS performance also improves

Optimizing Conversational MOS for FEC A larger packet interval  more delay Trade-off between quality and delay The E-model –Considers both delay and loss (and many other transmission quality factors) Optimizing FEC MOS with the E-model

Optimizing FEC MOS, contd. Validating E-model based prediction with real MOS test results

Localized IP Telephony Deployment: Architecture Component based and distributed architecture Allows easy integration of all SIP- compliant devices and programs

Deployment Issues PSTN interoperability –T1 configuration and PBX integration T1 line type (Channelized vs. ISDN PRI) Line coding and framing (layer 2) Trunk type: Direct-inward-dialing (DID) Access permission on the PBX side –SIP/PSTN gateway configuration Dial-peer: locates the proper SIP server or PSTN trunk Dial-plan (translating calls from/to PSTN)

Deployment Issues, contd. Security –Issue: gateway has no authentication feature –Solution: Use gateway’s access control lists to block direct calls SIP proxy server handles authentication using record-route –Allows easier change in authentication module (software-based) –Certain users can only make certain gateway calls Scalability –SIP server (DNS SRV scaling) –Gateway; voic server; conference server Billing –Initial implementation via transaction logging

On-going Research Measurement of the current Internet How well can it support VoIP? –Or, how easy can VoIP applications adapt to (unfavorable) network conditions? How fast does network condition change? Can network redundancy help improve VoIP quality? –Physical redundancy (access links) –Virtual redundancy (overlay networking)

Conclusions Completed research relating to many aspects of real-time multimedia, in particular VoIP On-going work calls for: –A comprehensive measurement of the Internet –Analysis of the to-be measurement data –An answer to the question: how good is it today, and, how much better can we do?