Performance Measurement and Analysis of H.323 Traffic Prasad Calyam (pcalyam@oar.net) Mukundan Sridharan (sridhara@oar.net) Weiping Mandrawa (wmandraw@oar.net) Paul Schopis (pschopis@oar.net) 19th April 2004, PAM 2004, Juan-Les-Pins, France

Background Today H.323 Videoconferencing is a popular collaboration technology in both industry and academia Demand for high-quality Video and Audio in H.323 applications is always on the rise even given the rapid advancements in the recent past The “Network” is the critical variable that needs to be better understood in assuring satisfactory end-user experience while using H.323 applications Our fundamental question: “What can any given network health diagnostic actually tell us about what the end-user experience might be when using an H.323 application such as Videoconferencing on that network?” www.oar.net

Topics of Discussion Goals of our study Terminology Test Setup and Methodology Results of our study Conclusions www.oar.net

Goals of our Study To obtain “Good”, “Acceptable” and “Poor” performance bounds for network metrics such as delay, jitter and loss for H.323 applications, based on simultaneous subjective and objective quality assessment of H.323 audio and video streams To identify the most dominating factor amongst delay, jitter and loss that affects end-user perception of audiovisual quality www.oar.net

H.323 Overview H.323 is an umbrella standard that defines how real-time multimedia communications such as videoconferencing can be supported on packet switched networks (Internet) Devices: Terminals, Gateways, Gatekeepers and MCUs Codecs: H.261, H.263, G.711, G.723.1 Signaling: H.225, H.245 Transport Mechanisms: TCP, UDP, RTP and RTCP Data collaboration: T.120 Many others… www.oar.net

H.323 Protocol Stack www.oar.net

Understanding the factors that affect H Understanding the factors that affect H.323 Traffic performance assessment… Human Factors Individual perception of audiovisual quality, Lack of training to use the system effectively, … Device Factors VoIP endpoints, Gateways, MCUs, Routers, Firewalls, NATs, Modems, Operating System, Processor, memory, … Network Factors Delay, Jitter, Loss, throughput, BER, … www.oar.net

E2E Measurements of Audio and Video Applications Two approaches to evaluating the performance of audiovisual quality Subjective Measurements Involve human participants to rate audiovisual quality Can you hear me now? We used the Mean Opinion Score (MOS) Ranking technique (ITU-T P.800) Not just “Good”! Objective Measurements Automated techniques to rate audiovisual quality We used the “E-Model” (ITU-T G.107) www.oar.net

MOS Rankings www.oar.net

What constitutes a “Task”? Any activity that takes place in a routine Videoconference Casual conversation, intense discussion, a class lecture, movements involving minimal physical activity, … Recommendations insist realistic scenarios, not just passive viewing Guidelines can be found for task selection, participant training for scoring the audiovisual quality, task ordering, overall environment setup for quality assessments, … H.323 Beacon loopback feature provided additional tasks for the various assessments scenarios www.oar.net

Resynchronization Delay Understanding Delay… SENDER SIDE NETWORK RECEIVER SIDE Compression Delay Transmission Delay Electronic Delay Propagation Delay Processing Delay Queuing Delay Resynchronization Delay Decompression Delay Presentation Delay Delay is the amount of time that a packet takes to travel from the sender’s application to reach the receiver’s destination application Caused by codecs, router queuing delays, … One-way delay requirement is stringent for H.323 Videoconferencing to maintain good interaction between both ends Good (0ms-150ms), Acceptable (150ms-300ms), Poor (> 300ms) Our Results! www.oar.net

Understanding Jitter… Jitter is the variation in delay of the packets arriving at the receiving end Caused by congestion, insufficient bandwidth, varying packet sizes in the network, out of order packets, … Excessive jitter may cause packet loss in the receiver jitter buffers thus affecting the playback of the audio and video streams Good (0-20ms), Acceptable (20ms-50ms), Poor (>50ms) Our Results! www.oar.net

Understanding Loss… Packet Loss is the packets discarded deliberately (RED, TTL=0) or non-deliberately by intermediate links, nodes and end-systems along a given transmission path Caused by line properties (Layer 1), full buffers (Layer 3) or late arrivals (at the application) Good(0%-0.5%), Acceptable (0.5%-1.5%), Poor (>1.5%) Our Results! www.oar.net

Overall Test Setup www.oar.net

World Sites involved in Testing… More than 500 one-on-one subjective quality assessments from Videoconferencing end-users and corresponding traffic traces were obtained from the Testing!!! www.oar.net

Various Tools Used in the Testing… OARnet H.323 Beacon – a tool to troubleshoot H.323 performance problems Network path characteristics determination, Audio loopback, customized tests, … Ethereal – a network traffic sniffer tool To capture traffic traces for Objective Ratings analysis Telchemy VQMon – a network traffic trace analyzer for VoIP Used for obtaining Objective Ratings for various network settings NISTnet – a network emulator tool To create realistic network scenarios by introducing Delay, Jitter and Loss Spirent SmartBits – a commercial network measurement suite Used to qualify NISTnet NLANR Iperf – a TCP/UDP bandwidth measurement tool Network path characteristics determination jaalaM appareNet – a commercial network measurement suite Polycom FX Videoconferencing Station – Audio/Video clips source www.oar.net

H.323 Beacon Screenshots… www.oar.net

H.323 Beacon Screenshots… (Contd.) www.oar.net

H.323 Beacon Screenshots… (Contd.) http://www.itecohio.org/beacon www.oar.net

Internet Tests Methodology… Design of Experiments → 2 Phase Approach Phase I: LAN with WAN Emulation Tests Full factorial (n**3 =27; n=3) Tasks with various network health scenario configurations on NISTnet Phase II: Internet Tests 9 equivalent tasks out of possible 27 tasks + 3 H.323 Beacon tasks using the loopback feature End-to-End delay, jitter and loss values configured on NISTnet for each Internet test site = (LAN NISTnet Settings) – (Inherent Path Characteristics for corresponding Internet test site) G G G A A A P P P G G A G A G A G G G G P G P G P G G BG BG BG BA BA BA BP BP BP G → Good; A → Acceptable; P → Poor; B(G/A/P) → Beacon Task www.oar.net

Results: Subjective and Objective MOS Vs Delay Pearson correlation co-efficient for Subjective and Objective Scores for Delay = 0.827 www.oar.net

Results: Subjective and Objective MOS Vs Jitter Pearson correlation co-efficient for Subjective and Objective Scores for Jitter = 0.737 www.oar.net

Results: Subjective and Objective MOS Vs Loss Pearson correlation co-efficient for Subjective and Objective Scores for Loss = 0.712 www.oar.net

Overall Results of Quality Grade Assessments for LAN/Internet Tests Delay G A P Jitter Loss Result A/P G*/A A*/P G → Good; A → Acceptable; P → Poor; S1 - S9 → Scenario 1-9 * next to a grade → end-user will more often perceive that Grade of quality in that particular scenario www.oar.net

Results: Effects of Normalized Delay, Jitter and Loss variations on Subjective MOS Normalized Scale: 1 Unit → 150ms Delay; 20ms Jitter; 0.5% Loss www.oar.net

Results: Effects of Normalized Delay, Jitter and Loss variations on Objective MOS End-user perception of audiovisual quality is more sensitive to changes in jitter than to changes in delay and loss www.oar.net

Conclusions We have determined performance bounds for latency, jitter and loss by conducting exhaustive testing in a LAN environment with WAN Emulation By using many incremental values of each of the 3 metrics meticulously, we determined what ranges of these values in isolation and in combination affect the user perception of audiovisual quality as "good", "acceptable" and "poor" grades We have demonstrated our theory of bounds developed in a LAN with WAN emulation, actually holds good in the “Internet” By using literally every type of last mile connection and by testing with sites all over the world, we proved our LAN results scale consistently to the Internet www.oar.net

Conclusions (Contd.) We have observed the H.323 Beacon loopback assessments (again both subjective and objective) to be in accord with the audiovisual tasks test results This demonstrates the handy utility of the H.323 Beacon in determining user perceived audiovisual quality in H.323 production systems without remote end-user intervention Our normalized results indicate that end-user perception of audiovisual quality is more sensitive to changes in jitter than to changes in delay and loss www.oar.net

Questions www.oar.net