1. International Telecommunication Union Workshop on End-to-End Quality of Service.What is it? How do we get it? Geneva, 1-3 October 2003 Are Existing Performance Metrics Adequate? Alan Clark CEO, Telchemy

2. ITU-T 2 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Outline o IP Performance Metrics o Network Characteristics o Packet Loss Characteristics and Measurement o Jitter Characteristics and Measurement o Recommendations

3. ITU-T 3 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Typical IP Performance Metrics o Packet Loss Equipment problems, misrouting, buffer overflow, transmission errors o Jitter Network congestion, contention and queuing delays o Delay Transmission delays, quasi-stable congestion levels

4. ITU-T 4 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Core IP Network characteristics o Class A networks High capacity optical fiber, high throughput routers Very low jitter and packet loss, occasional link failures o Class B networks T1/E1 trunk connections Significant jitter and loss due to network congestion

5. ITU-T 5 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Corporate IP Networks o Moving to 100BaseT Switched Ethernet but still some 10BaseT and Hubs o Access links often T1/E1/PRI, sometimes fractional o Mixture of digital leased line, frame relay and IP VPN o Small offices and teleworkers may use low bandwidth links

6. ITU-T 6 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Teleworkers and Residential IP o Low-mid bandwidth DSL and Cable Modem connections with little or no QoS control o Often 10BaseT and IEEE802.11 LAN o Can experience heavy usage of access links leading to high levels of jitter

7. ITU-T 7 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Packet Loss o Specifically, packets lost within the network or discarded before reaching the network API in the receiving system.

8. ITU-T 8 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Example packet loss distribution

9. ITU-T 9 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Example packet loss distribution 20% Loss Rate Consecutive Loss

10. ITU-T 10 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Consecutive Loss Distribution

11. ITU-T 11 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Lessons learned from analyzing real world traces o Packets are usually lost singly – much less frequent to lose 2-3 consecutive packets o Occasional very long consecutive loss periods due to link failure o Much more common to see high loss periods – seconds in length – with loss densities of 30% o Measurement of packet loss needs to be based around a burst model – e.g. Gilbert-Elliott, Markov Model with 3 or more states….

12. ITU-T 12 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Jitter o Specifically – short term variations in transmission delay o Real time systems often use a jitter buffer to remove jitter – but increases delay and packet loss o The impact on performance is due to the combined effect of jitter and the jitter buffer

13. ITU-T 13 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Example time series plot of packet delay

14. ITU-T 14 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Jitter Characteristics What should we measure? - Packet to packet delay? - Absolute delay? - Delay with respect to ?

15. ITU-T 15 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Jitter Characteristics What really counts is what the end system would do as a result of delay variations Lesson: estimate (or count) which packets would be discarded due to jitter

16. ITU-T 16 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Timing Drift Can occur between end systems and between end and measuring system. Effect on VoIP is usually small however can cause significant measurement errors for delay and jitter

17. ITU-T 17 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Estimating the effects of jitter Jitter Buffer Emulator Discard

18. ITU-T 18 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Measuring the combined effects of jitter and loss - midpoint Jitter Buffer Emulator Loss and Discard Events Burst Loss Model

19. ITU-T 19 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Measuring the combined effects of jitter and loss - endpoint DSP – Jitter Buffer Vocoder, Echo Canceller Burst Metrics API - percent lost - percent discarded - gap length/density - burst length/density

20. ITU-T 20 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Delay o Mid-stream systems often rely on RTCP to estimate round trip delay RTCP is often not implemented by end systems RTCP delay is only the VoIP packet path delay and does not include external circuit switched path delays

21. ITU-T 21 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Sources of delay Frame accumulation and encoding RTP transmission delay Receiving Jitter Buffer Decoding, PLC, playout External End End System Delays can be more substantial than transmission delay, particularly with high jitter levels and adaptive jitter buffers

22. ITU-T 22 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? RTCP XR VoIP metrics o RTCP XR reports:- Proportion of packets discarded Proportion of packets lost Length and density of bursts (periods of high loss/discard density) Length and density of gaps End system delay Packet path delay and more…………………

23. ITU-T 23 1-3 October 2003 Workshop on End-to-End Quality of Service. What is it? How do we get it? Recommendations o Understand that packet path properties are time varying o Dont measure jitter but do:- Measure packet discards resulting from jitter Count discards as lost packets o Use a proper model for packet loss distribution (e.g. Markov model with 3+ states, Gilbert-Elliott model…) o Incorporate end system delays – if known o Use RTCP XR!!!! (as this implements the above)