Quality and Performance Evaluation of VoIP End-points Wenyu Jiang Henning Schulzrinne Columbia University NYMAN 2002 September 3, 2002

Motivations The quality of VoIP depends on both the network and the end-points Extensive QoS literature on network performance, e.g., IntServ, DiffServ Focus is on limiting network loss & delay Little is known about the behavior of VoIP end-points

Performance Metrics for VoIP End-points Mouth-to-ear (M2E) delay C.f. network delay Clock skew Whether it causes any voice glitches Amount of clock drift Silence suppression behavior Whether the voice is clipped (depends much on hangover time) Robustness to non-speech input, e.g., music Robustness to packet loss Voice quality under packet loss

Measurement Approach Capture both original and output audio Use “adelay” program to measure M2E delay Assume a LAN environment by default Serve as a baseline of reference, or lower bound

VoIP End-points Tested Hardware End-points Cisco, 3Com and Pingtel IP phones Mediatrix 1-line SIP/PSTN Gateway Software clients Microsoft Messenger, NetMeeting (Win2K, WinXP) Net2Phone (NT, Win2K, Win98) Sipc (Solaris, Ultra-10) Operating parameters: In most cases, codec is G.711  -law, packet interval is 20ms

Example M2E Delay Plot 3Com to Cisco, shown with gaps > 1sec Delay adjustments correlate with gaps, despite 3Com phone has no silence suppression

Visual Illustration of M2E Delay Drop, Snapshot #1 3Com to Cisco 1-1 case Left/upper channel is original audio Highlighted section shows M2E delay (59ms)

Snapshot #2 M2E delay drops to 49ms, at time of 4:16

Snapshot #3 Presence of a gap during the delay change

Effect of RTP M-bits on Delay Adjustments Cisco phone sends M-bits, whereas Pingtel phone does not Presence of M-bits results in more adjustments

Sender Characteristics Certain senders may introduce delay spikes, despite operating on a LAN

Average M2E Delays for IP phones and sipc Averaging the M2E delay allows more compact presentation of end-point behaviors Receiver (especially sipc) plays an important role in M2E delay

Average M2E Delays for PC Software Clients Messenger 2000 wins the day Its delay as receiver (68ms) is even lower than Messenger XP, on the same hardware It also results in slightly lower delay as sender NetMeeting is a lot worse (> 400ms) Messenger’s delay performance is similar to or better than a GSM mobile phone. AB ABABBABA MgrXP (pc)MgrXP (notebook)109ms120ms Mgr2K (pc)96.8ms68.5ms NM2K (pc)NM2K (notebook)401ms421ms Mobile (GSM)PSTN (local number)115ms109ms

Delay Behaviors for PC Clients, contd. Net2Phone’s delay is also high ~ ms V1.5 reduces PC->PSTN delay PC-to-PC calls have fairly high delays AB ABABBABA N2P v1.1 NT P-2 (pc2)PSTN (local number) 292ms372ms N2P v1.5 NT P-2 (pc2)201ms373ms N2P v1.5 W2K K7 (pc)196ms401ms N2P v1.5 W2K K7 (pc)N2P v1.5 W98 P-3 (notebook2) 525ms350ms

Effect of Clock Skew: Cisco to 3Com, Experiment 1-1 Symptom of playout buffer underflow Waveforms are dropped Occurred at point of delay adjustment Bugs in software?

Clock Drift Rates Mostly symmetric between two devices Sipc has unusually high drift rates, > 300 ppm (parts per million) Drift Rates (in ppm) 3ComCiscoMediatrixPingtelsipc 3Com Cisco Mediatrix Pingtel sipc

Drift Rates for PC Clients Drift Rates not always symmetric! But appears to be consistent between Messenger 2K/XP and Net2Phone on the same PC Existence of 2 clocking circuits in sound card? AB ABABBABA MgrXP (pc)MgrXP (notebook) Mgr2K (pc) NM2K (pc)NM2K (notebook)?-33? Net2Phone NT (pc2)PSTN Net2Phone 2K (pc)16682 Mobile (GSM)00

Packet Loss Concealment Common PLC methods Silence substitution (worst) Packet repetition, with optional fading Extrapolation (one-sided) Interpolation (two-sided), best quality Use deterministic bursty loss pattern 3/100 means 3 consecutive losses out of every 100 packets Easier to locate packet losses Tested 1/100, 3/100, 1/20, 5/100, etc.

PLC Behaviors Loss tolerance (at 20ms interval) By measuring loss-induced gaps in output audio 3Com and Pingtel phones: 2 packet losses Cisco phone: 3 packet losses Level of audio distortion by packet loss Inaudible at 1/100 for all 3 phones Inaudible at 3/100 and 1/20 for Cisco phone, yet audible to very audible for the other two. Cisco phone is the most robust Probably uses interpolation

Effect of PLC on Delay No affirmative effect on M2E delay E.g., sipc to Pingtel

Silence Suppression Why? Saves bandwidth May reduce processing power (e.g., in conferencing mixer) Facilitates per-talkspurt delay adjustment Key parameters Silence detection threshold Hangover time, to delay silence suppression and avoid end clipping of speech Usually 200ms is long enough [Brady ’68]

Hangover Time Measured by feeding ON-OFF waveforms and monitor RTP packets Cisco phone’s is the longest ( sec), then Messenger ( sec), then NetMeeting ( sec) A long hangover time is not necessarily bad, as it reduces voice clipping Indeed, no unnatural gaps are found Does waste a bit more bandwidth

Robustness of Silence Detectors to Music On-hold music is often used in customer support centers Need to ensure music is played without any interruption due to silence suppression Tested with a 2.5-min long soundtrack Messenger starts to generate many unwanted gaps at input level of -24dB Cisco phone is more robust, but still fails from input level of -41.4dB

M2E Delay under Jitter Delay properties under the LAN environment serves as a baseline of reference When operating over the Internet: Fixed portion of delay adds to M2E delay as a constant Variable portion (jitter) has a more complex effect Preliminary results Used typical cable modem delay traces Tested sipc to Cisco No audible distortion due to late loss Added delay is normal

Conclusions Average M2E delay: Low (mostly < 80ms) for hardware IP phones Software clients: low (< 120ms) for Messenger, particularly Messenger 2000 (68.5ms) The application (receiver) is the most vital in determining delay Poorly implemented end-points can easily undo good network QoS Clock drift rates Are high for some software clients (sipc, Net2Phone) In some cases non-symmetric! Packet loss concealment quality Acceptable in all 3 IP phones tested, w. Cisco being most robust Silence detectors in Cisco phone, Messenger, NetMeeting Long hangover time, no audible unwanted gaps for speech input May falsely predict music as silence

Future Work Further tests with more end-points on how jitter influences M2E delay Measure the sensitivity (threshold) of various silence detectors Investigate the non-symmetric clock drift phenomena Additional experiments as more brands of VoIP end-points become available