1. Automated SIP Interoperability Testing Archana Rao and Henning Schulzrinne Department of Computer Science Columbia University http://www.cs.columbia.edu/IRT/voip-testbed/ SIP 2009 (Paris, January 2009)

2. Overview The problem of SIP interoperability Far from perfect Damages “brand”, harms customers, encourages single-vendor deployments Testbed Overview Architecture Components Experiments Interoperability study End-client characteristics Summary

3. Interoperability Hi Henning, You may remember me from IETFs past -- I haven't attended any in some time because I couldn't find any really interesting projects. I'm finally getting into SIP. I've got Speakeasy VoIP service, two sipphone accounts, a Cisco 7960 and a copy of x-ten on my Mac. And I still can't make it work. Voice flows in one direction only. I'm not even behind a NAT or firewall -- both machines have global addresses, with no port translations or firewalls. I've been working with Internet protocols for over 20 years. I've implemented and contributed to them. And if *I* can't figure out how to make this stuff work, how is the average grandmother expected to do so? SIP is unbelievably complex, with extraordinarily confusing terms. There must be half a dozen different "names" -- Display Name, User Name, Authorization User Name, etc -- and a dozen "proxies". Even the word "domain" is overloaded a half dozen different ways. This is ridiculous! Sorry. I just had to get this off my chest. Regards, Phil Karn. Is it a real problem ? Reference: http://www1.ietf.org/mail-archive/web/rai/current/msg00082.html Excerpts from an email posted on IETF RAI mailing list –

4. Interoperability issues Why do we have interoperability problems? SIP is complex More than 150 RFCs and 500 active Internet Drafts Efforts like – “draft-ieft-sip-hitchhikers-guide” help to a certain extent, but not sufficient SIP has no proposed architecture/profile SIP usage in variety of environments and domains make it impossible Standardization from IETF is different from that of ISO/ITU SIP is continually evolving Non trivial for implementers to follow the life-cycle from IDs to RFCs Walled gardens, non-standard environments Vendors need to make products that work in their customer’s environment Poor implementations, intentional non-interoperability etc.

5. Improving interoperability Current efforts to combat SIP interoperability issues SIP Interoperability (SIPit) events Week long gatherings of SIP implementers to test interoperability Coordinated by SIP Forum SIP Forum SIP Connect 1.0 (and 1.1)  outsourced enterprise VoIP IETF Basic Level of Interoperability for SIP Services (BLISS) Focus on resolving interpretability issues in SIP features Line sharing, call parking, automated call handling and call queuing University of New Hampshire – InterOperability Laboratory Vendor-neutral lab dedicated to testing data networking technologies 20 different testing programs, each costing about $20,000 per year

6. Interoperability How do we go about it? 1. Identify the basic real-world scenarios for SIP registration and session establishment 2. Use our VoIP testbed to configure a variety of SIP infrastructure 3. Study the behavior of SIP end-clients and servers for interoperability So, isn’t the problem solved? These forums focus on details of advanced SIP features Basic-level of interoperability between innumerable SIP devices in the market Can I take any SIP phone and make a VoIP call, through any SIP service provider?

7. Studying interoperability 1.Define a minimum set of call flows constituting basic interoperability RFC 3665, RFC 3666, RFC 4317 2.Use a VoIP testbed to realize a variety of real-world SIP infrastructure setups Columbia VoIP testbed 3.Study the behavior of SIP end-clients and servers in each of these scenarios Interoperability matrices Categorization of common issues  Lack of clarity in the specification Implementation of an older specification Incomplete implementation of the specification Incorrect implementation of the specification Failure against robustness tests

8. Columbia VoIP testbed What? VoIP infrastructure for experimentation, analysis, testing, prototyping and deployment of SIP/VoIP components in a variety of environments. Part of a multi-university research project supported by NSF University of North Texas, Purdue University and University of California, Davis Why? There’re no telecom testbeds available for research & experimentation VoIP implementations & experiments have lacked scientific rigor Continuously emerging standards Need to develop repeatable, accurate test frameworks Realistic VoIP experiments require a distributed testbed

9. your equipment here :-) http://www.cs.columbia.edu/IRT/voip-testbed/

10. Columbia VoIP testbed: servers 5 SIP servers Asterisk, Pbxnsip, Sipd, SER, OpenSER 3 Platforms Microsoft Windows XP Linux Fedora Core 6 Sun Solaris X 3-way Connectivity the Internet VPN PSTN

11. Columbia VoIP testbed: UAs 20+ SIP hard-phones, soft-phones, WI-FI phones with different capabilities More details – www.columbia.edu/~ahr2114/VoIPtestbed/end-clientswww.columbia.edu/~ahr2114/VoIPtestbed/end-clients

12. NSF VoIP testbed TheInternet UNIVERSITY-AUNIVERSITY-B PSTNNetwork UNIVERSITY-C UNIVERSITY-D Image courtesy: http://secnet.csci.unt.edu/7.15.funding_application.ppt

13. Columbia VoIP testbed experiments Planned or ongoing Interoperability study Can any two SIP devices talk to each other? SIP end-client characteristics Signaling and codec features in SIP devices Robustness VoIP with/despite NAT Performance issues Signaling delay, voice mouth-ear delay, packet loss resilience Servers – Scalability, maximum capacity Security Separate testbed for DOS and other attacks work in progress

14. Interoperability study: call flows Registration (RFC 3665) Successful new registration Unsuccessful registration Cancellation of registration Update of contact list Session Establishment (RFC 3665) Successful session establishment Session establishment through two proxies Session establishment with multiple proxy authentication Successful session with proxy failure Unsuccessful no answer Unsuccessful busy Unsuccessful no response from user agent Unsuccessful temporarily unavailable Codec Negotiation (RFC 4317) Audio and DTMF Audio, video and DTMF Audio and video codec reordering

15. Interoperability study: matrix Interoperability Matrix Test case #1 - Successful new registration Asterisk3ComsipdSER Eyebeam 1.5Yes WengophoneYesNo Yes Polycom PVXYesNo Yes Express Talk 3.0Yes Cisco 7940Yes Grandstream GXV 3000Yes Snom 320Yes Polycom VSX 7000eYesNo Yes

16. Interoperability study: auth name Issue 1 – Use of different formats for authentication name Authorization: Digest username=“user1”,realm=“proxy01.sipphone.com”,nonce=“xxx”, uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5 Authorization: Digest username=“sip:user1@sipphone.com”, realm=“proxy01.sipphone. com”,nonce=“xxx”,uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5 Implication Registration and call setup failure, if both formats are not supported by the UAs Instances Polycom PVX/VSX, Wengophone don’t register with 3Com / sipd proxies Workaround Use Asterisk between Polycom PVX and 3Com proxy

17. Interoperability study: rport Implication Incorrect/incomplete signaling resulting in multiple retransmissions and failed transaction Instances Calls to Cisco 7940 via Sipphone proxy are always unsuccessful Calls from Polycom VSX via 3com or sipd proxies are always unsuccessful Workaround Nothing Issue 2 – Behavior in the absence of rport parameter SIP serverUAC INVITE (Dest port:5060) 180 Ringing (Src port:12345) 200 OK (Src port:12345) ACK (Dest port:12345) 200 OK (Src port:12345)

18. Interoperability Study: codecs Implication – Audio/Video failure (mostly unidirectional, but bidirectional sometimes) Problem Instances Audio failure when Cisco calls voicemail service of sipphone provider Video failure when Polycom PVX calls Xlite via Asterisk Issue 3 – Codec Negotiation [Offer] v=0 o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67 s= SIP Call t=0 0 m=audio 31030 RTP/AVP 0 8 18 101 c=IN IP4 128.59.17.67 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:18 G729/0 a=fmtp:18 annexb=no a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=sendrecv [Answer] v=0 o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67 s= SIP Call t=0 0 m=audio 31030 RTP/AVP 18 0 8 101 c=IN IP4 128.59.17.67 a=rtpmap:18 G729/0 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=sendrecv

19. UA characteristics: features Basic Configuration, number of lines, call hold, call forwarding, DND, NAT support Advanced Encryption, symmetric RTP, conferencing, audio/video codecs, PoE Audio quality – silence suppression, echo cancellation, comfort noise Transport, signaling and support protocols – TCP, TLS, HTTP, DHCP, DNS, TFTP, NTP More details at – http://www.columbia.edu/~ahr2114/VoIPTestbed/EndClients.htm http://www.columbia.edu/~ahr2114/VoIPTestbed/EndClients.htm

20. UA characteristics: robustness SIP Torture Tests (RFC4475) A set of SIP messages aimed at stressing the SIP parser, based on experiences at SIPit. PROTOS Test suite with 4500+ INVITE requests, developed by University of Oulu Robustness SIP end- clients PROTOS Test- cases Behavior WengoPhoneGeneralCPU consumption is very high (varies between 50% and 90%), throughout the testing. SIPc368, 369Software crashes (with a message “Hostname variable too long”) ExpressTalk GeneralSoftware crashes, if an INVITE is received, when all the 6 lines are busy. 2363, 2366, 2371, 2374, 2383 – 2406 Software crashes (can’t handle improper values for “Content-length”) Example Results

21. UA characteristics: NATs SIP and NAT Problematic for both signaling (SIP) and media transfer (RTP) Circumventing NAT effect – solutions exist for both end-clients and proxies Quest to overcome this, could lead to techniques uglier than NAT itself! NAT test setup NAT realization - Linux iptables, CLICK router Studying the behavior in different scenarios NAT support enabled in both the proxy server and end-client NAT support enabled in only one of them etc Aiming to analyze the existing practices – the good, the bad and the ugly!

22. UA with NATs SIP devices in NAT environment

23. What now? Summary Tests indicate that basic-level interoperability is far from perfect Instances of interoperability failures illustrated in our IETF workshop paper Achieving Interoperability shouldn’t be left just to vendors non-interoperability damages SIP brand, causes grief for third parties and harms customers Going ahead – some ideas Establish designated interop liaison for each vendor calling customer support unlikely to be useful Encourage vendors to publish interoperability reports in standard format Provide self-certification tests, supported by a remotely accessible test rig

24. Summary Current status The VoIP testbed & VPN connectivity is operational Systematic study of real-world interoperability issues Basic-level interoperability is nowhere near 100% IETF 70 – The 1 st SIP Forum Workshop on SIP Interoperability Evaluating SIP end client characteristics Testbed plans Experiments on NAT, performance, security Make the testbed accessible and useful for engineers & deployments Create a knowledge repository about SIP devices & tools for interoperability testing