Horizontal Scaling and Geographic Distribution of OnSIP John Riordan, CTO Junction Networks August 9, 2011

Introduction to Junction Networks Mike Oeth, CEO Junction Networks was founded in 2004 on the premise that the future of business communications would rely heavily on IP Technologies. Junction Networks is privately owned, with no outside investors. John Riordan is CTO and lead of the Engineering Team at Junction Networks.  This slide is for Mike to expand a bit on the JN background, vision, and John. John first got involved with computers in the late 70′s. He studied computer science and cognitive science at MIT in the late 80′s. Since time, he’s been an Internet entrepreneur and engineer, building an Internet Service Provider in New York City, Interport Communications, before it was purchased by RCN in the late 1990's and then co-founding Junction Networks in 2004. Horizontal Scaling and Geographic Distribution of OnSIP  Page 2

Abstract OnSIP is a hosted SIP and XMMP service targeted at small and medium sized businesses. The hosted SIP platform presents itself as a single outbound proxy to end users who utilize standards based user agents to register one or more contacts for an AOR which they select. Horizontal scalability across an undefined number of geographically disparate data centers was one of the goals in the development of the current platform. Many open source projects such as OpenSIPS, FreeSWITCH and Asterisk are used in the implementation of the current platform. The "core" of the current SIP production implementation consists of a distributed set of dialog stateful SIP proxies based on OpenSIPS. State information in this implementation is maintained and propagated between proxies via integration with Cassandra, MySQL, and multicast event notifications. User agents are affiliated via Geo-DNS to a home proxy with which they register (Path headers). User agents can transition from one home proxy to another with no interruption of service. NAT issues are handled server side via message rewriting with the support of RTP proxies and keep-alives. To support sequential and future out of dialog requests, contacts which are not globally routable are mapped in initial requests and responses to GRUU's. The architecture and implementation of this distributed proxy core will be reviewed. Horizontal Scaling and Geographic Distribution of OnSIP  Page 3

Introduction to OnSIP Hosted SIP and XMPP service targeted at small and medium sized businesses. OnSIP Platform Marketed as Hosted PBX service VM, ACD, AA, MOH, E911, Conferencing PSTN IN/OUT Integrated XMPP based IM and Presence Web based user interfaces XML API XMPP API Horizontal Scaling and Geographic Distribution of OnSIP  Page 4

Introduction to OnSIP Hosted SIP and XMPP service targeted at small and medium sized businesses. The hosted SIP platform presents itself as a single outbound proxy to end users who utilize user agents to register one or more contacts for an AOR that they select. End users are assigned globally reachable SIP AORs (e.g. sip:john@junctionnetworks.com) Any standards compliant user agent may be utilized Many open source projects such as OpenSIPS, FreeSWITCH and Asterisk are used in the implementation of the current platform. Horizontal Scaling and Geographic Distribution of OnSIP  Page 5

Horizontal Scaling OnSIP platform is built with horizontal scalability across an undefined number of geographically disparate data centers Horizontal vs Vertical Scaling Vertical scaling: buying bigger and bigger hardware to run the same software, thus giving you more resources to use. Horizontal scaling: buying many of the same type of smaller hardware and distribute load. General Issues SIP-specific Issues Geographic distribution Out of dialog requests Maintaining global state NAT traversal Horizontal Scaling and Geographic Distribution of OnSIP  Page 6

Geographic Distribution New York Core Proxies App Application Servers Los Angeles Internet Miami London sip.onsip.com UA Geographic Distribution GeoDNS directs UAs to closest service location UAs for same user can register with different locations UAs can switch locations at anytime without disruption Horizontal Scaling and Geographic Distribution of OnSIP  Page 7

Core Proxies Internet New York Los Angeles Read Write Read Only OpenSIPS Proxies sip.onsip.com Cassandra Nodes New York Los Angeles MySQL Slaves PubSubContact Dialog Events Read Write Read Only Write Only Proxy Local Horizontal Scaling and Geographic Distribution of OnSIP  Page 8

Core Proxies Internet Read Write Read Only Read Write Write Only sip.onsip.com sip.onsip.com OpenSIPS Proxies OpenSIPS Proxies Cassandra Nodes Cassandra Nodes Read Write MySQL Slaves MySQL Slaves Read Only MySQL Proxy Local MySQL Proxy Local Read Write PubSubContact Dialog Events PubSubContact Dialog Events Write Only New York Los Angeles Horizontal Scaling and Geographic Distribution of OnSIP  Page 9

OpenSIPS sip.onsip.com Outbound proxy for user agents GeoDNS Home proxy for inbound requests NAT detection and handling GRUU mapping of contacts Dialog stateful Forking The "core" of the current SIP production implementation consists of a distributed set of dialog stateful SIP proxies based on OpenSIPS. Horizontal Scaling and Geographic Distribution of OnSIP  Page 10

GeoDNS A record for sip.onsip.com ISC Bind customization Response varies based on source IP of request IP2Location database Based on kernel.org paper* “GeoDNS is the idea of taking an incoming DNS request, doing the geographic look-up at the request time, and returning different results based on the incoming IP ad- dress.” – Hawley* Hawley, John. GeoDNS—Geographically-aware, protocol-agnostic load balancing at the DNS level. Linux Foundation: 2009. www.kernel.org/doc/ols/2009/ols2009-pages-123-130.pdf Horizontal Scaling and Geographic Distribution of OnSIP  Page 11

NAT Handling Contact and SDP rewriting REGISTER NAT info stored in location database Path information is stored on registration INVITE and other requests NAT info carried in Route header UAC behind a NAT UAS behind a NAT UAC and UAS behind same NAT When a proxy receives an INVITE request for an AOR, it will do a lookup through the user location service provided by the registrar. Horizontal Scaling and Geographic Distribution of OnSIP  Page 12

GRUU Mapping Out of dialog Request to Contact URI Globally Routable UA URU (GRUU) Out of dialog Request to Contact URI i.e. REFER & INVITE w/Replaces On Ingress of Request/Response Contact URI is translated to a GRUU GRUU hostport references proxy hostport On Egress of Request RURI GRUU translated Contact URI Horizontal Scaling and Geographic Distribution of OnSIP  Page 13

Core Proxies Internet Read Write Read Only Read Write Write Only sip.onsip.com sip.onsip.com OpenSIPS Proxies OpenSIPS Proxies Cassandra Nodes Cassandra Nodes Read Write MySQL Slaves MySQL Slaves Read Only MySQL Proxy Local MySQL Proxy Local Read Write PubSubContact Dialog Events PubSubContact Dialog Events Write Only New York Los Angeles Horizontal Scaling and Geographic Distribution of OnSIP  Page 14

Cassandra Open source column oriented DBMS Distributed key-value store A highly scalable, second-generation distributed database Open source column oriented DBMS initially developed by Facebook Distributed key-value store Eventual Consistency Google BigTable data model Amazon Dynamo like infrastructure Expiring Columns (TTL) User location data custom OpenSIPS db module Path header Registration structure Cassandra Nodes Horizontal Scaling and Geographic Distribution of OnSIP  Page 15

CAP theorem Consistency Availability Partition tolerance (can only have 2 out of 3 of the following) Consistency all nodes see the same data at the same time Availability a guarantee that every request receives a response about whether it was successful or failed Partition tolerance the system continues to operate despite arbitrary message loss Horizontal Scaling and Geographic Distribution of OnSIP  Page 16

Core Proxies Internet Read Write Read Only Read Write Write Only sip.onsip.com sip.onsip.com OpenSIPS Proxies OpenSIPS Proxies Cassandra Nodes Cassandra Nodes Read Write MySQL Slaves MySQL Slaves Read Only MySQL Proxy Local MySQL Proxy Local Read Write PubSubContact Dialog Events PubSubContact Dialog Events Write Only New York Los Angeles Horizontal Scaling and Geographic Distribution of OnSIP  Page 17

Slaved to master over Internet Addresses MySQL Slaves Slaved to master over Internet ssh reverse tunneling Addresses CPL Aliases PBX Extensions E.164 Numbers Domains MySQL Slaves Horizontal Scaling and Geographic Distribution of OnSIP  Page 18

Core Proxies Internet Read Write Read Only Read Write Write Only sip.onsip.com sip.onsip.com OpenSIPS Proxies OpenSIPS Proxies Cassandra Nodes Cassandra Nodes Read Write MySQL Slaves MySQL Slaves Read Only MySQL Proxy Local MySQL Proxy Local Read Write PubSubContact Dialog Events PubSubContact Dialog Events Write Only New York Los Angeles Horizontal Scaling and Geographic Distribution of OnSIP  Page 19

MySQL Proxy Local Regular OpenSIPS database tables Dialog state System local instance Dialog state dialog module Location state custom OpenSIPS db module usrloc module in support of NAT keep-alive MySQL Proxy Local Horizontal Scaling and Geographic Distribution of OnSIP  Page 20

Core Proxies Internet Read Write Read Only Read Write Write Only sip.onsip.com sip.onsip.com OpenSIPS Proxies OpenSIPS Proxies Cassandra Nodes Cassandra Nodes Read Write MySQL Slaves MySQL Slaves Read Only MySQL Proxy Local MySQL Proxy Local Read Write PubSubContact Dialog Events PubSubContact Dialog Events Write Only New York Los Angeles Horizontal Scaling and Geographic Distribution of OnSIP  Page 21

PubSub XMPP Publish-Subscribe (XEP-0060) User Agents Active Calls Contact dialog events XMPP Publish-Subscribe (XEP-0060) User Agents usrloc module events (customized module) Active Calls dialog/tm module events (customized module) Application level events (voicemail) Recent call history PubSub Horizontal Scaling and Geographic Distribution of OnSIP  Page 22

Thank You Contact developer@junctionnetworks.com OnSIP API Visit onsip.com/api Horizontal Scaling and Geographic Distribution of OnSIP  Page 23