1. 2001-03-19 ETX/D/XPS-01:002 Uen 1 Ulf Wiger Senior System Architect Strategic Product & System Management Ericsson Telecom AB Data Backbone & Optical Networks Division Four-Fold Increase in Productivity and Quality —Industrial-Strength Functional Programming in Telecom-Class Products (http://www.erlang.se/publications/Ulf_Wiger.pdf) FEmSys 2001 Deployment on distributed architectures

2. 2001-03-19 ETX/D/XPS-01:002 Uen 2 The setting 1995 PC Week Study of software projects: –16% successful –53% operational (but less than successful) –31% cancelled Butler Group 1997 on large software projects –5 out of 6 large projects fail –In >$300M companies, 9 out of 10 large projects fail How to approach this? –Use high-level modeling tools & generate code? –Raise the level of programming language? –Fight all causes for project failure!

3. 2001-03-19 ETX/D/XPS-01:002 Uen 3 The design of the AXD 301 switch Situation 1996 –Late entry into the ATM market –Needed to leapfrog the competition –Project members used to "traditional" large projects, wanted to try something new –No interesting 3rd party middleware platform Target: a backbone ATM switch with "Telecom Profile"

4. 2001-03-19 ETX/D/XPS-01:002 Uen 4 Executive summary AXD 301: A Telephony-Class, scalable (10-160 GBps) ATM switch designed from scratch in less than 3 years AXD 301 Success factors: –Highly pragmatic, holistic approach –Competent organisation –Efficient process –Excellent technology (e.g. Erlang/OTP) More than just technology... –Consider all factors together from the start –Erlang was a perfect match for our approach

5. 2001-03-19 ETX/D/XPS-01:002 Uen 5 ENGINE: Migrating today's vertical networks into a single multi-service backbone AXD 301 in the marketplace Central component in Ericsson's ENGINE offering Several major operators –British Telecom –Vodaphone –Worldcom –Telia –Diveo –...

6. 2001-03-19 ETX/D/XPS-01:002 Uen 6 Briefly about the term Carrier-Class To us, "Carrier-Class", "Telephony-Class" and "Telecom Profile" are synonymous The quality we've come to expect from public telephony networks The trend towards multimedia services requires Carrier-Class in more systems More than just duplication of hardware: –Fault-tolerant software –In-service hardware expansion –In-service software upgrade –Load tolerance –Flexibility (frequent changes + long service life) Target: 99,999% ("five nines") availability, including planned outages There's no such thing as "almost Carrier-Class"!

7. 2001-03-19 ETX/D/XPS-01:002 Uen 7 Optional Processors Mandatory Mated Processor Pair ATM Control Processors L3F CE Switch Core FR ATB Server Device Line Termination ATM Termination Clock & Synchronization Clock & Synchronization IO CP Device Processor on Each Board Telecom-Class System Architecture simple wire-speed logiccomplex soft-real-time logic

8. 2001-03-19 ETX/D/XPS-01:002 Uen 8 Fault tolerance: distributed recovery Failover: one processor dies; its mate restarts the lost applications Takeover: the failed processor returns; the O&M/Standby applications migrate O&M or Standby application Call-handling application "Luke-warm" standby sufficient to maintain quality of service

9. 2001-03-19 ETX/D/XPS-01:002 Uen 9 Call Troughput in % of maximum sustainable call handling (set-up + release) capacity Offered Load in % of maximum call troughput 100% 40% 100% 150% 1000% Rejected Calls 95% 115 call/s sustained 200 call set-up/s or Load from handling of rejected calls Load tolerance Call Handling Throughput for one control processor AXD 301 release 3.2 Traffic Case: ATM SVC UNI to UNI

10. 2001-03-19 ETX/D/XPS-01:002 Uen 10 History of Erlang 1984: Ericsson Computer Science Lab formed 1984-86: Experiments programming POTS with several languages 1998: Open Source Erlang 1987: Early Erlang Prototype projects 1991: First fast implementation 1993: Distributed Erlang 1995: Several new projects 1996: Open Telecom Platform (research on verification...) No language well suited for telecom systems development

11. 2001-03-19 ETX/D/XPS-01:002 Uen 11 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Functional programming language High abstraction level Pattern matching Concise readable programs

12. 2001-03-19 ETX/D/XPS-01:002 Uen 12 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Either transparent or explicit concurrency Light-weight processes Highly scalable

13. 2001-03-19 ETX/D/XPS-01:002 Uen 13 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Response times in the order of milliseconds Per-process garbage collection

14. 2001-03-19 ETX/D/XPS-01:002 Uen 14 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Simple and consistent error recovery Supervision hierarchies "Program for the correct case"

15. 2001-03-19 ETX/D/XPS-01:002 Uen 15 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Explicit or transparent distribution Network-aware runtime system

16. 2001-03-19 ETX/D/XPS-01:002 Uen 16 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Easily change code in a running system Enables non-stop operation Simplifies testing

17. 2001-03-19 ETX/D/XPS-01:002 Uen 17 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability "Ports" to the outside world behave as Erlang processes

18. 2001-03-19 ETX/D/XPS-01:002 Uen 18 Erlang Highlights Declarative Concurrency Soft real-time Robustness Distribution Hot code loading External interfaces Portability Erlang runs on any UNIX, Windows, VxWorks,... Supports heterogeneous networks

19. 2001-03-19 ETX/D/XPS-01:002 Uen 19 Erlang Examples 1 Basics - Factorial function n! = 1 n*(n-1)! n = 0 n  1 Definition -module(ex1). -export([factorial/1]). factorial(0) -> 1; factorial(N) when N >= 1 -> N * factorial(N-1). Implementation Eshell V5.0.1 (abort with ^G) 1> c(ex1). {ok,ex1} 2> ex1:factorial(6). 720

20. 2001-03-19 ETX/D/XPS-01:002 Uen 20 Erlang Examples 2 A few very high-level constructs - QuickSort -module(ex2). -export([qsort/1]). qsort([Head|Tail]) -> First = qsort([X || X <- Tail, X =< Head]), Last = qsort([Y || Y Head]), First ++ [Head|Last]; qsort([]) -> []. Eshell V5.0.1 (abort with ^G) 1> c(ex2). {ok,ex2} 2> ex2:qsort([7,5,3,8,1]). [1,3,5,7,8] "all objects Y taken from the list Tail, where Y > Head"

21. 2001-03-19 ETX/D/XPS-01:002 Uen 21 Erlang Examples 3 Concurrency - Simple timer -module(ex3). -export([send_after/3]). send_after(Time, Receiver, Message) -> spawn_link(fun() -> receive after Time -> Receiver ! Message end end). Eshell V5.0.1 (abort with ^G) 1> c(ex3). {ok,ex3} 2> ex3:send_after(1000, self(), hello). 3> flush(). Shell got hello ok Anonymous function passed as input argument

22. 2001-03-19 ETX/D/XPS-01:002 Uen 22 Erlang Examples 4 Concurrency - Finite State Machine ringing_a(A, B) -> receive {A, on_hook} -> back_to_idle(A, B); {B, answered} -> A ! {stop_tone, ring}, switch ! {connect, A, B}, conversation_a(A, B) after 30000 -> back_to_idle(A, B) end. back_to_idle(A, B) -> A ! {stop_tone, ring}, B ! terminate, idle(A). Selective receive Asynchronous send Optional timeout

23. 2001-03-19 ETX/D/XPS-01:002 Uen 23 Erlang Examples 5 1 P1PnP2... One for one supervision: If any child dies it is restarted a P1PnP2... One for all supervision: If any child dies, all children are terminated and all are restarted 1 1 a1 a Supervision tree: Process links + special supervisor processes The infrastructure handles recovery - program for the correct case "let it crash!"

24. 2001-03-19 ETX/D/XPS-01:002 Uen 24 Erlang/OTP Open Telecom Platform Middleware for Erlang development Designed for fault tolerance and portability Behaviors: A formalization of design patterns Components –Error handling, reporting and logging –Mnesia, distributed real-time database management system –CORBA –IDL Compiler, Java & C Interface Support –HTTP Server –SNMP Agent –...

25. 2001-03-19 ETX/D/XPS-01:002 Uen 25 OTP Behaviors "A formalization of design patterns" –A behavior is a framework + generic code to solve a common problem –Each behavior has built-in support for debugging and software upgrade –Makes it easier to reason about the behavior of a program Examples of OTP behaviors –applicationdefines how an application is implemented –supervisorused to write fault-tolerant supervision trees –gen_serverfor writing client-server applications –gen_eventfor writing event handlers –gen_fsmfor finite state machine programming

26. 2001-03-19 ETX/D/XPS-01:002 Uen 26 Project Highlights Early phases under one roof Powerful architecture team Clear chain of command Rapid decisions (within days) Organize according to product structure

27. 2001-03-19 ETX/D/XPS-01:002 Uen 27 Product Council Techn Mtg Subsyst Block Responsible SystemManagement Techn Mtg AXDProj Mgmt Mtg Subsyst Proj Mgr Profiling Task Force Project Structure Different forums for technical and project Issues –But partly the same people System Management = “glue” –“Owners” of the architecture –Keeps a system perspective –Writes design rules –Holds seminars –Profiling Task Force –Acts as Designer Help Desk Human Interaction Essential Proximity = “Under One Roof”

28. 2001-03-19 ETX/D/XPS-01:002 Uen 28 Process highlights "Lightweight" development process Careful functional decomposition Formal change management of interfaces Incremental design Prototyping of critical parts Only essential documents

29. 2001-03-19 ETX/D/XPS-01:002 Uen 29 ATS Basic Connection Control Support ATM resource Management PVC Control ATM Signalling and call control LSSFRSAVS Label Switch Routing Frame Relay Dynamic Trunking AHS NGS Narrowband Services for AXE SWS OMS CPS Switch Control and Maintenance OAM support Operating System Base ServicesAdd-on Services Not released as part of AXD 301 Tools Project Process Identify Subsystems

30. 2001-03-19 ETX/D/XPS-01:002 Uen 30 PROXYUNIILMI IISPAINIPNNI BICISPVC CHGANA SAALREH CAC TRH DTCVPNA LIMOSPFRIP IPSFIM FTM SCBDPCLKDPATBDPET622DPET155DP ET45DPET34DPABRDPET2CEDPDPG DPOS RCM CPC ATS AVS SWS CPSOMS LSS PNNIR ATSDAVSD LDP LSSD GSMP CNHPRMSWSD DPCNSYPRSW ABRDNACECP EVAWMS PERFSTO EQM DPS LMSOMSD SWM SYS CPMMCNC OTP CSSCPO CPSD Tools Project Process Identify Block Structure Real-time critical Non-real-time critical

31. 2001-03-19 ETX/D/XPS-01:002 Uen 31 Focus On Block Interfaces... Tools Project Process Consider Block Dependencies

32. 2001-03-19 ETX/D/XPS-01:002 Uen 32 Documentation Documentation must evolve with the incremental design Focus on a few vital document types, e.g.: –Requirement Specification –Implementation Proposals –Function Specifications –Block Descriptions –Interface Descriptions Highly readable programs reduce the need for documentation (...but do not eliminate it)

33. 2001-03-19 ETX/D/XPS-01:002 Uen 33 Programming languages (control system) Erlang: ca 1 million lines of code –Nearly all the complex control logic –Operation & Maintenance –Web server and runtime HTML/JavaScript generation C/C++: ca 500k lines of code –Third party software –Low-level protocol drivers –Device drivers Java: ca 13k lines of code –Operator GUI applets

34. 2001-03-19 ETX/D/XPS-01:002 Uen 34 Experiences from AXD 301 SW Design Using Erlang in Complex Systems –Fits very well with the incremental design method –High programmer satisfaction –Outstanding support for robustness and concurrency –Very few side-effects  easier to add/change single components –Small directed teams can achieve impressive results Productivity estimates –Similar line/hour programmer productivity –4-10 fewer lines of source code (compared to C/C++, Java, PLEX) 4-10x higher programmer productivity –Similar number of faults per 1000 lines of source code 4-10x higher quality