1. ONOS Open Network Operating System
An Experimental Open-Source Distributed SDN OS
Introduction:
Experimental, Open, Distributed
----- Meeting Notes (7/29/13 12:57) -----
Read the slide
This is just a glimpse
Pankaj Berde, Umesh Krishnaswamy, Jonathan Hart, Masayoshi Kobayashi, Pavlin Radoslavov, Pingping Lin, Sean Corcoran, Tim Lindberg, Rachel Sverdlov, Suibin Zhang, William Snow, Guru Parulkar

2. Agenda Overview of ONRC (Open Networking Research Center) ONOS
Architecture
Scale-out and high availability
Network graph as north-bound abstraction
DEMO
Consistency models
Development and test environment
Performance
Next steps

3. Leadership
Nick Mckeown
KP, Mayfield, Sequoia Professor, Stanford
Larry Peterson
Bob Kahn Professor Princeton
Chief Architect, ON.LAB
Scott Shenker
Professor, UC Berkeley
Chief Scientist, ICSI
National Academy of Engineering ACM SIGCOMM Award Winners Fellow of IEEE and ACM Entrepreneurs Impact on practice of networking/cloud
Let me begin at the top…
We are really fortunate to have a leadership of three very accomplished gentlemen – they provide the vision and direction and that defines who we are and what we do.
I thought I would simply list their common accomplishments and recognitions…
Each one of them is
o o o
Moreover they have been also successful entrepreneurs and have had a history of impacting real practice of networking….
Now you may also wonder why they came together for this venture?

4. Stanford/Berkeley SDN Activities With Partners
VM Migration
(Best Demo)
Trans-Pacific
Baby GENI
Nation Wide GENI
“The OpenFlow Show”
– IT World
SDN Concept
SIGCOMM08
GEC3
SIGCOMM09
GEC6
GEC9
Interop
2011
Demo
Other countries
Over 68 countries
(Europe, Japan, China, Korea,
Brazil, etc.)
Deployment
US R&E Community
GENI: 8 Universities + Internet2 + NLR
Many other campuses
Stanford University
~45 switch/APs ~25user
In McKeown Group
CIS/EE Building
Production Network
OpenFlow Spec
v0.8.9
v1.0
v1.1
Reference Switch
NetFPGA
Software
Network OS
NOX
SNAC
Beacon
Virtualization
FlowVisor
FlowVisor (Java)
Tools
Test Suite
oftrace
Mininet
Measurement tools
GENI software suite
Expedient/Opt-in Manager/FOAM
Development
Platform
Ethane
+Broadcom
2007
2008
2009
2010
2011

5. Scaling of SDN Innovation
Build strong intellectual foundation
Bring open source SDN tools/platforms to community
Standardize OpenFlow and promote SDN
~100 Members from all parts of the industry
Bring best SDN content; facilitate high quality dialogue
3 successive sold out events; participation of ecosys
SDN Academy
Bring best SDN training to companies
to accelerate SDN development and adoption

6. ONRC Organizational Structure
Berkeley
Scott Shenker
Sylvia Ratnasamy
Stanford
Nick McKeown
Guru Parulkar
Sachin Katti
Open Network Lab
Exec Director: Guru Parulkar
VP Eng: Bill Snow
Chief Architect: Larry Peterson
16-19 Engineers/Tech Leads
(includes PlanetLab team)
Tools/Platforms for SDN community
OpenCloud demonstration of XaaS and SDN
PhD/Postdocs
Research
To achieve our mission we came up with the following structure.
It has three parts:
Research at Berkeley
Research at Stanford
And an independent nonprofit Open Networking Lab.
We have formalized our collaboration between Berkeley and Stanford through this center. This center gives us more reasons to get together with our students and promote more interactions. And that is great.
I want to mention our collaborator Sachin Katti who is a professor of EE and CS has been working on OpenRadio and SDN based mobile wireless networks.
The ON.Lab is kind of unique to our center.
We created ON.Lab as an independent lab with the goal to develop, support, and deploy open source SDN tools and platforms for the benefit of the community.
And create SDN deployments and demonstrations for R&E networks.

7. Mission
Bring innovation and openness to internet and cloud infrastructure with open source tools and platforms

8. TestON with debugging support MININET, Cluster Edition
Tools & Platforms
3rd party
components
SDN-IP Peering
Apps
Apps
Apps
Apps
Open Interfaces
ONOS
Network OS
Network OS
Network Hypervisor
FlowVisor, OpenVirteX
NetSight
TestON with debugging support
Open Interfaces
MININET, Cluster Edition
Forwarding

9. Open Network OS (ONOS) Architecture Scale-out and high availability
Network graph as north-bound abstraction
DEMO
Consistency models
Development and test environment
Performance
Next steps

10. ONOS: Executive Summary
Distributed Network OS
Network Graph Northbound Abstraction
Horizontally Scalable
Highly Available
Built using open source components
ONOS
Version 0.1
- Flow API, Shortest Path computation, Sample application
- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)
- Deployment in progress at REANNZ (SDN-IP peering)
Status
Next is sharing our experience with community. Address the non-goals we discussed earlier and evaluate more use cases and applications on this experimental Open Distributed NOS.
Want to join us in this experiment? Lookout for more information on our website.
Exploring performance & reactive computation frameworks
Expand graph abstraction for more types of network state
Control functions: intra-domain & inter-domain routing
Example use cases: traffic engineering, dynamic virtual networks on demand, …
Next

11. ONOS – Architecture Overview

12. Open Network OS Focus (Started in Summer 2012)
Global network view
Routing TE
Mobility
Global Network View
Network OS
Scale-out
Design
Openflow
Packet
Forwarding
Two of them have to do with logical centralization of the control plane and design of Network OS.
Number 1, will the Network OS become a performance bottleneck? Or can we scale the Network OS horizontally as we need more horse power?
Number 2, will the Network OS become a single point of failure? Or can we make the Network OS and the control plane fault tolerant?
The third question has to do with Northbound API. What is the best abstraction the Network OS can offer to application writers that enables reusable and pluggable network control and management applications?
ONOS attempts to address exactly these issues…
Fault Tolerance
Packet
Forwarding
Programmable
Base Station
Packet
Forwarding

13. Community needs an open source distributed SDN OS
Prior Work
Distributed control platform for large-scale networks
Focus on reliability, scalability, and generality
Scale-out NOS focused on network virtualization in data centers
State distribution primitives, global network view, ONIX API
ONIX
Other Work
Helios (NEC), Midonet (Midokura), Hyperflow, Maestro, Kandoo
NOX, POX, Beacon, Floodlight, Trema controllers
ONIX did attempt to solve these issues. There are few more efforts. To enable more research in this area community needs an open distributed NOS.
----- Meeting Notes (7/29/13 12:57) -----
ONIX is scale out NOS focused on Network Virtualization in data center context and is closed source.
Community needs an open source distributed SDN OS

14. ONOS High Level Architecture
Network Graph Eventually consistent
Titan Graph DB
Cassandra In-Memory DHT
Distributed Registry Strongly Consistent
Zookeeper
Instance 1
Instance 2
Instance 3
OpenFlow Controller+
OpenFlow Controller+
OpenFlow Controller+
Built on two distributed data constructs
1> Network Graph which is the global network view containing the network state represented as a graph which is eventually consistent
2> Distributed Registry is the global cluster management state stored in Zookeeper using transactional consistency.
Multiple instances of ONOS control different parts of network and help realize a single global network view by cooperatively using these two distributed data constructs.
----- Meeting Notes (5/15/13 14:21) -----
Distribruted Registry keeps information on who is in control of the switch objects and has write permissions to update the network graph. In general it stores the resource ownership in a strongly consistent way.
----- Meeting Notes (7/29/13 12:57) -----
order animation
remove floodlight
Host
+Floodlight Drivers

15. Scale-out & HA

16. ONOS Scale-Out Network Graph Distributed Network OS Instance 1
Global network view
Distributed Network OS
Instance 1
Instance 2
Instance 3
A part of network is solely controlled by a single ONOS instance and the same instance is also solely responsible for maintaining the state of the partition into the network graph.
[We also refer this as Control isolation.]
This enables simpler scale-out design.
As the network grows beyond the control capacity one can add another instance which will be responsible for a new part of network . As this part is realized into Network Graph, applications will get a global network view.
----- Meeting Notes (7/29/13 12:57) -----
Fix animation
Data plane
An instance is responsible for maintaining a part of network graph
Control capacity can grow with network size or application need

17. ONOS Control Plane Failover
Master
Switch A = ONOS 2
Candidates = ONOS 3
Master
Switch A = ONOS 1
Candidates = ONOS 2, ONOS 3
Candidates = ONOS 2, ONOS 3
Master
Switch A = NONE
Candidates = ONOS 2, ONOS 3
Candidates = ONOS 2, ONOS 3
Distributed Registry
Distributed Network OS
Instance 1
Instance 2
Instance 3
Switch A is being controlled by Instance 1 and the registry shows it as master for switch A.
Instance 1 has a failure and dies.
Registry detects that instance 1 is down and release the mastership for Switch A.
Remaining candidates join the mastership election within registry. Lets say Instance 2 wins the election and is marked in registry as the master for Switch A.
The channel with Instance 2 becomes the active channel and other channel becomes passive.
This enables a quick failover of switch when there is a control plane failure.
----- Meeting Notes (7/29/13 12:57) -----
Mention strong consistency and elegent coordination
Host A B C D E F

18. Network Graph

19. ONOS Network Graph Abstraction
Id: 1 A
Id: 101, Label
Id: 103, Label
Id: 2 C
Id: 3 B
Id: 102, Label
Id: 104, Label
Id: 106, Label
Id: 105, Label
Titan Graph DB
Network graph is organized as a graph database. Vertices as network objects and connected by edges as relation between the vertices.
We use Titan as graph DB with Cassandra as its backend.
Cassandra is eventually consistent
Cassandra In-memory DHT

20. Network Graph Network state is naturally represented as a graph
port
switch
device on
link
host
Network is naturally a graph with switches, ports, devices as objects as vertices. Similarly links and attachment points are modeled as edges.
Applications can traverse and write to this graph to program the data plane.
How? Lets look at this example application
Network state is naturally represented as a graph
Graph has basic network objects like switch, port, device and links
Application writes to this graph & programs the data plane

21. Example: Path Computation App on Network Graph
port
switch
device
Flow path
Flow entry on
link
inport
outport
host
flow
Path Computation is an application which is using Network Graph.
The application can find a path from source to destination by traversing links and program this path with flow entries to create a flow-path.
These flow-entries are translated by ONOS core into flow table rules and pushed onto the topology.
Last bullet: Application is made simple and stateless. It does not need to worry about topology maintenance.
----- Meeting Notes (5/14/13 14:16) -----
start without text. Bring in text at end and make one point
Application computes path by traversing the links from source to destination
Application writes each flow entry for the path
Thus path computation app does not need to worry about topology maintenance

22. Example: A simpler abstraction on network graph?
Virtual network objects
Logical Crossbar
port
switch
device
Edge Port on
link
physical
host
Real network objects
Network graph simplifies applications but can it be used to accelerate innovations of simpler abstractions in control plane?
Here is an example of Logical Crossbar. The complexity of network state and topology is hidden.
One can build hierarchy of these abstractions further hiding the complexity.
Last bullet: We feel network graph will unlock innovations.
7 minute Marker
App or service on top of ONOS
Maintains mapping from simpler to complex
Thus makes applications even simpler and enables new abstractions

23. Network Graph Representation
Flow entry
Flow path
flow
Vertex with
10 properties
Switch
Vertex with
3 properties
flow
Vertex with
11 properties
Flow entry
Property
(e.g. dpid)
(e.g. state) …
Edge
Vertex represented as Cassandra row
Row indices for fast vertex centric queries
Column
Value
Label id + direction
Primary key
Edge id
Vertex id
Signature properties
Other properties
Edge represented as Cassandra column

24. Network Graph and Switches
Network Graph: Switches
Switch Manager
Switch Manager
Switch Manager
Let us see how ONOS builds the network graph. Each ONOS node has a switch manager. When switches connect, switches and ports are get added as switches register with an ONOS node. When switches disconnect, they get marked as inactive in the network graph. OF OF OF

25. Network Graph and Link Discovery
Network Graph: Links
Link Discovery
Link Discovery
Link Discovery SM SM SM
LLDP
LLDP
Each node sends out LLDP on the switches connected to it. Links with source and destination port controlled by different ONOS nodes can also be discovered using the network graph.

26. Devices and Network Graph
Network Graph: Devices
Device Manager
Device Manager
Device Manager SM LD SM LD SM LD
Host packet Ins are used to learn about devices, their attachment points. The network graph is updated with this information.
PKTIN
Host
PKTIN
PKTIN

27. Path Computation with Network Graph
Network Graph: Flow Paths
Flow 1
Flow entries
Flow 2
Flow entries
Flow 3
Flow entries
Flow 4
Flow entries
Flow 5
Flow entries
Flow 6
Flow entries
Flow 7
Flow 8
Flow entries
Flow entries SM LD DM SM LD DM SM LD DM
Flow paths are provisioned in ONOS.
The source dpid of a flow is used to partition which node will compute the path. Computed paths and flow entries are also stored in the network graph.
Flow entries have relationship to the switches.
Host

28. Network Graph and Flow ManagerPC PC PC
Network Graph: Flows
Flow 1
Flow entries
Flow 2
Flow entries
Flow 3
Flow entries
Flow 4
Flow entries
Flow 5
Flow entries
Flow 6
Flow entries
Flow 7
Flow 8
Flow entries
Flow entries
Flow Manager
Flow Manager
Flow Manager
Flowmod
Flowmod
Each flow manager programs the switches connected to it using the state in the network graph.
When a link fails, PC will recompute a new path and Flow Manager will push new flow entries.
Flowmod SM LD DM SM LD DM SM LD DM
Host

29. ONOS High Level Architecture
Network Graph Eventually consistent
Titan Graph DB
Cassandra In-Memory DHT
Distributed Registry Strongly Consistent
Zookeeper
Instance 1
Instance 2
Instance 3
OpenFlow Controller+
OpenFlow Controller+
OpenFlow Controller+
Built on two distributed data constructs
1> Network Graph which is the global network view containing the network state represented as a graph which is eventually consistent
2> Distributed Registry is the global cluster management state stored in Zookeeper using transactional consistency.
Multiple instances of ONOS control different parts of network and help realize a single global network view by cooperatively using these two distributed data constructs.
----- Meeting Notes (5/15/13 14:21) -----
Distribruted Registry keeps information on who is in control of the switch objects and has write permissions to update the network graph. In general it stores the resource ownership in a strongly consistent way.
----- Meeting Notes (7/29/13 12:57) -----
order animation
remove floodlight
Host
+Floodlight Drivers

30. DEMO

31. Consistency Deep Dive

32. Consistency Definition
Strong Consistency: Upon an update to the network state by an instance, all subsequent reads by any instance returns the last updated value.
Strong consistency adds complexity and latency to distributed data management.
Eventual consistency is slight relaxation – allowing readers to be behind for a short period of time.
Consistency is not a binary value but a fraction between 0 and 1 excluding both extremes. So lets see what is our consistency definition.
Strong consistency: Every update is instantly available on other instances of ONOS.
Next bullet: It cannot be instant in reality, there is always some cost/delay involved.
Our eventual consistency is much closer to 1 than many other eventual consistency definitions

33. Strong Consistency using Registry
Network Graph
Distributed Network OS
Instance 1
Instance 2
Instance 3
Switch A
Master = ONOS 1
Switch A
Master = NONE
A =
Switch A
Master = ONOS 1
Switch A
Master = NONE
Switch A
Master = ONOS 1
Switch A
Master = NONE
A = ONOS 1
Registry
We use Transactional Consistency for Master election.
Lets look at Registry: We see Switch has NO Master when we start.
Lets see the point in time where Master is elected and ONOS 1 is marked as master. After the Master is elected and mastership is consistently propagated on all instances.
All subsequent reads for master on any instance shows Master as ONOS 1 for Switch A. This guarantees that we have control isolation.
All instances
Switch A Master = NONE
Instance 1 Switch A Master = ONOS 1
Instance 2 Switch A Master = ONOS 1
Instance 3 Switch A Master = ONOS 1
All instances
Switch A Master = NONE
Timeline
Master elected for switch A
Delay of Locking & Consensus

34. Why Strong Consistency is needed for Master Election
Weaker consistency might mean Master election on instance 1 will not be available on other instances.
That can lead to having multiple masters for a switch.
Multiple Masters will break our semantic of control isolation.
Strong locking semantic is needed for Master Election
Locking and latches are distributed primitives which need consistency.

35. Eventual Consistency in Network Graph
Distributed Network OS
Instance 1
Instance 2
Instance 3
Switch A
State = ACTIVE
SWITCH A
STATE= INACTIVE
Switch A
State = ACTIVE
Switch A
State = INACTIVE
Switch A
STATE = INACTIVE
Switch A
STATE = ACTIVE
DHT
Lets take a concrete example.
Before switch is connected: it is marked INACTIVE on our Network Graph.
As soon as the switch connects to ONOS 1 it is marked ACTIVE immediately on INSTANCE 1.
For a fraction of second, instance 2 and instance 3 might still show the switch as INACTIVE till they are eventually consistent (white line)
From that point the switch state is consistent on All instances.
Timeline
All instances
Switch A STATE = INACTIVE
Instance 1 Switch A = ACTIVE
Instance 2 Switch A = INACTIVE
Instance 3 Switch A = INACTIVE
All instances
Switch A STATE = ACTIVE
Switch Connected to ONOS
Delay of Eventual Consensus

36. Cost of Eventual Consistency
Short delay will mean the switch A state is not ACTIVE on some ONOS instances in previous example.
Applications on one instance will compute flow through the switch A while other instances will not use the switch A for path computation.
Eventual consistency becomes more visible during control plane network congestion.
What does it mean? Switch is not available instantly for path computation on some instances. Some flows can be computed without using switch A.
Similarly when flow entries are written to Network graph they become eventually available on all instances and each instance picks its flow entries and pushes onto switches.

37. Why is Eventual Consistency good enough for Network State?
Physical network state changes asynchronously
Strong consistency across data and control plane is too hard
Control apps know how to deal with eventual consistency
In the current distributed control plane, each router makes its own decision based on old info from other parts of the network and it works fine
Strong Consistency is more likely to lead to inaccuracy of network state as network congestions are real.
Asynchronously changing state is not transactional and does not need strong consistency. Control apps know that they are always working on a stale state and know to deal with it.
Today’s routers work in the same fashion.
Network state is most often used by current gen control applications with assumed eventual consistency.
Time check: 13 minutes

38. Consistency learning One Consistency does not fit all
Consequences of delays need to be well understood
More research needs to be done on various states using different consistency models

39. Development & test environment

40. ONOS Development & Test cycle
Source code on github
Agile: 3-4 week sprints
Mostly Java and many utility scripts
CI: Maven, Jenkins, JUnit, Coverage, TestON
Vagrant-based development VM
Daily 4 hour of Continuous Hours of Operations (CHO) tests as part of build
Several CHO cycles simulating rapid churns in network & failures on ONOS instances

41. ONOS Development Environment
Single installation script creates a cluster of Virtual Box VMs

42. Test Lab Topology

43. ON.LAB ONOS Test implementation
ON.LAB team has implemented following aspects of automated tests
ONOS Unit Tests (70% coverage)
ONOS System Tests for Functionality, Scale, Performance and Resiliency test (85% coverage)
White Box Network Graph Performance Measurements
All tests are executed nightly in Jenkins Continuous Integration environment.

44. Performance

45. Key performance metrics in Network OS
Network scale (# switches, # ports) -> Delay and Throughput
Link failure, switch failure, switch port failure
Packet_in (request for setting reactive flows)
Reading and searching network graph
Network Graph Traversals
Setup of proactive flows
Application scale (# operations, # applications)
Number of network events propagated to applications (delay & throughput)
Number of operations on Network Graph (delay & throughput)
Parallelism/threading for applications (parallelism on Network Graph)
Parallel path computation performance

46. Performance: Hard Problems
Off the shelf open source does not perform
Ultra low-latency requirements are unique
Need to apply distributed/parallel programming techniques to scale control applications
Reactive control applications need event-driven framework which scale

47. ONOS: Summary ONOS Status Next Distributed Network OS Version 0.1
Network Graph Northbound Abstraction
Horizontally Scalable
Highly Available
Built using open source components
ONOS
Version 0.1
- Flow API, Shortest Path computation, Sample application
- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)
- Deployment in progress at REANNZ (SDN-IP peering)
Status
Next is sharing our experience with community. Address the non-goals we discussed earlier and evaluate more use cases and applications on this experimental Open Distributed NOS.
Want to join us in this experiment? Lookout for more information on our website.
Exploring performance & reactive computation frameworks
Expand graph abstraction for more types of network state
Control functions: intra-domain & inter-domain routing
Example use cases: traffic engineering, dynamic virtual networks on demand, …
Next