1. SPRING-OPEN SDN based WAN Control of Open Segment Routers An ONF TAG Project Saurav Das Project Lead & ONF Consultant Ciena talk, Oct 23 rd, 2014

2. Outline Motivation & Project Goals Project Description Progress What next?

3. But when it comes to Hardware switches, misconceptions abound OpenFlow is not mature OpenFlow does not work with current hardware OpenFlow does not scale SDN/OpenFlow is about centralized control Motivation: ONF Point of View SDN/OpenFlow successful in DataCenters with Software Switches and Overlay networks

4. statebehaviorInterface msg match Eth, VLAN, IP, L4 flows 1.0 Q4 ‘09 single message queue w/optional barriers ports forward {0, 1, n} 1.1 Q1 ‘11 + Multiple Tables/Pipelines: + Group Tables + forward 1-in-n (ECMP) + match QinQ, MPLS, SCTP + match virtual ports + extensible match + extensible actions 1.2 Q4 ‘11 + IPv6 + multiple controllers 1.3 Q2 ‘12 + per-flow metering + tunnel-id OpenFlow has evolved towards production readiness. + multiple channels (auxiliary connections) 1.4 Q4 ‘13 + optical ports + synchronized tables + bundle messages

5. ONF TAG Project Goals 1.Demonstrate maturity and scale of the ONF work product in hardware readily available today using the latest stable versions of ONF protocols – eg. OF 1.3.4. 2.Provide feedback to ONF WGs on their work product from an implementation of the chosen networking scenario. 3.Promote adoption by creating a core-kernel that is extensible for value-add towards deployment, interoperability and differentiation.

6. Non - Goals 1.Not creating GA product; no QA; will not be ready for production nor interoperate with other networks and network control planes. Will support some elements helpful for productization (eg. config, troubleshooting/OAM, visibility etc.) 2.Not delivering a specific service like Bandwidth-TE /VPN/NFV. Instead supporting core-capabilities to build such services on top (extensibility options) 3.Not a plugfest – data and control plane choices will be made; however choices should be replaceable by other parts, both commercial and open-source as long as they conform to the requirements

7. N/w. Scenario: SDN based WAN Control OpenFlow Routing, Recovery, Label imposition Requests SR Labels imposed by controller OSR FIB built by controller Routing Service Requests Controller System Discovery Service Open Segment Routers (OSR) Forwarding Service

8. One Way to Implement SR Controller PCE BGP-LS for topology info PCEP for tunnel req & label imposition OpenDaylight or Cisco ONE or Juniper NorthStar Eg. Cariden/Cisco or WANDL/Juniper OSPFv2 OSPFv3 ISIS Routing, Recovery, Label distribution (new in SR) IETF working on extending all of these protocols for Segment Routing Controller/PCE not required for certain use cases - just configure routers for SR via CLI

9. Why Segment Routing Eliminates label distribution protocols – LDP and RSVP-TE Thereby eliminates synchronization and state management complexities Label distribution via OSPF or ISIS with suitable extensions (see IETF drafts) Source routing via ‘segments’ maps to ‘labels’ in MPLS data plane; MPLS data plane unchanged – SR operations PUSH, NEXT, CONTINUE maps to MPLS operations PUSH, POP, SWAP (with same label) resp. Introduces globally significant labels - node segments retains locally significant labels – adjacency segments can use ECMP shortest-paths and Explicit Paths (loose, strict); can be used for TE/VPN/PBR/Service-chains Segment Routing (SR) or SPRING (IETF name) – Source Packet Routing In NetworkinG Think of Segment Routing as giving new meaning to labels allowing different network operations and a simpler control plane without changing the data plane!

10. Outline Motivation & Project Goals Project Description Progress What next?

11. ONF TAG Project Core Requirements Must work on Hardware + Must use ONF Protocols + Must use Available Commodity Parts + Provide Feedback to Standards + Diversity of Solutions + Must be Extensible

12. Project Deliverables 1.Open Segment Router on 1 hardware platform 2.WAN Controller Supports Discovery and Routing Services Label imposition for segment-routing/stitching GUI/CLI, troubleshooting, stats 3.System Prototype & Demonstration Segment routed island Demonstrate discovery & several routing scenarios Extensible towards deployment & interoperability 4.Feedback What was not implemented and why? Gaps/inefficiencies in protocol HW requirements

13. Project Milestones & Timeline June 1st Aug 1st Oct 1st Dec 1st Open Segment Router (OSR) WAN Controller Controller-OSR Integration System Prototype & Demonstration

14. Routing Service: Scenario # 1 106 102 103 105 104 101 101, 102 … 106 are Node Segments allocated out of the SRGB, and bound to the router loopback addresses. 10.10.1.0/24 10.10.6.0/24 Default Routing using Node Segments, ECMP and PHP 10.10.4.0/24 10.10.3.0/24 Global label 106 imposed on pkts dst. to the 10.10.6 subnet ECMP Paths Still 106 PHP

15. Routing Service: Scenario # 2 106 102 103 105 104 101 10.10.1.0/24 10.10.6.0/24 Policy Routing 10.10.4.0/24 10.10.3.0/24 Policy#1 – Traffic from.3 to.6 should avoid link X link X 104 106 Policy#2 – Flow ‘f’ from.1 to.6 should stay in upper plane Anycast Node Segment 999 102 106

16. Routing Service: Scenario # 3 106 102 103 105 104 101 10.10.1.0/24 10.10.6.0/24 10.10.4.0/24 10.10.3.0/24 TE Support: Load-balancing among non-ECMP Paths Same adjacency segment 12009 assigned to both outgoing links for load-balancing at 102, to 104 or 103 Non-ECMP paths 12009 106 Once at 104 or 103, it’s just SPF to 106

17. Routing Service: Scenario # 4 106 102 103 105 104 101 10.10.1.0/24 10.10.6.0/24 TE Support: Explicit Routing 10.10.4.0/24 Desired Explicit Path Requires label stack: 102 103 105 104 106 102 103 105 Pop 105 Push 104 106 Stitching Segments Deep-stacks can cause problems in merchant silicon 1)Cannot push many labels all at once 2)Can cause loss of entropy if hw cannot read down to L3/L4 headers Solution: use Segment stitching

18. Routing Service: Scenario # 5 106 102 103 105 104 101 10.10.1.0/24 10.10.6.0/24 Service Chaining 10.10.4.0/24 Firewall DPI Adjacency Segment 9002 Adjacency Segment 16555 Desired Chain 103 9002 105 16555 106 Note: Could have used segment-stitching or label- swapping to avoid deep label stack

19. Bare-metal Hardware CPUASIC OS Distribution SDK Gluework OF ClientSupporting Processes SPRING-OPEN Data Plane Requirements

20. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System

21. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System

22. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains SFP Routing with Node Segments Use of ECMP and PHP Convergence Protection Connectivity management – ACL policies Avoiding links, nodes TE support – explicit strict paths Load balancing over non-equal-cost paths Service chaining

23. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System

24. SPRING-OPEN Control Plane Requirements Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Node/link characteristics, capabilities & constraints (eg. table-types, bw etc.) LLDP based distributed Link/Neighbor Discovery Scope of identifiers, namespaces & association with nodes/intfs Verifying configuration vs. discovered resources Proxy edge services – eg. ARP, ICMP Data Plane Stats Data Plane Troubleshooting Network wide view of topology, traffic, capabilities and resource limits Maintains API for requests from routing, forwarding services & external req. Provides versioning

25. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System

26. SPRING-OPEN Control Plane Requirements Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager Responsible for syncing controller-to- controller forwarding state Responsible for consistency requirements when updating Multiple entries in a table Multiple tables in a switch Multiple switches in a network

27. SPRING-OPEN Control Plane Requirements Routing Service Default Routing Manager Recovery Services Policy Routing Manager – ACL, TE Support, Service-Chains Policy Routing Manager – ACL, TE Support, Service-Chains Discovery Service Stats/OAM Manager Config. Manager Resource Manager Link/Nbr. Disc. Network Snapshot Manager Forwarding Service C2D Sync Manager C2C Sync Manager Consistent Update Manager HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System

28. SPRING-OPEN Control Plane Requirements HA Manager Visibility/Debug Fwk. Dist. DB Conn. Mgr. / Event Engine Controller System debugCounters debugEvents REST GUICLI Dashboard Tseries Config RT view OpenFlow 1.3 New Handshake – better error hd -- better SM TTP1TTP2TTP3 Typed Table Abstraction Support for EQUALS Dist. key-value store Persistence Notifications Leader Election Routing Service Controller System Discovery Service Forwarding Service Routing Service Controller System Discovery Service Forwarding Service Routing Service Controller System Discovery Service Forwarding Service

29. Outline Motivation & Project Goals Project Description Progress What next?

30. Project Members CommittedConsidering Switch Development NTT (Lagopus) Dell (FTOS) Intel Broadcom Controller Development ON.Lab (ONOS) ONF Switch Contribution Delta Dell NTT Intel Broadcom Advisory, Engineering Testbed Verizon NTT Google Tencent

31. ON.Lab Involvement SPRING-OPEN IPv4 unicast routing using MPLS labels, following Segment Routing rules ONOS Typed Table Hardware A platform for multiple switch types: Software Switches Un-typed tabled hardware Optical Switches A platform for multiple services: Multi-layer Overlay Security

32. Network Graph ( Eventually consistent global view ) ONOS v0.1.5 (current) Host Instance 1 Instance 2 Instance 3 OpenFlow Manager + OpenFlow Manager + OpenFlow Manager + + Floodlight Drivers Control Application Distributed Registry (Strongly Consistent) Zookeeper Control Application Coordination ONOS Graph API Topology Replica Event Notifications Hazelcast 32 ONOS System Architecture Persistence RAMCloud Low-latency k/v store (Strongly Consistent) Topology Replica Topology Replica Intent F/W Intent F/W Intent F/W

33. Progress master mid-May onos13 1 st June 1 st July 8 th August 1 st Sept mid Oct end Nov onos13integration -- Unit tests -- Manual Integration -- OF 1.3 support -- Driver Manager -- I/O State Machine -- Role management -- Debug framework 25 26 27

34. New ONOS (1.0 + 1.3) Old ONOS (1.0 switches) - old state machine (or lack thereof) - old switch/port handling - registry service (zookeeper) - role management/changer - ONOS storage + upper functionality - old controller Newer Floodlight (1.0 switches) - new I/O state machine - new switch/port handling - new role management - new debug framework - new storage/sync-manager - new controller - switch manager - role manager modified New Changes (1.3 switches) - new OF Library (Loxigen) - new support for different switches - DriverManager - support for Role. EQUAL - simultaneous support for 1.0 and 1.3 switches - prototyping Test & Integration - integration with master - unit test coverage > master - ensured nightly tests are passing - ensured global context and app functionality - reviewed and merged to master

35. master mid-May onos13 1 st June 1 st July 8 th August 1 st Sept mid Oct end Nov onos13integration -- Unit tests -- Manual Integration -- OF 1.3 support -- Driver Manager -- I/O State Machine -- Role management -- Debug framework 25 26 27 -- Prototyping -- CPqD13 -- OVS13 -- Dell13 Progress

36. Ingress Port Incoming Packet VLAN Flow Table [0] Termination MAC Flow Table [10] Unicast IPv4 Routing Flow Table [20] z MPLS Forwarding Flow Table [30] ACL Policy Flow Table [50] Apply Actions -push/pop -TTL mpls -Set -Output -Group Outgoing Packet z Group Table Entries: L3 Unicast MPLS Unicast ECMP Pkt. + Meta- Data + Action Set {} Egress Port or Group SPRING-OPEN Hardware Abstraction

37. master mid-May onos13 1 st June 1 st July 8 th August 1 st Sept mid Oct end Nov onos13integration -- Unit tests -- Manual Integration -- OF 1.3 support -- Driver Manager -- I/O State Machine -- Role management -- Debug framework 25 26 27 -- Prototyping -- CPqD13 -- OVS13 -- Dell13 Progress -- Network Config Manager

38. Channel Config Service ONOS NetworkConfigManager Network Config Mgr. Config file Startup Config CLI/ REST Running Config Topology Publisher host s switcheslinks Channel ONOS Instance Startup Config Startup Config Startup Config Running Config

39. Filtering Logic Restrict switche s? Yes Default Deny No Default Allow Has Config? No DENY ACCEPT Allowed ? Yes Allowed ? Yes No DENY No Yes ACCEPT & ADD Allow list Deny list

40. master mid-May onos13 1 st June 1 st July 8 th August 1 st Sept mid Oct end Nov onos13integration -- Unit tests -- Manual Integration -- OF 1.3 support -- Driver Manager -- I/O State Machine -- Role management -- Debug framework 25 26 27 -- Prototyping -- CPqD13 -- OVS13 -- Dell13 -- Network Config Manager onos-spring SR Prototype -- Saurav (ONF) -- Sangho (ON.Lab) -- Srikanth (Ericsson/ON.Lab partner) cli gui ntt dell Progress

41. Dell Switch Progress Delivered two switches with pre-alpha software for integration with controller

42. Demo Default Segment Routing with MPLS (node-segments) and ECMP shortest-paths - Communication between subnets across the SR WAN works ARP/ICMP handling, subnet-configuration, pinging router-IPs (normal router behavior) works Link and Switch failure recovery works Policy routing works for one use-case - creating an SR tunnel and assigning flow(s) to it Segment stitching works (where tunnel requires pushing more than 3 labels, and so we stitch-segments of the tunnel to get around hardware limitations)

43. h1 h6 7.7.7.0/24 10.0.1.0/24 101 192.168.0.1 192.168.0.2 102 103 192.168.0.3 Demo 192.168.0.5 105 104 192.168.0.4 106 192.168.0.6

44. Outline Motivation & Project Goals Project Description Progress What next?

45. Extend the controller for hierarchical, geographically distributed control Options for Extensibility

46. SDN WAN Architecture WAN links Local Controllers Global Controllers

47. © 2013 SDN Academy, LLC™. All Rights Reserved. Google’s B4 Architecture Site A Data Center OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch Data Center Site B Data Center Site C B4 WAN Servers RAP TE-AGENT OFC RAP TE-AGENT OFC paxos Quagga Paxos Site B Controllers Servers Site C Controllers Servers Switch hardware iBGP, ISIS eBGP Site controllers Gateway Central TE Servers Global TE 47

48. Microsoft’s SWAN Architecture

49. SDN WAN Architecture WAN links Local Controllers Global Controllers

50. Extend the controller for hierarchical, geographically distributed control Add E-BGP on the controller for exchanging reachability information, route selection and more Options for Extensibility

51. Extend the controller for hierarchical, geographically distributed control Add E-BGP on the controller for exchanging reachability information, route selection and more Provide L3VPN/VPLS/VPWS services Provide full blown TE solution with bandwidth optimization, calendaring etc. Extend control plane to work with optical switches / networks Interoperability with traditional LDP/IGP control plane Options for Extensibility

52. 100 101 102 103 104 105 106 107 108 109 110 100 101 102 103 104 105 106 107 108 109 110 100 101 102 103 104 105 106 107 108 109 110 Consistent updates – loop free updates IP Routing without an IGP

53. Segment Stitching 106 102 103 105 104 101 10.10.1.0/24 10.10.6.0/24 10.10.4.0/24 Desired Explicit Path Requires label stack: 102 103 105 104 106 102 103 105 Pop 105 Push 104 106 Stitching Segments Deep-stacks can cause problems in merchant silicon 1)Cannot push many labels all at once 2)Can cause loss of entropy if hw cannot read down to L3/L4 headers Solution: use Segment stitching

54. © 2013 SDN Academy, LLC™. All Rights Reserved. B4’s In-Place Replacement Model Site A Data Center OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch OFA Switch Data Center Site B Data Center Site C B4 WAN Servers RAP TE-AGENT OFC RAP TE-AGENT OFC paxos Quagga Paxos Site B Controllers Servers Site C Controllers Servers Switch hardware iBGP, ISIS eBGP Site controllers Gateway Central TE Servers Global TE 54

55. Parallel Network parallel SDN fabric, interacts with traditional network and outside world using E-BGP small number of sites low volume of production traffic as confidence is gained, grow users at site, increase footprint to more sites Traditional Network SDN Fabric SPRING-OPEN’s Parallel Nw Model

56. Extend the controller for hierarchical, geographically distributed control Add E-BGP on the controller for exchanging reachability information, route selection and more Provide L3VPN/VPLS/VPWS services Provide full blown TE solution with bandwidth optimization, calendaring etc. Extend control plane to work with optical switches / networks Interoperability with traditional LDP/IGP control plane In-band control Add FRR to data plane recovery Deeper buffers & QoS in white-box platform Scale-out Segment Routers with white-boxes More OAM / troubleshooting features Security features Multicast/IPv6 … and much more Options for Extensibility

57. Summary Motivation & Project Goals Demonstrate maturity & scale of ONF work product Promote adoption by creating core-kernel Project Description SDN based WAN control of Open Segment Routers Controllers, Bare-metal, merchant-Si, MPLS, OF1.3 Prototype & Demonstrate several Segment Routing scenarios in 6 months – multi-member-company effort Progress Prototyping with software switches using OF1.3 Integration with Dell hardware switch beginning Nov Next Lots of extensibility options for value-add, interoperability and deployment