1. A User Driven Dynamic Circuit Network Implementation Evangelos Chaniotakis Network Engineering Group DANMS 2008 November 30 2008 Energy Sciences Network Lawrence Berkeley National Laboratory Networking for the Future of Science

2. Contents Introduction ESnet Network Architecture Virtual Circuit Implementation User-Driven VCs Layer 2 and 3 support Path Computation Authentication and Authorization Oversubscription and soft reservations Collaboration Network use Future work Acknowledgments Questions

3. Introduction ESnet's mission: provide the network infrastructure for DOE researchers Rapid growth in scientific computing Highly distributed collaboration reaching the global scale – LHC, eVLBI Distribution of large data sets becoming more and more common (40Tb / day projected for LHC)‏ ESnet must reliably and economically accommodate large flows and regular Internet traffic But: Large flows don't work too well al TCP/IP Our solution: Isolate large flows into VCs Provides predictable bandwidth, allows impolite protocols without disruption to other traffic

4. A Multi-Domain Environment End points will be at independent institutions – campuses or research institutes - that are served by ESnet, Abilene, GÉANT, and their regional networks – Complex inter-domain issues – typical circuit will involve five or more domains – For example, a connection between FNAL and DESY involves five domains, traverses four countries, and crosses seven time zones FNAL (AS3152)‏ [US] ESnet (AS293)‏ [US] GEANT (AS20965)‏ [Europe] DFN (AS680)‏ [Germany] DESY (AS1754)‏ [Germany]

5. ESnet Network Architecture A core 10G best-effort IP network A logically distinct Science Data Network Virtual circuits are generally engineered and provisioned only on SDN links Engineered OSPF metrics ensure that best effort traffic uses IP core and avoids SDN In case of IP network bifurcation, the SDN network will be used by best-effort traffic. QoS is used to engineer this backup mechanism

6. ESnet 4 Core Network – December 2008 Las Vegas Seattle Sunnyvale LA San Diego Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC SLC Port. Baton Rouge Houston Pitts. NYC Boston Atlanta Nashville ESnet IP core ESnet Science Data Network core (N X 10G)‏ ESnet SDN core, NLR links (backup paths)‏ Lab supplied link LHC related link MAN link International IP Connections Layer 1 optical nodes - eventual ESnet Points of Presence ESnet IP switch/router hubs ESnet SDN switch hubs Layer 1 optical nodes not currently in ESnet plans Lab site SDSC StarLight 20G MAN LAN (AofA)‏ Lab site – independent dual connect. USLHC GA LLNL LANL ORNL FNAL BNL PNNL Phil Denver ? LHC/CERN ESnet aggregation switch Chicago

7. Virtual Circuit Implementation Source Sink MPLS labels are attached onto packets from Source and placed in separate queue to ensure guaranteed bandwidth. Regular production traffic queue. Interface queues SDN IP IP Link SDN Link RSVP, MPLS, LDP enabled on internal interfaces standard, best-effort queue high-priority queue LSP between ESnet border routers is determined using topology information from OSPF-TE. Path of LSP is explicitly directed to take SDN network where possible. On the SDN Ethernet switches all traffic is MPLS switched (layer 2.5). Layer 3 VC Service: Packets matching reservation profile IP flow-spec are filtered out (i.e. policy based routing), “policed” to reserved bandwidth, and injected into an LSP. Layer 2 VC Service: Packets matching reservation profile VLAN ID are filtered out (i.e. L2VPN), “policed” to reserved bandwidth, and injected into an LSP. Label Switched Path SDN Link

8. QoS parameterization Classes of service in ESnet: –network control, –expedited-forwarding, –best-effort, –scavenger

9. User-driven Virtual Circuits On-demand Secure Circuit Advance Reservation System –Virtual circuits are requested by end-users –Parameters: endpoints, bandwidth, duration –OSCARS decides on the VC path, implements the VCs inside ESnet, and forwards requests to other domains –Web interface for general users –SOAP interface for automated provisioning tools –Advance reservations allow orchestration

10. Authentication and Authorization SOAP API –Signed messages using X.509 certs –User id determined by the cert subject Web Interface –Username and password Authorization: –Complex underlying resource and privilege system. –Simplified with roles: user, engineer, site admin, operator –Support for one-time authorization tokens

11. Layer 2 and Layer 3 VCs Ethernet Layer 2 VCs –VLAN id can be requested by the user or assigned by the system –Multi-domain negotiation is done –Coordination with end-sites needed IP layer 3 VCs –User provides flow specs –Source & destination IP, port, protocol, DSCP –CE router injects matching packets in LSP

12. Path Computation OSCARS periodically harvests full topology information for ESnet When a path needs to be computed for a new VC request, a topology graph is populated from that data as well as all concurrent VCs. Then, all links that cannot satisfy the new VC are pruned. Finally, a Djikstra shortest-path algorithm is run on the pruned graph The base graph currently stands at ~1000 nodes and 1500 edges.

13. Automated Device Configuration After a VC has been reserved the network devices must be configured Cisco and Juniper platforms are supported Users can use the SOAP API to signal VC setup and teardown OSCARS has a scheduler component that periodically checks for pending configuration tasks A platform-specific configuration template is filled out and pushed to the routers. Currently 10-100 seconds are needed to instantiate a circuit in this manner.

14. Over-subscription and Soft Reservations Original concept did not allow for any kind of over- subscription or over-booking. Emerging user requirements: –User-managed load-balancing –Redundant VCs We decided to allow users to oversubscribe their VCs. Packets below reserved bandwidth are marked expedited-forwarding (normal VC traffic)‏ Any packets exceeding that are marked as scavenger.

15. Collaboration DICE: Dante, Internet2, Caltech/USLHCNet, ESnet Close partnership with Internet2 Interoperability with AutoBAHN, Phosphorus Automated provisioning with TeraPaths, LambdaStation and Phoebus Standardization efforts: –OGF: NSI WG, NML WG, NM WG –GLIF: GNI API WG

16. Network Use Currently in pre-production. 16 long-term VCs, total ~40 Gbps reserved –Almost all related to LHC T0-T1 and T1-T2 –Almost all are “soft” reservations Primary users: Fermilab, Brookhaven Our users consistently demand production-quality availability for LHC T0-T1 and T1-T2 VCs. Cross-domain VCs with Internet2 using LambdaStation and Terapaths Demos at SC07, SC08, multiple Joint Techs and I2 Member Meetings VCs minimally disrupted during full replacement of network gear in two of our PoPs.

17. OSCARS Managed Production VCs

18. Future work Outage management –Automated VC rerouting based on network management system data, and scheduled or unscheduled outages Multi-layer VCs –Integrated solution for services provisioned across multiple layers – ie an L3 service over a L2 circuit over a L1 lightpath. Optimizations –Support for short-lived just-in-time VCs (<15 min) –Provisioning and instantiation speed-up

19. Acknowledgments Tom Lehman, ISI East John Vollbrecht, Internet 2 Andrew Lake, Internet 2 Afrodite Sevasti, AutoBAHN project Guy Roberts, DANTE Radek Krzywania, PSNC

20. Thank you! Questions?

21. Authors Chin P. Guok, ESnet chin@es.net David W. Robertson, LBNL dwrobertson@lbl.gov Evangelos Chaniotakis, ESnet haniotak@es.net Mary R. Thompson, LBNL mrthompson@lbl.gov William E. Johnston, ESnet wej@es.net Brian Tierney, ESnet tierney@es.net