1. TeraPaths TeraPaths: Establishing End-to-End QoS Paths through L2 and L3 WAN Connections Presented by Presented by Dimitrios Katramatos, BNL Dimitrios Katramatos, BNL

2. 2 Outline  The TeraPaths project  Motivation  Concept and implementation  View of the world (network)  Interoperating with WAN services  L2 vs. L3  What is required from the site LAN?  Status/future

3. 3 Motivation  The problem: support efficient/reliable/predictable peta-scale data movement in modern high-speed networks  Capacity is not limitless  Multiple data flows with varying priority  Default “best effort” network behavior can cause performance and service disruption problems  Solution: enhance network functionality with QoS features to allow prioritization and protection of data flows  Classify traffic  Schedule network usage

4. 4 Prioritized vs. Best Effort Traffic

5. 5 TeraPaths TheTeraPaths Service: Reserve End-to- End Paths with Guaranteed Bandwidth WAN WAN web services TeraPaths 1 2 3

6. 6 Data Flow Information  Owner info (user)  Data flow ID  Source IP and port  Destination IP and port  IPs and ports can be ranges (multiple flows)  Direction (unidirectional/bidirectional)  Protocol  Bandwidth (class of service)  Multiple flows will share (best effort within the class)  Start time and duration  Minute resolution

7. 7 Path Setup  Participating end site subnets are controlled by TeraPaths software instances (TeraPaths Domain Controllers or TDCs)  TDCs configure end site LANs to prioritize authorized flows via the DiffServ framework at the network device level  Source site polices/marks authorized flow packets  Destination site admits/re-polices/re-marks packets  End site LANs hand over/receive marked packets to/from the WAN  WAN provides MPLS tunnels or dynamic circuits  Initiating TDC requests MPLS tunnel or dynamic circuit with matching bandwidth and lifetime, or…  TDC funnels several flows into MPLS tunnel or dynamic circuit with aggregate bandwidth and lifetime  WAN preserves packet markings

8. 8 Path Setup (ii)  WAN domains must interoperate  Each end site’s TDC has a single point of contact for WAN services  TDCs have no knowledge of WAN internals other than what is exposed by the WAN services  End sites have no direct control over the WAN  Either tunnel or circuit through WAN  Cannot mix and match

9. 9 Conceptual View of the Network TeraPaths Site A Site B Site C Site D WAN 1 WAN 2 WAN 3 service invocation data flow peering WAN chain

10. 10 TeraPaths Testbed ( ) current US ATLAS T2 sites

11. 11 TeraPaths TeraPaths Web Services Architecture Internal Services Public Services Web Interface Admin Module NDC Database protected network API remote local WAN Services WAN Services proxy CLIs/w client

12. 12 Interoperating with WAN Services  TeraPaths “proxy” servers  Implement interface required by TeraPaths core  Hide WAN service differences  Clients to WAN web services (OSCARS and DRAGON)  Close cooperation with ESnet and I2 development teams  Submit reservations for MPLS tunnels or dynamic circuits  Handle security requirements  Handle errors  MPLS tunnels vs. dynamic circuits  Utilization requires drastically different approach

13. 13 L2 vs. L3 (i)  MPLS tunnel starts and ends within WAN domain  Packets are admitted into the tunnel based on flow ID information (IP src, port src, IP dst, port dst )  WAN admission performed at the first router of the tunnel (ingress) WAN border router MPLS tunnel ingress/egress router MPLS tunnel ingress/egress router

14. 14 L2 vs. L3 (ii)  Dynamic circuit appears as VLAN connecting end site border routers with single hop  Cannot use flow ID data directly  Flow must be directed to the proper VLAN  WAN admission performed within end site LAN  Select VLAN with Policy Based Routing (PBR) WAN switch border router

15. 15 Site LAN Setup (DiffServ)

16. 16 Site LAN Setup (DiffServ w/pass-thru)

17. 17 3 rd Party WAN Segments  Some WAN segments may not be automatically configurable  Static configuration allows DSCP bits to go through  Only allow specific interfaces  ACLs and aggregate policers

18. 18 L2-Specific Issues  Limitations with VLANs  Tag range - tentatively selected 3550-3599 (50 VLANs)  Tag conflicts - eliminate by synchronizing site databases  Scalability problems  Flow grouping  Logistics  PBR overhead  Virtual border router  Sensitive/3 rd party network segments  VLAN pass-thru

19. 19 Additional Setup for L2

20. 20 Summary  TeraPaths stitches together virtual paths with guaranteed bandwidth…  through end-site LANs (direct control)…  and end-site interconnecting WANs (indirectly, automatically)…  from end host to end host  TeraPaths…  utilizes DiffServ for LAN QoS…  makes arrangements for WAN MPLS tunnels or dynamic circuits by interfacing with WAN (web) services…  schedules bandwidth usage with advance reservations…  utilizes “pass-thru” techniques for sensitive or 3 rd party network segments

21. 21 Status and Future  Currently: basic software ready, infrastructure tested  API and web interface, simple negotiation  Statically allocated bandwidth classes  L3 paths (MPLS tunnels) through ESnet  Elementary AAA  BNL  UMich  In the works, future  Testbed expansion to US ATLAS Tier 2 sites  Utilization of L2 paths (dynamic circuits) through ESnet and Internet2  Dynamic bandwidth allocation within service classes  CLI, extended API, configurable negotiation  Grid-style AAA (GUMS/VOMS)  Admin module to facilitate end site LAN setup  http://www.racf.bnl.gov/terapaths