1. TeraPaths: A QoS Enabled Collaborative Data Sharing Infrastructure for Petascale Computing Research The TeraPaths Project Team CHEP 06

2. 2 The TeraPaths Project Team  Scott Bradley, BNL  Frank Burstein, BNL  Les Cottrell, SLAC  Bruce Gibbard, BNL  Dimitrios Katramatos, BNL  Yee-Ting Li, SLAC  Shawn McKee, U. Michigan  Razvan Popescu, BNL  David Stampf, BNL  Dantong Yu, BNL

3. 3 Outline  Introduction  The TeraPaths project  The TeraPaths system architecture  Experimental deployment and testing  Future work

4. 4 Introduction  The problem: support efficient/reliable/predictable peta-scale data movement in modern high-speed networks  Multiple data flows with varying priority  Default “best effort” network behavior can cause performance and service disruption problems  Solution: enhance network functionality with QoS features to allow prioritization and protection of data flows

5. 5 Tier 1 Tier 1 site Online System CERN Tier 1 siteBNL Tier 3 site Workstations ~GBps 100-1000 Mbps ~PBps ~10-40 Gbps ~10 Gbps Tier 0+1 Tier 2 e.g. ATLAS Data Distribution Tier 2 site Tier 3 Tier 4 ATLAS experiment ~2.5-10 Gbps Tier 3 site UMich muon calibration

6. 6 The QoS Arsenal  IntServ  RSVP: end-to-end, individual flow-based QoS  DiffServ  Per-packet QoS marking  IP precedence (6+2 classes of service)  DSCP (64 classes of service)  MPLS/GMPLS  Uses RSVP-TE  QoS compatible  Virtual tunnels, constraint-based routing, policy-based routing

7. 7 Prioritized vs. Best Effort Traffic

8. 8  The TeraPaths project investigates the integration and use of LAN QoS and MPLS/GMPLS-based differentiated network services in the ATLAS data intensive distributed computing environment in order to manage the network as a critical resource  DOE: The collaboration includes BNL and the University of Michigan, as well as OSCARS (ESnet), Lambda Station (FNAL), and DWMI (SLAC)  NSF: BNL participates in UltraLight to provide the network advances required in enabling petabyte-scale analysis of globally distributed data  NSF: BNL participates in a new network initiative: PLaNetS (Physics Lambda Network System ), led by CalTech The TeraPaths Project

9. 9 BNL Site Infrastructure LAN/MPLS TeraPaths resource manager MPLS requests traffic identification: addresses, port #, DSCP bits grid AAA Bandwidth Requests & Releases OSCARS ingress / egress LAN QoS M10 data transfer management monitoring GridFtp & dCache/SRM SE network usage policy ESnet remote TeraPaths Remote LAN QoS requests

10. 10 Envisioned Overall Architecture TeraPaths Site A Site B Site C Site D WAN 1 WAN 2 WAN 3 service invocation data flow peering

11. 11 Automate MPLS/LAN QoS Setup  QoS reservation and network configuration system for data flows  Access to QoS reservations:  Manually,through interactive web interface  From a program, through APIs  Compatible with a variety of networking components  Cooperation with WAN providers and remote LAN sites  Access Control and Accounting  System monitoring  Design goal: enable the reservation of end-to-end network resources to assure a specified “Quality of Service”  User requests minimum bandwidth, start time, and duration  System either grants request or makes a “counter offer”  Network is setup end-to-end with one user request

12. 12 TeraPaths System Architecture Site A (initiator) Site B (remote) WAN web services WAN monitoring WAN web services hardware drivers Web page APIs Cmd line QoS requests user manager scheduler site monitor … router manager user manager scheduler site monitor … router manager

13. 13 TeraPaths Web Services  TeraPaths modules implemented as “web services”  Each network device (router/switch) is accessible/programmable from at least one management node  Site management node maintains reservation etc. databases and distributes network programming by invoking web services on subordinate management nodes  Remote requests to/from other sites invoke corresponding web services (destination site’s TeraPaths or WAN provider’s)  Web services benefits  Standardized, reliable, and robust environment  Implemented in Java and completely portable  Accessible via web clients and/or APIs  Compatible and easily portable into Grid services and the Web Services Resource Framework (WSRF in GT4)

14. 14 TeraPaths Web Services Structure AAA Module (AAA) Remote Negotiation Module (RNM) Network Programming Module (NPM) Advance Reservation Module (ARM) Hardware Programming Module (HPM) Hardware Programming Module (HPM) Hardware Programming Module (HPM) Remote Request Module (RRM) Network Configuration Module (NCM) DiffServ Module (DSM) Route Planning Module (RPM) MPLS Module (MSM) Web Interface … APIs future capability Remote Invocations TeraPaths

15. 15 Site Bandwidth Partitioning Scheme Minimum Best Effort traffic Dynamic bandwidth allocation Shared dynamic class(es) Dynamic microflow policing Mark packets within a class using DSCP bits, police at ingress, trust DSCP bits downstream Dedicated static classes Aggregate flow policing Shared static classes Aggregate and microflow policing

16. 16 Route Planning with MPLS WAN WAN monitoring WAN web services TeraPaths site monitoring (future capability)

17. 17 Experimental Setup  Full-featured LAN QoS simulation testbed using a private network environment:  Two Cisco switches (same models as production hardware) interconnected with 1Gb link  Two managing nodes, one per switch  Four host nodes, two per switch  All nodes have dual 1Gb Ethernet ports, also connected to BNL campus network  Managing nodes run web services, database servers, have exclusive access to switches  Demo of prototype TeraPaths functionality given at SC’05

18. 18 Acquired Experience  Enabled, tested, and verified LAN QoS inside BNL campus network  Tested and verified MPLS paths between BNL and LBL, SLAC (Network Monitoring Project), FNAL, also MPLS/QoS path between BNL and UM for SC’05  Integrated LAN QoS with MPLS paths reserved with OSCARS  Installed DWMI network monitoring tools  Determined effectiveness of OSCARS in guaranteeing and policing bandwidth reservations on production ESnet paths and its effect on improving jitter for applications requiring predictable delays http://www-iepm.slac.stanford.edu/dwmi/oscars/ http://www-iepm.slac.stanford.edu/dwmi/oscars/  Examined impact of prioritized traffic on overall network performance and the effectiveness and efficiency of MPLS/LAN QoS

19. 19 Simulated (testbed) and Actual Traffic BNL to Umich. – 2 bbcp dtd xfers with iperf background traffic through ESnet MPLS tunnel Testbed demo – competing iperf streams

20. 20 In Progress / Future Work T Develop and deploy remote negotiation/response, etc. services to fully automate end-to-end QoS establishment across multiple network domains T Dynamically configure and partition QoS-enabled paths to meet time- constrained network requirements T Develop site-level network resource manager for multiple VOs vying for limited WAN resources T Support dynamic bandwidth/routing adjustments based on resource usage policies and network monitoring data (provided by DWMI) T Integrate with software from other network projects: OSCARS, lambda station, and DWMI  Further goal: widen deployment of QoS capabilities to tier 1 and tier 2 sites and create services to be honored/adopted by CERN ATLAS/LHC tier 0