1. US CMS Tier1 Facility Network Andrey Bobyshev (FNAL) Phil DeMar (FNAL) CHEP 2010 Academia Sinica Taipei, Taiwan

2. Outline of the talk :  USCMS Tier1 Facility Resources  Data Model/requirements  Current status  Circuits/CHIMAN/USLHC Net  Network tools graphs and snapshots

3. Tape (TB)Disk (TB) CPU (kHS06) SwitzerlandCH-CERN216004500106100 FranceFR- CCIN2P3 3876134211146 GermanyDE-KIT5700195015000 ItalyIT-INFN- CNAF 5200220015000 SpainES-PIC24249026971 TaiwanTW-ASGC2000180016000 UKUK-T1-RAL4192156012058 USAUS-FNAL- CMS 21000650056000 Summary of CMS resources http://gstat-wlcg.cern.ch/apps/pledges/

4. CMS resources (2011 pledges) http://gstat-wlcg.cern.ch/apps/pledges/

5. Model of USCMS-T1 Network traffic cmsstor/dCache nodes Federated File System T0 2.2Gbps 3.2Gbps Data processing /~1600 Worker nodes 30-80Gbps EnStore Tape Robots BlueArc NAS CMS-LPC/SLB Clusters Tier2s/Tiers1 Interactive users 10-20Gbps 1Gbps 3-10Gbps QoS

6. Upgrade of Core Fabric Recently Completed Cisco 6509-based core Cisco Nexus 7000-based core

7. USCMS-T1 Network in 2010-11 $AB, 2010-10-01

8. site-core1 site-core2 r-s-bdr End-To-End Circuits USCMS Tier1 Network in 2013-14 10GbE N x 100GbE ESnet / 100GE 10GbE vPC 100GbE Site Network ?

9. Network use (2%) Interactive (2%) Real-Time (Database,monitoring) (2%) Critical (34%) NAS (10%) Best effort (50%) QoS Classes of Traffic in USCMS-T1 8 Gbps 40Gbps 27.2Gbps 1.6Gbps

10. Redundancy Within the Tier-1 Today:  FCC2 Nexus is core switch fabric GCC-CRB Nexus = redundant core  Switches connected at 4 (or 8) x 10GE Virtual Port channel (vPC) Gateway Load-balancing protocol (gLPB) for failover to redundant core Near Future:  Interconnected Nexus’s @ 80-160Gb/s Function as distributed fabric  Switches still configured w/ vPC & gLPB Most connections to nearest core switch

11. Off-Site Traffic: End-to-End Circuits USCMS-T1 has a long history of using ESNet/SDN and Internet2 DCN circuits SLA monitor, IOS Track objects to automatically fail over traffic if circuit is down

12. PerfSonar Monitoring  Monitoring status of circuits  Alert on a change of link status  Utilization  PingEr RTT measurements  PerfSonar-BUOY – Active measurements, BWCTL & OWAMP o Two NPI Took kit boxes  Two LHCOPN/MDM monitoring boxes

13. Circuits SLA monitoring (Nagios) Each circuit has an SLA monitor running icmp-echo application. Status of SLA monitor is tracked by SNMP

14. Weather Map Tools (I) SNMP-Based

15. Weather Map Tools (II) Flow Data-Based Drill-down Capabilities

16. Work in progress:  New monitoring approaches  Any-to-Any/cloud-to-cloud performance  Any production host can become an element of monitoring infrastructure  Combination of passive and active measurements  10GBase-T/IEEE 802.3an for end system  Intel E10G41AT2 NIC PCI-Express  AristaNetworks DCS-7120T-4S (as ToR solution)  Directly to Nexus7K (via fiber at the moment)  Cat6E regular physical infrastructure deployed ~100m

17. Summary  Two buildings, four computing rooms (fifth one is coming)  Two Nexus 7000 switches for 10G aggregation, interconnected 80-160Gbps  2 x10Gbps to the Site Network (read/write data to tapes )‏  10Gbps to the Border Router (non-US Tier2s, other LHC- related traffic)‏  20Gbps toward ESNET CHIMAN and USLHCNET, SDN/DCN/E2E circuits

18. Summary (continued)  ~200 dCache nodes with 2x1GE  ~1600 worker nodes with 1GE  ~150 various servers  2X 20G for BlueArc NAS storage  C6509 access switches connected by 40-80GE  Redundancy/loadsharing at L2 (vPC) and L3 (GLBP)‏  IOS based Server Load Balancing for interactive clusters  19 SDN/DCN End-To-End Circuits  Virtual port channelling (vPC)  QoS, 5 major classes of traffic

19. Additional slides

20. USCMS Tier1: Data Analysis Traffic R-CMS-FCC2 R-CMS-FCC2-2 R-CMS-FCC2-3 30 of 40G 18 of 20G 29.8 of 30G 65 of 80G 20 of 20G 25 of 30G 20 0f 30G 18 of 30G 30G S-CMS-GCC-1 S-CMS-GCC-3 S-CMS-GCC-5 S-CMS-GCC-4 S-CMS-GCC-6 R-CMS-N7K-GCC-B https://fngrey.fnal.gov/wm/uscms