1. Brookhaven Science Associates U.S. Department of Energy 1 Network Services LHC OPN Networking at BNL Summer 2006 Internet 2 Joint Techs John Bigrow July 18, 2006

2. Brookhaven Science Associates U.S. Department of Energy 2 Network Services n LHC Overview (very simple overview, I’m not a physicist) LHC / Atlas Experiments Overview (The What) The Physics Architecture (The Why) Preliminary Network and Security Architecture (The How)

3. Brookhaven Science Associates U.S. Department of Energy 3 Network Services CERN Accelerator Ring Aerial View

4. Brookhaven Science Associates U.S. Department of Energy 4 Network Services

5. Brookhaven Science Associates U.S. Department of Energy 5 Tier 1 Tier2 Center Online System CERN ~5M SI2K >1 PB Disk Tape Robot BNL: ~2M SI2K; 2PB Tape Robot IN2P3 Center INFN Center RAL Center Institute Workstations < GBytes/sec 2.5 Gbps 100 - 1000 Mbits/sec Physics data cache ~PByte/sec ~10 Gbits/sec Tier2 Center ~2.5 Gbps Tier 0 +1 Tier 3 Tier 4 Tier2 Center ATLAS Experiment CERN:Outside Resource Ratio ~1:2 Tier0:(  Tier1):(  Tier2) ~1:1:1 Tier 2 Tier 0: DAQ, reconstruction, archive Tier 1: Reconstruction, simulation, archive, mining and (large scale) analysis Tier 2+: Analysis, simulation Tier 3+: Interactive analysis Network Services

6. Brookhaven Science Associates U.S. Department of Energy 6 130.199.48.0…… 130.199.185.0 130.199.48.0 The same host name for dual NIC dCache door is resolved to different IP addresses depending on which DNS is inquired. Network Services

7. Brookhaven Science Associates U.S. Department of Energy 7 Network Services

8. Brookhaven Science Associates U.S. Department of Energy 8 Network Services

9. Brookhaven Science Associates U.S. Department of Energy 9 Other connections MAN LAN CERN (?) NLR ESnet GEANT, etc. BNL internal Network Services

10. Brookhaven Science Associates U.S. Department of Energy 10 Network Services

11. Brookhaven Science Associates U.S. Department of Energy 11 n Network Security Limitations Current firewall Architecture –6 virtual 1 Gb/Sec EtherChannel to Catalyst backplane –Rated total throughput of 5 Gb/Sec –EtherChannel Overhead Loss –Single 1 Gb/Sec flow / interface New Cisco ACE blade might address these limitations Network Services

12. Brookhaven Science Associates U.S. Department of Energy 12 n Network Security Limitations (Continued) Current Router Architecture –Single Access Control List (ACL) / interface -1 inbound and 1 outbound per interface -Default behavior Implicit deny -Policy route map for traffic flow –A single ACL can become unwieldy in a complex WAN environment (what are the network prefixes, DHCP, NAT) –Manual changes to the route map for additional access Network Services

13. Brookhaven Science Associates U.S. Department of Energy 13 n BNL LHC Overview cont. Networking resources –IP Address space allocations / access –10Gig interfaces / 20Gig Etherchannels –Performance Monitoring Network Services

14. Brookhaven Science Associates U.S. Department of Energy 14 n IP Address Allocation Tier 0 to Tier 1 (BNL - CERN) Requires routable IP Address space Direct dedicated access with CERN to / from BNL Limited route advertisements between T0 and T1 –For the LHC OPN Circuit BNL will use 192.12.15.0/24 –No direct T1 to T1 access through CERN at this time Network Services

15. Brookhaven Science Associates U.S. Department of Energy 15 n BNL OPN to Tier 2 and others Tier 2 and other traffic dependant on Internet connectivity –Path to BNL via all service providers (ES Net now, NYSERNET, Broadwing in the future ?) –Dedicated paths to other institutions welcome (you buy) Network Services

16. Brookhaven Science Associates U.S. Department of Energy 16 Network Services

17. Brookhaven Science Associates U.S. Department of Energy 17 n Future BNL LHC OPN Enhancements Dedicated Cisco Firewall Service Modules (ACE) when available –Eliminate router ACL Functionality / Maintenance –Connection Logging –Each FWSM circuit will not impede the 10 Gb/Sec. –Stateful FWSM redundancy IDS / IPS when available Network Services

18. Brookhaven Science Associates U.S. Department of Energy 18 Network Services

19. Brookhaven Science Associates U.S. Department of Energy 19 Network Services n Mon browser-based IP service monitor Internet-centric WAN based monitor application Interrogates essential BNL network services

20. Brookhaven Science Associates U.S. Department of Energy 20

21. Brookhaven Science Associates U.S. Department of Energy 21 Network Services n MonaLisa Java based SNMP monitoring tool n External WAN based monitor n Tracks BNL 10G/Sec. Interfaces n Firewall Service Module n 20 Gb/Sec. Uplinks to the BNL core

22. Brookhaven Science Associates U.S. Department of Energy 22 Network Services

23. Brookhaven Science Associates U.S. Department of Energy 23 Network Services

24. Brookhaven Science Associates U.S. Department of Energy 24 Network Services n Cacti SNMP monitoring tool Replacement for MRTG Tracks most BNL core network interfaces Firewall Service Module EtherChannel interfaces also

25. Brookhaven Science Associates U.S. Department of Energy 25 Network Services

26. Brookhaven Science Associates U.S. Department of Energy 26 Network Services

27. Brookhaven Science Associates U.S. Department of Energy 27 Network Services

28. Brookhaven Science Associates U.S. Department of Energy 28 Network Services

29. Brookhaven Science Associates U.S. Department of Energy 29 Network Services

30. Brookhaven Science Associates U.S. Department of Energy 30 n Thanks (a few kind words to so many) Thanks to the many individuals and groups who have donated their time, code, and talents to make the Internet what it is today. Without their efforts, this infrastructure we take for granted would not exist. We owe many our gratitude. Network Services

31. Brookhaven Science Associates U.S. Department of Energy 31 Questions/Comments ??? Network Services

32. Brookhaven Science Associates U.S. Department of Energy 32 BNL Points of Contact n Scott Bradley, Manager of Network Services 631.344.5745, bradley@bnl.gov n John Bigrow, Senior Network Architect 631.344.2648, big@bnl.gov Network Services