INFN CNAF TIER1 Network Service Bologna, 1-3-2006 Network Service status and evolution Stefano Zani

WAN Connectivity CERN CERN FZK 10 Gb/s GARR GARR 1Gb/s In 1 Year Enterasys ER16 Old-CORE Extreme BD 10808 NEW-CORE CNAF-CERN (LHC) 10Gb/s T0-T1 10 Gb/s General purpose Wan Access GARR POP Hosted at CNAF 2x10Gb/s Cisco 7600 General Purpose WAN ACCESS Router CNAF(LHC) T1-T1 FZK 2° 10Gb/s T1-T1 Connectivity and T0-T1 backup GARR POP Hosted at CNAF 1Gb/s General Purpos Wan Access Cisco 7600 General Purpose WAN ACCESS Router CNAF-CERN (LHC) 10Gb/s T0-T1 SSR 8600 General Purpose WAN ACCESS Router In 1 Year Enterasys ER16 Old-CORE 2x10Gb/s Extreme BD 10808 NEW-CORE

LAN Elements Tier1’s “BRICK” 42U Rack up to 36-37 Core Switches: servers CORE Switch-Router Core Switches: VLANs (802.1Q) Routing (Wire speed) ACLs (Firewall) N x 1Gb/s Load Sharing Rack Aggregation Switches: VLANs (802.1Q) single port granularity 48 Gigabit Ports 10Gb/s Ready Only few mission critical Servers are connected Directly to the Core. The network is segmented in VLANs Example: LGC Experiments VLAN, KVM Vlan, PowerVlan, etc.. All routing activity is performed at wire speed by the core Switches. .

Core Switch-Routers 2 Core Switches: Expedition ER16 and Black Diamond 10808 ER16 (96xOptical Gb, 2x10Gb) – FULL! 2 CPU Modules (Redundancy) Redundant Power Supply BD10K(128xOptical Gb Ports, 64 Copper Gb,12x10Gb/s) – 3 FREE SLOTS 1Router (Wan Access) Cisco 7600 (Sup7203B) 4x10Gb/2 + 4x1Gb 2 Sup (Redundancy)

28 Extreme Networks Summit 400 Aggregation Switches 28 Extreme Networks Summit 400 48 Gigabit Ethernet Ports (2-4 Optical Fiber) 2x 10Gb/s ports (4 on 28) 160 Gb/s MAX throughput (NON Blocking) Routing and ACL evaluation at Wire speed 12 Enterasys Matrix E1 48 Fast Ethernet Ports + 2 Optical Fiber Gigabit Ports 3 Cisco 3550 48 Fast Ethernet Ports + 2 Optical Fiber Gigabit

Current Network general layout SSR8600 sw-04-03 St.1(gi.3.1,gi.5.1) sw-04-01 sw-04-02 3(3-4) 5(5-6) 7(7-8) LHCBSW1 9(9-10) 49(49-50) St.1(gi.5.1,gi.6.1) 29(29-32) SW-04-06 SW-04-07 SW-04-08 SW-04-09 SW-04-10 SW-03-06 FarmSWF1 St.2(gi.1.1,gi.2.1) St.3(gi.1.2,gi.2.2) St.4(gi.1.3,gi.2.3) St.5(gi.1.4,gi.2.4) St.6(gi.1.5,gi.2.5) St.7(gi.1.6,gi.2.6) St.9(gi.1.8,gi.2.8) St.15(gi.15,16.1,2) gi.3.2 192.16 8.10.73/30 Matrix M5 Sez. Di Bologna sw-06-06 St.12(gi.5.4,gi.6.4) SW-03-07 SW-05-01 SW-05-02 SW-05-03 SW-05-04 SW-05-05 SW-03-08 SW-03-09 St.17(gi.15.4,gi.16.4) St.18(gi.15.5,gi.16.5) St.19(gi.15.6,gi.16.6) St.20(gi.15.7,gi.16.7) St.21(gi.15.8,gi.16.8) St.22(gi.12.6,gi.13.6) St.23(gi.12.7,gi.13.7) SW-06-01 St.24(gi.12.8,gi.13.8) St.16(gi.15.3,gi.16.3) St.25(gi.10.1,gi.11.1) HP Babar St.13(gi.5.5,gi.5.8,gi.6.5) SW-08-04 SW-03-10 SW-03-11 St.28(gi.10.5,gi.11.5) St.29(gi.10.6,gi.11.6) St.30(gi.5.2,gi.6.2) SW-04-04 SW-04-05 St.31(gi.10.7,gi.11.7) St.32(gi.5.6,gi.6.6) SW-06-08 2,3 gi.12.1 SW-08-05 1/1 7600 192.168.150.156 FarmSWG2 1 7:3-8:3 7:4-8:4 7:5-8:5 7:6-8:6 BD FarmSWG4 7:1-8:1 (5/2,6/1) St.34(gi.3.1,gi.4.1) St.36(gi.3.3,gi.4.3) SW-08-03 gi.12.4 gi.12.3 SW-AC St.35(gi.3.2,gi.4.2) St.37(gi.3.4,gi.4.4) gi.12.2 SW-08-02 (sc) 1:1,2:1 St.33 SW-03-02 SW-03-03 SW-03-04 SW-03-05 Bo GARR gi.6.8 FarmSWG1 SW-08-08 SW-08-07 Cat6500 St.11(gi.5.3,gi.6.3) SW-06-05 (xg.7.1,xg.14.1) FarmSWF2 FarmSWG3 TIER1 – Network 10Gb/s

Monitoring The Local Area Network is tuned up to make the Wide Area Bandwidth available to the main services without loss of performances due to bottle necks. Network Monitoring Tools: MRTG :Every port of every switch is monitored with MRTG NAGIOS: All the network infrastructure is controlled with NAGIOS (Email and SMS alarms on critical events) NTOP: An NTOP server is used for troubleshooting or Instant analysis of the traffic.

T1 Security ACLs Are applied (in ASIC) on the WAN interfaces of the Switch Routers and between different internal networks (More than 500 rules!). IDS systems (argus) are in test phase. Logs are centrally managed and log analysis tools are in study. (Now only simple parsing scripts are used in production).

10 Gb/s on Wan (CERN-CNAF) Sperimentation 10 Gb/s on Wan (CERN-CNAF) Tests on 10 Gb/s on LAN and WAN (Recent tests between CERN and CNAF reached 7.5 Gb/s from a single host!) Service challenges support Network configuration and tuning activity for Service Challenges (Make bandwidth available from WAN to disk and Tape servers).

Evolution and milestones Before LHC data taking starts… Better maintenance contracts on Core Switches with on-site intervention in 4-8 hours and guaranteed problem solving (or device substitution) in 1 day. (negotiation in progress…) Second 10 Gb/s For T1-T1 and Backup for T1-T0 Connection (in 2007). Core Switch acquisition (in substitution of the Enterasys ER16) (In one or two years .. Depending on the growth of the center)

Manpower People: Stefano Zani (Staff) Riccardo Veraldi (Staff) Donato De Girolamo (Temporary) Giuseppe Sansonetti (Temporary) only 50% Every member of the Network Service has to be able to operate on the Routing Switching and firewalling infrastructure (3 of us). One more person is necessary to guarantee continuity on service availability (holiday time, Conferences, etc..)