Network to and at CERN Getting ready for LHC networking Jean-Michel Jouanigot and Paolo Moroni CERN/IT/CS

Summary  Current situation  T0-T1 planning: LAN  T0-T1 planning: WAN

21 January 2005T0/1 network meeting3 Current situation  General purpose network  Technical network  Experimental areas (pre-production)  External network  (firewall / HTAR)

21 January 2005T0/1 network meeting4 Technical Network COMPUTER CENTER REMOTE MAJOR STARPOINTS..etc.. Server Farms Firewall CIXP,Internet..etc.. General-purpose network

21 January 2005T0/1 network meeting5 SR2 SR1 SR3 SR4 SR5 SR6 SR7 SR8 Technical network MCR TCR PCR CCR General Purpose Network

21 January 2005T0/1 network meeting6 Tests + LHC pre- production CIXP GÉANT +SWITC H ….…. ….…. External network Internet General purpose network Chicago PoP

21 January 2005T0/1 network meeting7 Firewall This slide is intentionally left blank

21 January 2005T0/1 network meeting8 T0-T1 planning (LAN)  New 2.4 Tb/s backbone to interconnect  LHC experiments (CERN Tier0)  general purpose network  CERN Tier1  T0-T1 WAN (regional Tier1’s)  Based on 10GE technology  Layer 3 interconnections  No central switch(es)  Redundancy via multiple 10GE paths (OSPF)

21 January 2005T0/1 network meeting9 More about T0-T1 LAN  Random paths through the backbone for load balancing (OSPF)  IP addressing:  depends on the LHC WAN implementation,  RFC1918 addresses are likely for a lot of end systems  a data mover facility can help a lot (already successfully implemented for the BABAR experiment at IN2P3)  Default route? Maybe not necessary  Call for tender for the equipment being issued

21 January 2005T0/1 network meeting GE->88*GE ~6000 CPU servers ~2000 Tape and Disk servers GPN 4 LHC experimental areas 4 LHC experimental areas T0-T1 WAN T0-T1 WAN 10GE->88*GE ….…. ….…. 10GE->32*GE 10GE->n*10GE T0-T1 network at CERN (LAN) Raw LHC data Externa l network multiple 10GE 10GE GbE CERN Tier1

21 January 2005T0/1 network meeting11 Tier0 network (LHC experimental areas) T0-T1 LAN DAQ LHC experiment control network GPN Low speed (management) High speed: redundant 10GE (data) T0-T1 WAN CER N Tier1 LHC experiment

21 January 2005T0/1 network meeting12 T0-T1 WAN: progress  A lot of progress has been made:  10 Gb/s equipment is commonly available (although not yet cheap): STM-64 (10GE WAN PHY), 10GE LAN  10 Gb/s capacity (SDH, wavelength, WDM over dark fibre) is affordable  long-distance, high-speed TCP is feasible, although with special Linux tuning

21 January 2005T0/1 network meeting13 T0-T1 WAN: progress (continued)  More progress being made:  GN2 is coming in Europe with new services and research activities  Several interesting initiatives in North America and in Europe (dark fibre-based networks, etc.)  Several interesting monitoring tools exist or are being developed  Pre-production simulation (robust data challenge): a useful ongoing experience  Firewall with HTAR works for non-LHC traffic and for some pre- production

21 January 2005T0/1 network meeting14 T0-T1 WAN: issues  Still several open questions:  how will Tier1’s connect to Tier0 (directly, one upstream, layered upstreams, …)?  backup routing ?  non-homogeneous Tier1 requirements?  any Tier1-Tier1 traffic via Tier0?  IP addressing: routable or RFC1918 ?  does every Tier1 have enough routable addresses?  and …

21 January 2005T0/1 network meeting15 T0-T1 WAN: more issues  …what about  security ?  Tier2’s ?  compatibility between GRID middleware and network design?  special tuning for WAN data transfers?  compatibility between high speed flows and some network devices (Juniper M160)?  management, monitoring, troubleshooting?  Anything else?

21 January 2005T0/1 network meeting16 Recommendations (I)  Allow for diverse regional requirements, but standardise NOW on the T0-T1 physical interface:  10GE LAN PHY (LR/SR ?)  STM-64/OC192  10GE WAN PHY (?)  Other interfaces also possible in the pre-production phase (GbE, multiple GbE, STM-16)  Take advantage of useful experience (robust data challenge)  Define clearly the operational responsibilities across multiple administrative domains

21 January 2005T0/1 network meeting17 Recommendations (II)  Select equipment which is expected to work reliably for some years  A data mover facility (spooling system) helps with several issues:  IP addressing needs  security  WAN data transfer optimisation  Select proven and stable technology: smooth network operations and easy troubleshooting are essential

21 January 2005T0/1 network meeting18 Recommendations (III)  Security is essential  Monitoring is essential  Allocate suitable (routable) subnets, dedicated to LHC production purposes  If not enough routable IP addresses, ask RIPE-NCC for more, via the appropriate upstream LIR, and do so NOW (or ask ARIN, or APNIC, according to the region)

21 January 2005T0/1 network meeting19 Recommendations (IV)  Never mind if the network is just a boring production tool: being at the bleeding edge is not essential in this situation  LHC physics is the research target, not LHC networking

21 January 2005T0/1 network meeting20 LHC WAN: a possible design  Assumptions: if …  Tier1’s connect at layer 3  backup routing is a requirement and it is acceptable via research IP networks (not more than two-three Tier1’s down at the same time)  Tier1-Tier1 traffic is allowed via Tier0 (although this would not be Tier0’s preference…)  Tier1 and Tier0 addresses are publicly routable and every Tier1 has allocated a SMALL number of subnets for inter-Tier0/1 traffic  BGP routing using the “natural” ASN and routable prefixes  no default route (or no default route towards T0): is it possible?  …

21 January 2005T0/1 network meeting21 A possible design (continued)  …and if …  basic security is provided via layer 3 ACLs (allowed subnets and, if possible, port numbers)  Tier1’s may have some non-homogeneous requirements  no Tier2 directly connected to Tier0, but some may be allowed to exchange traffic at less that 10 Gb/s  alternatively, some T0-T2 traffic may transit via an intermediate T1  a spooling system (data mover) is used as buffer between sites to optimise long-distance data transfer and reduce public IP addresses needs  … then …

21 January 2005T0/1 network meeting22 LHC LAN Tier1 ….…. A possible T0-T1 WAN network Externa l network multiple 10GE 10GE or STM-64 Tier1 10GE or multiple GbE Data mover (spool) Tier 2

Thank you Questions?