A follow-up on network projects 10/29/2013 HEPiX Fall Co-authors: IT/Communication Systems HEPiX Fall 2013

Agenda Latest network evolution Network connectivity at Wigner Business Continuity Status about IPv6 deployment TETRA deployment 10/29/2013 3HEPiX Fall 2013

Latest network development 10/29/2013 HEPiX Fall 20134

Upgrade of Geneva data center Migration to Brocade routers completed 2 year project No service interruption Benefits: 100Gbps links Simplified topology (from 22 to 13 routers) Lower power consumption per port Margin for scalability Enhanced features (MPLS, Virtual routing) 10/29/2013 5HEPiX Fall 2013

Backbone 100 Gbps links ∑ 140 Gbps ∑ 20 Gbps ∑ 60 Gbps Internet ∑ 12 Gbps External Network RacksDistribution CERN Data Center today Switching fabric 1.36 Tbps Switching fabric 5.28 Tbps LCG GPN Firewall CORE Network Wigner ∑ 200 Gbps Switching fabric 1 Tbps 100 Gbps links MPLS 10/29/2013 HEPiX Fall 20136

StorageDistribution Routers Backbone Routers 100s of 10 Gbps 100 Gbps links Scaling the Data Center 7 10s of 100 Gbps Try to skip 40 Gbps interface HEPiX Fall /29/2013 CPU

Service capacity depends on Service purpose Blocking Factor: 2 for CPU, 5 for Storage x 1Gbps m x 10 Gbps 10 Gbps x n x 10 Gbps CPU servers Storage servers Distribution Router 8 Scaling the Top of the Rack x 10Gbps m x 100 Gbps n x 100 Gbps 10GBase-T ? 40Gbps ? HEPiX Fall /29/2013

CORE Network Internet ∑ 240 Gbps RacksDistribution Backbone Extending the Tier0 to Wigner 9HEPiX Fall /29/2013 Geneva Switzerland Budapest Hungary

Internet CORE Backbone Routers RacksDistribution Racks Distribution MPLS Backbone WLCG Tier0 10HEPiX Fall /29/2013

Business Continuity 11 HEPiX Fall /29/2013

LCG Extended CORE MPLS Wigner for Business Continuity 12 LCG GPN External Network Firewall External Network Firewall Internet HEPiX Fall /29/2013 Virtual Routers

2 nd Network Hub at CERN 13 Backbone ∑ 140 Gbps ∑ 20 Gbps ∑ 60 Gbps Internet ∑ 12 Gbps External Network RacksDistribution LCG GPN Firewall CORE Network Wigner ∑ 200 Gbps HEPiX Fall /29/2013 Single Building

2 nd Network Hub at CERN 14 ∑ 140 Gbps ∑ 20 Gbps ∑ 60 Gbps Internet ∑ 12 Gbps LCG GPN External Network Firewall CORE Network Wigner ∑ 200 Gbps External Network Firewall CORE Network GPN LCG HEPiX Fall /29/2013

IPv6 deployment at CERN 15HEPiX Fall /29/2013

Network Database: Schema and Data IPv6 Ready Configuration Manager supports IPv6 routing Admin Web: IPv6 integrated 2013 The Data Center is Dual-Stack Gradual deployment on the routing infrastructure starts NTPv6 and DNSv6 DHCPv6 Infrastructure is Dual-Stack Firewallv6 automated configuration User Web and SOAP integrate IPv6 Today HEPiX Fall /29/2013 Automatic DNS AAAA configuration

IPv4 / IPv6 same portfolio Identical performance, common tools and services Dual Stack, dual routing OSPFv2/OSPFv3 BGP ipv4 and ipv6 peers Service managers decide when ready for IPv6 Devices must be registered IPv6 auto configuration (SLAAC) disabled RAs: Default Gateway + IPv6 prefixes no-autoconfig DHCPv6 MAC addresses as DUIDs: painful without RFC6939 ISC has helped a lot (βcode implementing classes for ipv6) DHCPv6 clients might not work ‘out of the box’ 17HEPiX Fall /29/2013

Lots of VMs Two options: A) VMs with only public IPv6 addresses + Unlimited number of VMs - Several applications don't run over IPv6 today (PXE, AFS,...) - Very few remote sites have IPv6 enabled (limited remote connectivity) + Will push IPv6 adoption in the WLCG community B) VMs with private IPv4 and public IPv6 + Works flawlessly inside CERN domain - No connectivity with remote IPv4-only hosts (NAT solutions not supported or recommended) HEPiX Fall Current VM adoption plan will cause IPv4 depletion during /29/2013

TETRA deployment 19HEPiX Fall /29/2013

What is TETRA? A digital professional radio technology E.T.S.I standard (VHF band – MHz) Make use of “walkie-talkies” Voice services Message services Data and other services Designed for safety and security daily operation 20HEPiX Fall /29/2013

The project Update the radio system used by the CERN Fire Brigade 21 Fully operational since early years work A fully-secured radio network (unlike GSM) Complete surface and underground coverage Cooperation with French and Swiss authorities Enhanced features and services HEPiX Fall /29/2013

Which services? Interconnection with other networks Distinct or overlapping user communities security, transport, experiments, maintenance teams… Outdoor and indoor geolocation Lone worker protection 22HEPiX Fall /29/2013

Conclusion The Network is ready to cope with ever- increasing needs Wigner is fully integrated Development of Business Continuity Before end-2013, IPv6 will be fully deployed and available to the CERN community TETRA system provides CERN with an advanced, fully-secured radio network. 23HEPiX Fall /29/2013

Thank you! Question 24HEPiX Fall /29/2013

Some links A short introduction to the Worldwide LCG, Marteen Litmaath – Physics computing at CERN, Helge Meinhard – WLCG – Beyond the LHCOPN, Ian Bird – LHCONE – LHC Use case – LHC Open Network Environment. Bos-Fisk paper – Introduction to CERN Data Center, Frederic Hemmer – The invisible Web – CERN LHC Technical infrastructure monitoring – Computing and network infrastructure for Controls – 25HEPiX Fall /29/2013

Extra Slides 26HEPiX Fall /29/2013

CERN Area~600,000m 2 Buildings 646 Staff and Users14,592 Devices Registered170,475 Data CentersGenevaWigner 2013Wigner 2014 Power3,500KW~900KW~1200KW Racks Servers10,173~1,200~1800 Routers2268+Firewall 100Gbps ports6018 ToR Switches ToR Switching 1Gbps ports22,7763, Gbps ports4, Storage Disks 76,697 Raw disk capacity (TiB) 116,948 Tape Drives 160 Data on Tape (PiB)100 L2 Switching Switches Gbps ports Gbps ports5656 L3 Switching Routers161 1 Gbps ports Gbps ports Gbps ports78 WiFi Access Points1,514 Devices seen/day~7,000 27HEPiX Fall /29/2013

LHCONE LHC Open Network Environment Enable high-volume data transport between T1s, T2s and T3s Separate LHC large flows from the general purpose routed infrastructures of R&E Provide access locations that are entry points into a network private to the LHC T1/2/3 sites. Complement the LHCOPN 28HEPiX Fall /29/2013