1. INFN CNAF TIER1 and TIER-X network infrastructure
Pre GDB
CERN, 10/1/2017
Stefano Zani

2. INFN CNAF data-center
CNAF runs the main INFN computing center providing computing and storage services to ~30 scientific collaborations
4 LHC
(accounting for ~80% of the installed resources)
several non-LHC
(mainly astro-particle physics experiments)
CNAF is the main computing center of the INFN and it is providing computing and storage to 30 scientific collaborations
4 LHC Experiments and several non LHC… Mainly astro particle experiments
CNAF datacenter is one of the ~14 WLCG Tier-1s

3. INFN CNAF Computing resources
FARMING (NOW): Cores ->221 KHS-06 ~ 20K Job Slots
FARMING 2017: Cores
FARMING 2018: Cores
STORAGE (NOW): ~22 net PB of disk and ~46 PB of tapes
STORAGE 2017: 27PB Disk, 69 PB Tape
STORAGE 2018: 40PB Disk, 90 PB Tape
2016 CPU shares
These are some numbers: Farm is composed by about 20 thousand Cores
2016 DISK shares
2016 Tape shares

4. WAN@CNAF (Staus) LHC ONE LHC OPN RAL SARA (NL) PIC KR-KISTI TRIUMPH
BNL
FNAL
TW-ASGC
NDGF
KR-KISTI
RRC-KI
JINR
LHC ONE
Main Tier-2s
IN2P3 T1
GARR Mi1
KIT T1
GARR-MI
General IP
NEXUS 7018
GARR Bo1
GARR BO1
GARR-BO
60 Gb Physical Link (6x10Gb)
Shared by LHCOPN and LHCONE.
20 Gb/s For General IP Connectivity
20Gb/s
60Gb/s
LHCOPN 40 Gb/s CERN
LHCONE up to 60 Gb/s (GEANT Peering)
Cisco6506-E
CNAF TIER1

5. { WAN bandwidth usage OPN Upgrade from 2x10 to 4x10 Gb/s LHC OPN
60% GRID-FTP
40% Xrootd
TOTAL LHC OPN+ONE (6x10Gb/s) {
MAX IN: 40 Gb/s AVERAGE IN: Gb/s
MAX OUT: 39.1Gb/s AVERAGE OUT: Gb/s
LHC ONE
MAX IN: 58.8 Gb/s AVERAGE IN: Gb/s
MAX OUT: 48.8 Gb/s AVERAGE OUT: Gb/s
52% Xrootd
48% GRID-FTP
ANALISYS AT A GLANCE
WAN connection 6x10Gb/s is well used but not saturated
LHC OPN is used more intensely than LHCONE and sometime reaches 40 Gb/s.
Data movers are able to use all the bandwidth available when needed.
MAX IN: 25.9 Gb/s AVERAGE IN: Gb/s
MAX OUT: 48.8 Gb/s AVERAGE OUT: Gb/s
MAX IN
AVERAGE IN
MAX OUT
AVERAGE OUT

6. GARR NETWORK EVOLUTION
GARR has completed last year the GARR-X Progress project in southern Italy connecting Bari, Catania, Cosenza and Naples with new generation technology devices (Infinera DTN-X XTC-10) Capable of connecting end sites at multiple 100Gb/s
In June 2017 GARR will extend multiple 100Gb/s lambdas capability also to Centeral and Northern Italy introducing alien waves on the existent fiber infrastructure.

7. Network Upgrade using ALIEN WAVES
Slide kindly provided by GARR (M.Carboni)

8. Local Area Network interconnection schema (NOW)
LHCONE
LHCOPN
General INTERNET
IT Resources
TIER1 current LAN Infrastructure
1 CORE Switch Router (Nexus 7018 ~300 10Gb/s ports)
Farming worker nodes are connected through aggregation switch (TOR) with 2 or 4 x10Gb/s
Disk Servers are directly connected to the CORE
Older servers at 1x10Gb/s
More recent servers at 2x10 or 4x10 Gb/s
cisco
6506-E
nexus
7018
4x10Gb/s
10Gb/s
Disk Servers
Switch Farming
Worker Nodes
Up to 4x10Gb/s
Essetially all the scientific resources are connected through a nexus 7018 and the IT resources are connected to a catalyst 6506-E.

9. TIER1 Network evolution
LHC OPN/ONE
2 new Core switch/routers (Nexus 9516) are entering production before end of January 2017.
Scalability (Up to GE ports or GE ports or GE ports). Current resources uses only 3 slots per core ->13 slots for next years expansions
A 6x40 GE card inserted on Nexus 7018 to connect the new infrastructure
Servers connected with more than one interface and switches will be connected to both COREs
The geographical links will be moved on the two COREs as well
Than we’ll be ready to upgrade the WAN Uplink with multiple links at 40 or 100 Gb/s
100 G
100 G
VPC Link
Nexus 9516
Nexus 9516
4x40Gb/s
New infrastructure is necessary for two reasons: Resiliency ->2 Cores and Scalability Numbers of high speed ports scalability
Nexus 7018
LHC OPN/ONE
Single Homed

10. INFN Main Computing Facilities
T2 PADOVA/LEGNARO
Total Cores:5200 (55000 HEPSpec)
Disk Space: 3000 TB
Geographical connectivity GARR: 20 Gb/s
T2 MILANO
Total Cores: 2448 (22720 HEPSpec)
Disk Space: 1850 TB
Geographical connectivity GARR : 10 Gb/s
T2 TORINO
Total Cores: 2000 (21000 HEPSpec)
Disk Space: 2200 TB
Geographical connectivity GARR: 10 Gbs
T1 CNAF BOLOGNA
Total Cores:21250 ( HEPSpec)
Disk Space: TB
Geographical connectivity GARR: 80 Gb/s
Tape Space: TB
T2 PISA
Total Cores: ( HEPSpec)
Disk Space: 2000 TB
Geographical connectivity GARR 20 Gb/s
T2 FRASCATI
Total Cores: 2000 (20000 HEPSpec)
Disk Space: 1350 TB
Geographical connectivity GARR : 10 Gb/s
10 Computing Centers
Total Resources (NOT ony LHC)
Total Cores: (677 KHSpec)
Disk Space: 44 PB
Tape Space: 45 PB
Connectivity (GARR) : 220 Gb/s
T2 ROMA
Total Cores: (32373HEPSpec)
Disk Space: 2160 TB
Geographical connectivity GARR : 10 Gb/s
T2 BARI
Total Cores: 9000 (81000 HEPSpec)
Disk Space: 4160 TB
Geographical connectivity GARR : 20 Gb/s
T2 Napoli
Total Cores: 8440 (HEPSpec)
Disk Space 2805 TB
Geographical connectivity GARR 20 Gb/s
This is a picture of the main INFN computing facilities
1 TIER1 and 9 TIER2 centers.
T2 CATANIA
Total Cores: 3000 (30000 HEPSpec)
Disk Space: 1500 TB
Geographical connectivity GARR : 20 Gb/s

11. Some activities for a (semi-)elastic Data Center extension
First production use case: part of 2016 pledged resources for WLCG experiments at CNAF (2k Cores 21 KHS06) are in Bari-ReCaS Datacenter extension on 20 Gb/s dedicated link Bologna- BARI (800Km) provided by GARR.
OK for low I/O demanding jobs, critical for I/O intensive Jobs (Local Analysis).
Cloud bursting on commercial provider.
Participation to Helix Nebula SCicloud EU PCP project
Tests of opportunistic computing on Cloud providers
(POCs with: ARUBA, Unicredit, CloudItalia, Microsoft)
Foreseen a huge increase in request for CPU there is strong interest in usage of remote resources to (dynamically) extend the Tier-1 farm

12. Conclusions
CNAF computing center can grow in terms of storage and farming resources. Some improvements in cooling system are on going to fit RUN3 resources.
LAN: Tier1 core network can scale in terms of 40 or 100 Gb Ethernet ports
Evaluating to integrate or change the passive cabling system from multimode duplex fibers to MPO multimode fibers or single mode duplex fibers (Depending on the transceiver’s costs).
WAN: Tier1 and Tier2s could be connected at 100Gb/s starting from this summer with the completion of the 100G backbone with the introduction by GARR of the alien lambdas.
Data center extension is an option to increase rapidly the TIER1 computational power using resources available on different datacenters connected with dedicated bandwidth (For example L3 VPN on dedicated circuits provided by GARR or other P2P technology)
Other extensions to “Closer” datacenters are under investigation
Lower latency
More bandwidth for less money
Tier1 extension on public cloud provider is also an opportunity (different POCs are on going and some tools for a “Transparent extension” have been developed).
1) Scalability guaranteed to fit all RUN 3 inside the current computing center.
2) The Core Switch infrastructure can potentially scale up to more than a thousand 100 GE Ports. Not clear the cabling evolution
3) Geographical connections will be improved by the introduction of alien lambdas on GARR backbone. TIER1 potentially connected at Nx100Gb/s
4) Data center extension could be a good solution to optimize the use of resources available on the Italian computing infrastructure. (Limits encoutered for I/O intensive wprkflows)
Scouting for “Closer” datacenter interconnection (Metro distance)
5) Tier1 extension on public cloud provider is also an opportunity (different POCs are on going and some tools for a “Transparent extension” have been developed).

13. That’s it!

14. Backup Slides

15. Experiments requirements (Computing models evolution)
The total amount of data to be analyzed and the necessary CPU power are essential to dimension the Farming and Storage infrastructure.
I/O requirements and the computing models are fundamental for a correct network evolution design.
Per Job I/O requirements Necessary to size the LAN resources.
5MB/s per Job in case of 32 cores per mainboard (commodity 2016) is the limit connecting WNs at 1 Gb/s  1,2 Gb/s per WN
Ratio between Local Data Access/ Remote Data access –>Necessary to size WAN access and design the geographical interconnections between Research networks .
As the total amount of data to be analyzed and the CPU necessary for the computations are essential to dimension Farming and Storage as JOB I/O requirements and computing models are essential to design the network …

16. Network evolution driving factors
Increasing Number of COREs per CPU means More I/O per board
More high speed ports at the CORE and in the aggregation switches
Storage technologies: HD SSD NVMe moving to 3 GB/s capable devices
Storage Area Network technologies moved beyond 100 Gb/s
Mellanox evolved IB from EDR (100Gb/s) to HDR (200 Gb/s)
Intel OmniPath works at 100Gb/s (Cheaper solution than IB)
Disk servers connected at 100 Gb (Or multiple 25/50 or 40 Gb Ethernet)
Network components costs are relevant:
100GE port costs are falling but transceivers are still expensive and in many cases cabling systems have to be changed.
25 and 50 Gb Ethernet are quite “young” and transceivers are not “Commodity”.
40 Gb Ethernet (an apparently stillborn technology) with BiDi transceivers seems to be a cost effective technology (it costs about 2 times a 10GE transceiver and uses the twin multimode 50/125 fibers standard cabling systems). NO MPO or Single Mode cabling required.
Other technological Network evolution factors are: New CPU cores density and very fast storage systems and interconnection technologies.

17. The GARR-X network More that 15.000 km of GARR owned fibers
~9.000 Km of backbone
~6.000 Km of access links
About 1000 user sites interconnected
> 1 Tbps aggregated access capacity
> 2 Tbps Backbone Capacity
2x100 Gbps IP capacity to GÉANT
Cross border fibers with Slovenia, Switzerland
… work in progress with France (Frejus)
Several 10G e2e provided by GÉANT
> 100Gbps Internet Commodity
Direct peering with Google + Cache
Akamai Cache
MIX+NAMEX+TOPIX+VSIX, ecc.

18. INFN CNAF Computing resources
FARMING: Cores ->221 KHS-06 ~ 20K Job Slots
GRID access and “Non GRID” Access allowed
1 Batch System (IBM LSF)
LSF very robust and reliable (but expensive!)
Evaluating HTCondor adoption in about 1/2 years
STORAGE: Currently ~22 net PB of disk and ~46 PB of tapes
1 tape library with slots (currently up to 85 PB capacity)
17 T10kd drives managed by 6 servers
Standard HSM service for all experiments
GEMSS (Grid Enabled Mass Storage System) i.e. GPFS “glued” with TSM
Both local and grid access (via StoRM)
Standard protocol set (file, GridFTP, XrootD, http/webdav)
2016 CPU shares
These are soma numbers: Farm is composed by about 20 thousand Cores
2016 DISK shares
2016 Tape shares