1. COMPUTING for ALICE at WLCG TIER-2 SITE in PRAGUE in 2015/2016
D. Adamova, J. Chudoba, J.Svec, V. Rikal, A. Mikula, M. Adam and J. Hampl
INFN Catania,

2. Outline Status of the WLCG Tier-2 site in Prague
ALICE operations in 2015/2016
Report on the delivery of mandatory resources
A couple of various issues
Summary and Outlook

3. 10Gbps dedicated fiber link
The geographical layout
Nuclear Physics Institute AS CR (NPI)
a large ALICE group, no ATLAS involvement
A (smaller) part of ALICE storage resources
10Gbps dedicated fiber link
Institute of Physics AS CR (FZU)
Regional computing center
WLCG Tier-2 site praguelcg2
All the CPU resources for ALICE
A (larger) part of storage resources for ALICE
Quite small ALICE group, much larger ATLAS community
Originally, the xrootd storage servers for ALICE were only at NPI. Subsequently, xrootd servers were also put in operation at FZU and now even the redirector is there.

4. HEP Computing in Prague: site praguelcg2 (a.k.a. the farm GOLIAS)
A national computing center for processing data from various HEP experiments
Located in the Institute of Physics (FZU) in Prague
Basic infrastructure already in 2002,
but officially started in 2004
Certified as a Tier2 center of LHC Computing Grid (praguelcg2)
Collaboration with several Grid projects.
April 2008, WLCG MoU signed by Czech Republic (ALICE+ATLAS).
Very good network connectivity: Multiple dedicated 1 – 10 Gb/s connections to collaborating institutions and 2x10 Gb/s connection to LHCONE (upgraded from 10 Gb/s on ).
Provides computing services for ATLAS + ALICE, Auger, Nova, CTA, D0, Solid state physics ... 4

5. Some history: 2002 -> 2016 Started in 2002 with:
32 dual PIII 1.2GHz,
1 GB RAM, 18 GB SCSI HDD, 100 Mb/s Ethernet rack servers
Storage - disk array 1TB:
HP server TC4100

6. OCTOBER 2002

7. ALICE PDC 2004 resource statistics: 14 sites (Prague 3. 5%, Catania 7
ALICE 2014 resource statistics: Prague 2.9% from 74 sites (Catania 1.3%)
(87 sites in 2016) 7 7

8. ALICE PDC resources statistics - 2005
25 sites in operation
Running jobs (8 November 2005)
Farm
Min
Avg
Max
Sum
1160
1651
1771
CCIN2P3
134
210
231
CERN-L
268
286
304
CNAF
255
362
394
FZK
531
600
Houston 3 14
Münster 2 58 81
Prague 43 61 71
Sejong
Torino 33 41 8

9. Some recent pictures Servers xrootd1 (125 TB) and xrootd2 (89 TB)
Server xrootd5 (422 TB)

10. Current numbers
1 batch system (torque + maui)
2 main WLCG VOs: ALICE, ATLAS
Other users: Auger, Nova, CTA, FNAL's D0 user group
220 WNs, 4556 CPU cores, published in the Grid
~4 PB on disk storage (DPM, XRootD, NFS), a tape library
Regular yearly upscale of resources on the basis of various financial supports,
mainly the academic grants.
The WLCG services include:
Apel publisher, Argus Authorization service, BDII, several UIs, ALICE vobox, Cream CEs,
Storage Elements, lcg-CA
Monitoring: Nagios, Munin, Ganglia, Netflow, Observium
Log processing and visualization: ElasticSearch, Kibana + Logstach, i.e. the ELK stack
Services management by Puppet
The use of virtualization at the site is quite extensive.
ALICE disk XRootD Storage Element ALICE::Prague::SE
~ 1.6 PB of disk space in total
Redirector+ 5 FZU, 4 NPI Rez
a distributed storage cluster
10 Gb/s dedicated connection from FZU to the NPI ALICE storage cluster 10

11. Annual growth of total on site resources since 201211

12. External connectivity, by CESNET
Core load on external connections comes from LHC Experiments ATLAS and ALICE.
Before the upgrade to 2x10 Gbps the LHCONE link was often saturated up to the limit.

13. IPv6 deployment
All worker and storage nodes have IPv4 and IPv6 addresses.
All local transfers via gridftp protocol utilize preferably IPv6.

14. All disk servers are connected
via 10 Gbps to the central
CISCO Nexus 3172PQ switch.
Worker node connection is
1 Gbps to top of the rack switches that have 10 Gbps connection to the central switch.

15. The ELK Stack is used for log processing and visualization.
An example of Kibana filtering.

16. ALICE operations in 2015
The number of concurrently running ALICE jobs at FZU in 2015 was in peaks over 3300.
Due to a steady operation and active usage of idle opportunistic resources, ALICE was the largest CPU consumer in 2015.
ATLAS
ALICE
Running jobs profile in 2015 as shown in ALICE MonALISA.
Almost 5.5 million of jobs was processed in Prague during
Avg. number of jobs 1736, peaks up to 3374.

17. ALICE operations in 2016
Running jobs profile at the Prague Tier-2 in Nov – Nov The ALICE share in 2016 is lower than in Opportunistic resources are not so easy to use because ATLAS and NOVA are very active.
ATLAS
ATLAS
multicore
ALICE
Running jobs profile in 2016 as shown in ALICE MonALISA. The avg. number of jobs is 1238, in peaks up to 3044.

18. CPU resources share & site availability in 2015
During 2015, there was a continuous uninterrupted ALICE data processing at the computing center at FZU.
The Czech Republic share in the Tier-2 CPU resources delivery was ~4.6%.
Cf. Catania 1.6% + 3.3%
100% availability in November 2015

19. CPU resources share & site availability in 2016
In 2016, the Czech Republic share in the
Tier-2 CPU resources delivery dropped
to ~3%.
Cf. Catania 0.4% + 2.1%
Almost 100% availability in 2016

20. Upgrade of the xrootd cluster
Storage in 2016
Outgoing traffic on ALICE::Prague::SE in 2016
> 4.7 GBytes/s in peaks
PBytes of data read
Upgrade of the xrootd cluster
Traffic on the dedicated line
NPI <-> FZU. Maximum rate close
to 4 Gbits/s, Which corresponds
to 4 servers with 1 Gb/s Ethernet
cards.
This year we will hopefully finish an upgrade of the performance of the NPI cluster: the servers will be equipped with the 10 Gb/s Eth cards and the corresponding switch will be
replaced with a more efficient one
(Cisco Catalyst 4500-X 16 Port 10G). As shown on a previous slide, the storage nodes at ZU have already the 10 Gb/s connectivity.

21. traffic to/from XRootD storage nodes xrootd5 and xrootd4 on a local switch

22. Czech Republic resource share
MANDATORY ACCORDING TO THE ALICE CONSTITUTION: 2.1% of the total required Tier-2 resources
Resources delivery in 2015: CPU (kHepSpec06) ALICE requirement: 7,2 REBUS info: pledged 3,5 REBUS info: delivered 8,86 Resources delivery for 2016: REBUS info: pledged 5,0 REBUS info: delivered 8,83
Resources delivery in 2015:
Disk (PB):
ALICE requirement: ,6
REBUS info: pledged ,030
REBUS info: delivered 1,591
Resources delivery for 2016:
ALICE requirement: ,92
REBUS info: pledged , 4
REBUS info: delivered 1,791
For the ALICE Czech group easier to scale up the storage capacity than to pile up CPUs.
ALICE requirements in 2015 of computing and storage resources were satisfied
(for storage to > 200% ), although our CPU pledges were low :
- by sharing the hardware between various projects
by stable operations with almost no outage
by using the servers out of warranty
ALICE requirements in 2016: looks like we will deliver OK, in excess of what is required especially with storage

23. Czech Republic resources for ALICE – an outlook into 2017
The outlook for 2017 is not optimistic.
A couple of ALICE servers will probably have to be removed from production.
We are currently waiting the outcome of the submission of a grant application. If it is accepted, we plan to purchase in the end of 2017 a CPU cluster with a performance of 20 – 25 kHS06.
Will be shared between ATLAS and ALICE.
If we manage to get this new cluster, then for 2018 we can pledge for ALICE 6 kHS06.
Otherwise we have to rely on utilizing opportunistic resources, the idle CPUs provided by other projects using the site.

24. A couple of various issues
The CPU resources of the ALICE project in the CR were not upgraded since 5 years ago.
In 2016, we had to switch off one of the obsolete clusters of ALICE and we are able to deliver CPU share required by ALICE experiment only by utilizing opportunistic resources.
We hope for a break in this situation in 2017.
The status of the storage is much better. By now we deliver most (~ 1.6 PB) of all the ALICE Tier-2 sites. ( ALICE::Catania::SE 1.1 PB, quite close)
Most of the nodes of the ALICE::Prague::SE are upgraded to the latest recommended XRootD version v
New redirector machine was put into operation recently, without the client functionality, due to overload of the original redirector+client. We were the first users of the new installation method using RPMs released in July 2015.
The process of upgrading the NPI storage cluster to 10 Gb/s connectivity is ongoing.

25. Aspects of future upscale
The future LHC runs will require a huge upscale of computing, storage and network resources
which will be impossible for us to achieve with the expected flat or even declined budget.
Activities towards getting access to external resources, HPC centers or computational clouds are ongoing at all LHC experiments.
On the local level, there are lengthy discussions with the HPC center IT for Innovations (IT4I)
in Ostrava.
Another possibility: using services of the OpenNebula IaaS cluster/cloud of CESNET (MetaCloud, FIWARE lab) or CERIT-SC, another academic computing center providing cloud services. Not really significant amount of resources for us.
The showstopper is the absence of tools for submission of ALICE jobs to these sites.

26. NGI_CZ  all the events and relevant information about praguelcg2
Since 2010, NGI_CZ is recognized and in operation:
 all the events and relevant information about praguelcg2
2 sites involved: praguelcg2 and prague_cesnet_lcg2
significant part of the services provided by the praguelcg2 team
Services provided by NGI_CZ for the EGI infrastructure:
Accounting (APEL, DGAS, Cesga portal)
Resources database (GOC DB)
Operations -
ROD (Regional Operator on Duty)
Top level BDII
VOMS servers
Meta VO
User support (GGUS/RT) -
Middleware versions: UMD 3.0.0, EMI 3.0

27. Summary and Outlook
- Czech Republic was performing well during 2015/2016 concerning delivery
of computing and storage resources despite the problems with zero support for the upgrade of CPU resources
- Mandatory requirements satisfied to > 100%.
- High-level accessibility, reliable delivery of services, fast response to problems.
- Into the upcoming years, we will do our best to keep up the reliability and performance level of the services and deliver the pledges.
- Crucial is the high-capacity network infrastructure :
- Endeavour to upgrade the NPI storage nodes for higher traffic performance.
- Planned upgrade of the network system at the Tier-2 site at FZU.
- We plan to test availability for our production of existing Czech academic cloud resources like CERIT-SC.
- We’ll try to be prepared to migrate to a cloud site from the established grid site in agreement with possible WLCG trends (OpenNebula/OpenStack).
- Work on the development of the local ALICE monitoring tools is planned.

28. Backups

29. 2006
2007
ALICE vobox set-up
fixing problems with the vobox proxy (unwanted expirations)
AliEn services set up
manually changing the RBs used by the JAs
 successful participation in the ALICE PDC'06:
Prague site delivered ~ 5% of total computing resources (6 Tier1s, 30 Tier2s)
Problems with the fair-share of the site local batch system (then PBSPro)
still problems with functioning of the ALICE vobox-proxy
during the PDC'07 problems with job submission due to malfunctions of the default RBs  the failover submission configured
Prague site delivered ~ 2.6% of total computing resources (significant increase of the number of Tier-2s)
2009
2008
migration to the Torque batch system on a part of the site: some WNs on 32bit in PBS and some on 64bit in Torque
installation and tuning of the creamCE
hybrid state:
‘glite’ vobox and WNs, 32bit
‘cream’ vobox submitting JAs directly to creamCE  Torque, 64bit
Dec: ALICE jobs submitted only to the creamCE
migration to gLite3.1 ALICE vobox on 64-bit SLC4 machine
upgrade of the local CE serving ALICE to lcg-CE 3
repeating problems with job submission through RB's in Oct. the site re-configured for the WMS submission

30. 2010 2011 creamCE 1.6 / gLite 3.2/sl5 64bit installed in Prague
  we were the first ALICE Tier-2 where cream1.6 was tested and put in production
NGI_CZ set in operation
2011
Start of IPv6 implementation
The site router got an IPv6 address
Routing set-up in special VLANs
ACLs directly implemented in the router
IPv6 address configuration: DHCPv6
Set-up of an IPv6 testbed

31. 2013 2012 Almost all machines migrated to SL6
Optimization of the ALICE XRootD storage cluster performance
an extensive tuning of the cluster motivated by a remarkably different performance of the individual machines:
data was migrated from the machine to be tuned to free disk arrays at another machine of the cluster.
the migration procedure done so that the data was accessible all the time.
the empty machine re-configured.
number of disks in one array reduced.
set-up of disk failure monitoring.
raid controller cache carefully configured.
readahead option set to a multiple of (stripe_unit * stripe_width) of the underlying RAID array.
no partition table used to ensure proper alignment of the file systems: they were created with right geometry options ("-d su=X, w=YYk“ mmkfs.xfs switches).
mounting performed with the noatime option.
Parameters of one of the optimized XRootD servers before and after tuning
2013
Almost all machines migrated to SL6
CVMFS installed on all machines
Connected to LHCONE

32. Status in 2014 Rather close to what we have now
1 batch system (torque + maui)
2 main WLCG VOs: ALICE, ATLAS
~ 4000 cores published in the Grid
~ 3 PB on new disk servers (DPM, XRootD, NFS)
Our CPU/average running jobs share among ALICE Tier-2s was 5.6% - better than in 2015 and (Catania 2.4%, similar to the current rate)
The connection to LBCONE was only 10 Gb/s
ALICE disk XRootD Storage Element ALICE::Prague::SE
~ PB of disk space in total
Redirector/client + 3 FZU, 5 NPI Rez
a distributed storage cluster 32