1. Future challenges for the BELLE II experiment
Miniworkshop di CCR e Corso di formazione su: "HTCondor per
amministratori di batch system"
Bologna 24/10/2016
Dr. Silvio Pardi
INFN-Napoli

2. Outline Belle II Collaboration The Computing Model
The production system: DIRAC
Monte Carlo Campaign
Storage Protocols
Http/WebDav and Dynafed R&D

3. The Belle II Collaboration
BELLE II is an international collaboration that aims at develop a large scientific program on the Physics of flavor through a new e+e- collider named SuperKEKB, in development at the Japanese laboratory KEK (Tsukuba).
SuperKEKB is an upgrade of KEKB, it plans to improve of a factor 50 the quantity of data to respect the previous facility (BELLE).

4. The Site

5. The Belle II Collaboration
~2500 members

6. Belle II Time Line

7. Belle II Computing Model
The BELLE II Computing model has to accomplish, the following main tasks, in a geographically distributed environment:
RAW data processing
Monte Carlo Production
Physics analysis
Data Storage and Data Archiving
On going activities
Resource Estimation
Define strategy for analysis and data distribution
Individuating technologies

8. Site Classification
The BELLE II Computing Sites are classified as follow:
Raw Data Center: Who stores the RAW Data and makes data processing and/or data reprocessing.
Regional Data Center: Large data center that stores mDST and participates at the Monte Carlo production
MC Production site: Data Center that produces and stores Monte Carlo simulations, that included:
Grid Site
Cloud Site
Computing Cluster Site

10. Belle II Network estimation Update
TYPE OF DATA
DATA MOVEMENT STRATEGY
RAW Data
Produced at KEK and sent to PNNL in the first fase and then to PNNL, Italy, Germany, Canada, Korea, India since 2021(2 full copies)
MDST
Produced at KEK and PNNL and sent to all sites up to 2020 then Produced at KEK, PNNL, Italy, Germany, Canada, Korea, India since 2021 and sent to all sites. (2 full copies)
MC Data
Produced by all sites and sent to all sites (2 full copies)
Data Reprocessing
Produced at PNNL and sent to all sites up to 2020 then Produced at PNNL, Italy, Germany, Canada, Korea, India since 2021 and sent to all sites(2 full copies)
MC Reprocessing
Produced by all Sites and sent to all sites (2 full copies)
MC pre-DataTaking
Produced by all Sites and sent to all sites(2 full copies)
Skimming
DataChallenge
Produced at KEK and PNNL then sent to all sites in 2018
Analysis
Chaotic

11. Traffic per regions N.B. USER ANALYSIS TRAFFIC NOT INCLUDED
<0,5Gbps
~7Gbps
~7Gbps
~4Gbps
<0.5Gbps
<0,5Gbps
<0,5Gbps
<0,5Gbps

12. Network Update at KEK Schedule

13. March 2016 - SINET5 migration
SINET Tokyo-London : 20G
SINET Tokyo-LA : 100G
SINET Tokyo-MANLAN : 10G
VRF
Osaka
Tokyo
Y. Kubota (NII)
perfsonar2.cc.kek.jp vs perfsonar-ps-01.desy.de
100 ms
Latency (ms)

15. PRODUCTION SYSTEM

16. Manage Interoperability with DIRAC

17. Belle II Distributed Computing system and
services in use

18. The infrastructure managed with Dirac
17 Countries Involved in the last MC production
~40 Site/Cluster with multiple access interfaces (i.e. Grid, Cloud)
27 Storage Elements for ~ 2PB of reserved Storage
In the 5th MC production ( )
~ 25K Jobs constantly running
~ 50K data transfer jobs for days
353 Scheduled Shifts

19. Monitoring Tools
DIRAC Web Portal
Ganglia
Happy Face
FTS Dashboard

20. The 5° MonteCarlo Production

22. File Read protocols
The Belle II software for simulation and analysis (basf2) is based on the standard ROOT I/O library.
Moreover Belle II analysis model is not totally in the “jobs go to data” paradigm, but, jobs may access “remote data”
So we have a set of possible backend protocols (supported by ROOT):
file – for the files downloaded on the WN (or access with NFS to some local SE)
xrootd – direct access to the files on SE, (local or remote)
http – direct access to the files on SE, (local or remote)

23. R&D Activities
Study and overcome issues relate the usage of HTTP/WEBDAV with the Belle II Software for simulation and analysis Basf2.
Configure and tune the HTTP/WEBDAV interface on Grid Storages.
Implement a first example of http federation for Belle II with dynafed software.
Test federation with dynafed component
Learn more about this topic.
The work is ongoing with the precious support of several sites of Belle II collaboration, and with the teams of DPM, dCache and SToRM.

24. Basf2 with Davix support
The Belle II software for simulation and analysis is based on the standard ROOT I/O library
Since the release build distributed to each sites via CVMFS, basf2 supports natively http/webdav thanks to the introduction of TDavixFile.h library. ……
filelistSIG=['
inputMdstList(filelistSIG)

25. The testbed infrastructure
STORGE DIRAC NAME
HOSTNAME
TYPE
DESY-DE
dcache-belle-webdav.desy.de
DCACHE
GRIDKA-SE
f e.gridka.de
NTU-SE
bgrid3.phys.ntu.edu.tw
SIGNET-SE
dcache.ijs.si
UVic-SE
charon01.westgrid.ca
Adelaide-SE
coepp-dpm-01.ersa.edu.au
DPM
CESNET-SE
dpm1.egee.cesnet.cz
CYFRONNET-SE
dpm.cyf-kr.edu.pl
Frascati-SE
atlasse.lnf.infn.it
HEPHY-SE
hephyse.oeaw.ac.at
Melbourne-SE
b2se.mel.coepp.org.au
Napoli-SE
belle-dpm-01.na.infn.it
CNAF-SE
ds cr.cnaf.infn.it
STORM
McGill-SE
gridftp02.clumeq.mcgill.ca
In January 2016 we started the investigation to verify http/webdav performances
At now 14 Storages of the 23 SRM endpoints registered in DIRAC have activated the webdav interface.
3 different storages technologies represented dCache, DPM, STORM

26. Dynamic Federations
The Dynamic Federations system allows to aggregate remote storage. The aggregation is performed on the fly, by distributing quick WebDAV queries towards the endpoints and scheduling, aggregating and caching the responses.
HTTP and WebDAV clients can browse the Dynamic Federation as if it were a unique partially cached name space, which is able to redirect them to the right host when they ask for a file replica.
Dynamic Federations is based on a open source project that allows to aggregate storages, that expose different protocols: HTTP/WebDAV, Cloud S3, moreover the system is generic enough to support others, if a suitable frontend exists.
Currently we have multiple example of aggregated storage on a server at desy
LHCB testbed (14 storages)
ATLAS testbed (~50 storages)

27. DYNAFED Server in Napoli https://dynafed01.na.infn.it/myfed/
Aggregate view of all the storages
PerSiteView that shows the file system of each storage

28. Performance Tests Software used: bsaf2 Skim used: D+rho_skim.py
6 different tests done analysing 1 single file (1000 events, size 10MB)
Local input (download via http)
Local input (download via xrootd)
Local input (download via lcg-cp)
Input via http streaming
Input via xrootd streaming
Dynafed access
5 trials for each test
Test are performed from Grid WorkerNodes in different sites vs All the storage of type DPM and dCache, while storage running STORM are still in evaluation.
WN Site X
WN Site Y
Storage-A
Storage-B
Storage-N

29. Analysis 1. File download with Http, Xrootd and Lcg-cp
Description
File download performances in function of the latency from the two different Sites.
Comments
http, xrootd performs quite similar and in all cases performs better than lcg-cp through srm interface
Average time from
CLIENT grid-wn0358.desy.de (DESY)
Average time from
CLIENT recas-bellewn06.na.infn.it (RECAS-NAPOLI)

30. Analysis 2. Http vs Xrootd Streaming
Description
File streaming performances in function of technologies and latency (McGILL case study)
Comments
In case of DPM Storages the two protocols performs quite similar in most cases.
DPM Storages

31. Conclusions
Belle II Computing model is design to address distributed resources with multiple access paradigms.
MC Campaigns have demonstrated that we are able to deal with resources heterogeneous thanks to the DIRAC framework. Next challenge will be manage even larger production that will start in November.
There are several activities on going: Resource Estimation, Framework Optimization, R&D in interaction with key working groups i.e. WLCG, LHCONE, HTTPTF

32. Thank you!