Status of BESIII Computing Yan Xiaofei IHEP-CC

Outline 1 Local Cluster and Distributing Computing 2 Storage Status 3 Network Status 4 Others

HTCondor Cluster Upgrade new HTCondor version Problems (after upgrade) “hung” problem, leading scheduler process to be restart, disappeared. Upgrade new HTCondor version From 8.4.3 to 8.4.11 Bugs fixed and better performance provided Completed in summer maintenance Problems (after upgrade) “/ihepbatch/bes” with heavy pressure, leading jobs “hold” Some users run a lot of analysis jobs in the directory Cause low performance Bugs about maximum jobs in the queue found in the new version Cause the scheduler unstable Limit 10,000 jobs per user

HTCondor Cluster New feature ---- submit bulk jobs Submit bulk jobs quickly Less submitting time Decrease submitting failure rate Commands for BES users “boss.condor” for batch jobs Job option files should exist $ boss.condor -n 300 jobopt%{ProcId}.txt

Jobs statistics (Jun,2017 - Sep,2017) BES jobs occupied most of CPU time Experiment Job count Walltime(hr) User count BES 6,334,436 14,164,509.61 305 JUNO 2,754,221 1,729,473.14 51 DYW 861,264 1,136,054.19 39 HXMT 614,257 793,408.08 11 LHAASO 624,279 699,813.88 41 CEPC 1,208,345 592,382.61 24 ATLAS 406,818 202,126.59 23 CMS 1,096,388 142,736.3 15 YBJ 323,144 140,068.56 36 OTHER 311,202 56,856.52 8

SLURM Cluster Resources Jobs（2017.6~2017.8） Research 1 master node 1 accounting & monitoring node 16 login nodes 131 work nodes: 2,752 CPU cores, 8 GPU cards Jobs（2017.6~2017.8） # Jobs : 1,834 CPU hours : 2,001,800 Research hep_job tools : unified User Interface with the HTCondor Cluster Malva : a resource monitoring system Jasmine : a multi-purpose job test suite

BESIII distributed computing(1)

BESIII distributed computing(2) In the last three months, about 67.5K BESIII jobs have been done in the platform 11 sites joined the production The central DIRAC server has been successfully upgraded from v6r15 to v6r18 during summer maintenance Accordingly the distributed computing client in AFS has been upgraded to the latest version All the changes are transparent to end users

BESIII distributed computing(3) The distributed computing client is expected to migrate from AFS to CVMFS soon More convenient for users outside IHEP, better performance than AFS The BOSS software repository used for sites has been successfully migrated to newly built CVMFS infrastructure which is with latest version and new hardware The CNAF CVMFS Stratum1 server in Italy has replicated IHEP Stratum0 server This help facilitate access to BOSS for users from Europe collaborations OSG S1 CNAF S1 IHEP S1 IHEP S0 CERN S0 Proxy Europe Asia America

Outline 1 Local Cluster and Distributing Computing 2 Storage Status 3 Network Status 4 Others

user/group data, raw data Storage Usage Guide Age(year) 1 2 5 5,0 Usage public data user/group data, raw data user data temporary user data BESIII dedicated Yes No Quota 50GB 5GB,50k files 500GB Backup

Disk Usage During Last 3 months BESIII computing is the major source of disk throughput in IHEP Cluster, >90% R/W throughput, peak read 15GB/s, peak write 6 GB/s /besfs increased 200 TB disk usage after summer maintenance Summer Maintenance /besfs write

Storage Evaluation The density of disk increase quickly in last few years 8 TB/10TB disk is the mainstream product Saves budget for better storage hardware The Storage capacity managed by single server increase Evolutions of new hardware and storage architecture SSD disk array (better IOPS ) Storage Area Network (Flexibility and Sharing of high-end SSD disk array) Bonding of 10 Gbit Ethernet (Upgrade the Network of each server from 10Gb to 20Gb or more) Multipath of disk Fiber channel (Upgrade the theoretical channel throughput from 16Gb to 32Gb) 1. Switch Single path to multiple paths . 2. aims: a. more stable without single failure point , b. double the bandwidth from 10 Gb to 20Gb which increase the I/O performance

Outline 1 Local Cluster and Distributing Computing 2 Storage Status 3 Network Status 4 Others

Network-WAN IHEP-USA(Internet2) IHEP-EUR(GEANT) IHEP-Asia IHEP-CSTNet-CERNET-USA Bandwidth: 10Gbps IHEP-EUR(GEANT) IHEP-CSTNet-CERNET-LONDON-Euro IHEP-Asia IHEP-CSTNet-HONGKONG-Asia Bandwidth: 2.5Gbps IHEP-*.edu.cn IHEP-CSTNet-CERNET-University Bandwidth：10Gbps 国际国内链路； 1、（加入lhc1 qfz），进展情况； 2、中欧链路问题；

New Network Architecture Separate wireless network from cable network Add an interconnect switch Collect separated networks Dual Firewall supported Both in data center network and campus network Add two DMZs in campus network OA services Public cloud services IPv6 supported Both in campus network and data center network The Grid data transfer is currently 8770 更换问题；

Outline 1 Local Cluster and Distributing Computing 2 Storage Status 3 Network Status 4 Others

New Monitoring at IHEP Motivation Goal Various cluster monitoring tools are independent Integration of the multiple monitoring data can provide more information Improve the availability of computing cluster Goal Correlation among the monitoring sources Analyze various sources monitoring data Unified display system provides health status from multiple levels Show the trend of error and abnormity. 改字体

Remote Computing Sites Service Maintenance services for remote computing center Ustc, Buaa, and Clas sites are running well USTC site increased 448 cores and 1.1P lustre storage Site Name CPU Cores Storage(TB) Ustc 1088 1843 buaa 160 81 clas 224 37 We setup central morntorning system for remote site,

Public services SSO IHEP APP IHEPBox Vidyo Total users：7036 Bes3：642 (IHEP：227, others：415) IHEP APP Support phone book, can call directly News、Academic events、Media… Support Personal agenda later IHEPBox Updated to the latest version Total users: 1000+ Space usage: 3T/154T Vidyo Users:692 Conferences:1059

Others Password self reset: Helpdesk http://202.38.128.100:86/ccapply/userfindpasswd.action Helpdesk http://helpdesk.ihep.ac.cn Email: helpdesk@ihep.ac.cn Tel: 88236855

Summary Computing platform keeps running well. New feature added for users. New storage architecture is under evaluation. Unified monitoring system developed. New Network architecture is ready.

Thank you! Question?