Progress of Work on SE and DMS YAN Tian April. 16, 2014

Outline Deployment of StoRM SE Data transfer between SEs Squid+CVMFS new version test Status of new sites (INFN, BUAA, SDU) Status of DMS work plan

SE installation & test USTC SE is installed to be d-Cache type for emergency data transferring (Feb. 24). A stable test instance of StoRM SE is set up at IHEP (Mar. 17) – Presure test: About 20,000 test files (137GiB), the SE server work fine, no longer crash. – https webDAV component is added and tested. Users can view file list in webbroser and download data by wget with certificate. WHU SE (36TB) is installed with StoRM (Apr. 02); Pressure test: – 28.9 TB data transferred to USTC SE, downloaded by wget – 4.4TB data transferred to WHU SE – download 3.9TB data from USTC SE to WHU SE

Current Status of SEs #SE NameCapacityTypeSpeed to IHEP webDAVLaunch Date 1IHEPD-USER126 TBdCachein IHEP2012 2JINR-USER7.3 TBdCache~ 25 MB/stesting2012 3USTC-USER24 TBdCache~ 37 MB/sanonymous WHU-USER39 GBStoRM~ 21 MB/shttps YANT-USER167 GBStoRM (test)in IHEP2014.3

24.5 TB XYZ DST Data Transferred from IHEP SE to USTC SE Motivation: USTC physicist desire; testing SE; testing transfer system. Time: From Mar. 9 th to 25 th, 2014 Average speed: 37.0 MB/s (peak 60 MB/s at night), about 3.2 TB per day. USTC physicists download data from their SE by wget at about 75 MB/s. BatchData TypeFilesData SizeAverage SpeedSuccess Rate 1xyz_423012, TB36.7 MB/s99.31% 2xyz_426010, TB38.7 MB/s99.19% 3xyz_43605, TB37.4 MB/s98.80% 4xyz_4260scan5, TB36.9 MB/s99.65% 5xyz_4360scan TB30.4 MB/s98.92% Total35, TB37.0 MB/s99.23%

4.4 TB randomtrg Data Transferred from IHEP SE to USTC,JINR,WHU SE Motivation: for simulation + reconstruction test Target Data: round 06 random trigger data, 12,468 files with 4.04 TB size. Time: From Mar. 13 th to Apr. 5 th, 2014, intermittently. The speed reduction at USTC since half of the files are small.idx files. The speed of JINR and WHU is much higher than previous test result (download from JINR at 220 KB/s and WHU iperf test at 2MB/s). BatchDestination SEFilesData SizeAverage SpeedSuccess Rate 1USTC-USER12, TB23.4 MB/s99.37% 2JINR-USER12, TB24.9 MB/s99.76% 3WHU-USER12, TB21.4 MB/s99.76% Total37, TB

24.5 TB XYZ DST Data Transferred from IHEP SE to USTC SE Motivation: USTC physicist desire; testing SE; testing transfer system. Time: From Mar. 9 th to 25 th, 2014 Average speed: 37.0 MB/s (peak 60 MB/s at night), about 3.2 TB per day. USTC physicists download data from their SE by wget at about 75 MB/s. BatchData TypeFilesData SizeAverage SpeedSuccess Rate 1xyz_423012, TB36.7 MB/s99.31% 2xyz_426010, TB38.7 MB/s99.19% 3xyz_43605, TB37.4 MB/s98.80% 4xyz_4260scan5, TB36.9 MB/s99.65% 5xyz_4360scan TB30.4 MB/s98.92% Total35, TB37.0 MB/s99.23%

4.4 TB randomtrg Data Transferred from IHEP SE to USTC,JINR,WHU SE Motivation: for simulation + reconstruction test Target Data: round 06 random trigger data, 12,468 files with 4.04 TB size. Time: From Mar. 13 th to Apr. 5 th, 2014, intermittently. The speed reduction at USTC since half of the files are small.idx files. The speed of JINR and WHU is much higher than previous test result (download from JINR at 220 KB/s and WHU iperf test at 2MB/s). BatchDestination SEFilesData SizeAverage SpeedSuccess Rate 1USTC-USER12, TB23.4 MB/s99.37% 2JINR-USER12, TB24.9 MB/s99.76% 3WHU-USER12, TB21.4 MB/s99.76% Total37, TB

SE Transfer Speed: IHEP to USTC

SE Transfer Speed: IHEP to JINR/WHU

SE Transfer Quality: IHEP to JINR/WHU

Problems on Data Transfer permission control (randomtrg r06 should change to 777 before transfer) perserve timestamp info. (USTC physicist desire) transfer stalled dataset compatibility multi-request transfer UI refinement

webDAV HTTP Interface of USTC SE

webDAV HTTP Interface of WHU SE

HTTP Access to SE Motivation: site without SE can use http client to access dst data or randomtrg data WebDAV module: support http access to SE. wget download test : – 3.87 TB data from USTC webDAV interface to WHU. – typical speed: 40~60 MB/s – experienced bandwidth limitation, even 0 for some time, wget just waiting and continue working – final average speed 12.9 MB/s, time 3d 7h 23m 27s. Problems: need md5sum check

New Version of squid and CVMFS squid is upgraded from to CVMFS is upgraded from to Deployment: – BUAA (SL5.8): squid CVMFS – WHU (SL6.4): squid CVMFS Install guide in Twiki is updated.

Status of Sites #Site NameCPU CoresLRMSSEStatus 1IHEP-PBS.cn96PBSdCache 126 TBRunning 2GUCAS.cn152PBSshare IHEPRunning 3USTC.cn ~768PBS + condordCache 24 TBRunning 4PKU.cn88PBSRunning 5JINR.ru40~128gLitedCache 7.3 TBRunning 6UMN.us768PBS/condorRunning 7WHU.cn ~300PBSStoRM 39 TBWN’s CVMFS is deploying 8INFN-Torino.it~200gLiteStart Running at Apr. 2 9SDU.cn~102PBSSimple sh job is working 10BUAA.cn~256commercial?Squid+CVMFS installed Total2032~ TB

Status of DMS Work Plan Sub taskStatus Clean up DFC and SEDone by YAN Tian Upgrade gangaBOSSDone by ZHAO Xianghu Dataset and dst upload toolkit and APIDone by ZHANG Gang SE transfer toolkit test and refineDone by YAN Tian and LIN Tao WMS-log and randomtrg upload toolDone by ZHAO Xianghu New version DFC testHandling by ZHAO Xianghu Upgrade DIRAC to v6r10p20 and testTo do Tag, intergrated upgrade & test, releaseTo do

Thanks Thank you for your attention! Q & A