Performance measurement of transferring files on the federated SRB KEK Computing Research Center Yoshimi Iida

Outline The Belle experiment at KEK What is SRB? HEP Data Grid workshop Performance measurement Transfer measurement between sites in the Belle SRB federation Using WAN emulator Conclusions 29 April 2005 ISGC 2005

The Belle experiment at KEK The Belle experiment at KEK is one of several ongoing large experimental collaborations The accumulated data up to now is about 2PB including simulation data for analysis The large amount of data should be analyzed promptly to get statistically improved data for exploring new physics results Large numbers of files need to be managed consistently and shared easily among the 400 collaborators 29 April 2005 ISGC 2005

What is SRB? SRB provides a uniform interface for connecting to heterogeneous data resources and accessing data sets SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on logical attributes http://www.sdsc.edu/srb/ 29 April 2005 ISGC 2005

SRB federation SRB zone Federated MCAT consist of one or more SRB servers along with one MCAT Federated MCAT allow users to access resources and data sets across zones Server 1.1 MCAT 1 Server 1.2 MCAT 3 Server 3.1 MCAT 2 Server 2.1 Server 2.2 29 April 2005 ISGC 2005

Logical file system Single SRB system / - Zone1/ - container/ - home/ - srbUserA.domain/ - srbUserB.domain/ data.txt - styles/ - trash/ Federated SRB system / - Zone1/ - container/ - home/ - styles/ - trash/ - Zone2/ - Zone3/ : 29 April 2005 ISGC 2005

SRB data management Logical name space Data replication Mapping of each logical name to physical attributes UNIX like API and utilities for collections (directories) and data objects (files) Data replication replicate onto different resources 29 April 2005 ISGC 2005

User management Single Global User Name Space Uniquely identify by their usernames combined with their domain ‘iidayo@kek' Maintained in MCAT Single sign-on, access all resources No need for UNIX account at remote sites 29 April 2005 ISGC 2005

HEP Data Grid workshop Pre-workshop Workshop 1 - 3 December, 2004 Participants from 8 institutes at 7 countries SDSC (US) Australia National U., U. Melbourne (Australia) ASCC (Taiwan) KNU (Korea) IHEP (China) Krakow (Poland) KEK (Japan) Workshop 6 - 7 December, 2004 http://www-conf.kek.jp/hepdg/ 29 April 2005 ISGC 2005

SRB federated MCAT Zone IHEP (China) Zone ASCC (Taiwan) Zone KNU (Korea) 10Mbps Internet 100Mbps 100Mbps Zone ANU (Australia) Zone Krakow (Poland) 622Mbps 100Mbps Zone KEK (Japan) FW FW: Firewall M: MCAT enabled SRB server S: SRB server C: SRB client C S M C 29 April 2005 ISGC 2005

Belle SRB federation Install SRB server, a storage resource and MCAT at each site SRB version 3.2.1p Open the KEK firewall from the fixed IP address of the other sites After the installation, the functionality of the federation was tested and confirmed to be working 29 April 2005 ISGC 2005

Transfer measurement from KEK Copy files from KEK to the other sites 10 files from 0.5 to 100MB in size ‘Scp’ (SRB copy) use parallel I/O by default Configure to allocate one thread per 2MB of the file size up to a maximum of 16 threads (i.e. 32MB) 29 April 2005 ISGC 2005

Transfer rates from KEK Measured by ‘Scp’ 29 April 2005 ISGC 2005

Transfer performance between sites in the Belle SRB federation Source Destination KNU ASCC ANU Krakow IHEP KEK 9.65 5.89 3.06 2.69 0.59 - 5.68 4.45 4.47 0.61 7.54 2.84 5.09 0.64 4.79 2.53 3.44 0.58 1.21 1.64 4.19 0.60 0.70 0.39 0.44 0.25 Results of 100MB data file transfer in MB/s 29 April 2005 ISGC 2005

WAN emulation NIST Net Network emulation package Machine Specification NIST Net allows a single Linux PC set up as a router to emulate a wide variety of network conditions http://www-x.antd.nist.gov/nistnet/ Machine Specification CPU: Pentium4 3.2GHz Memory: 1GB OS: Red Hat Linux 8 NIC: GbE×2 Internet KEK network GbE GbE SRB server SRB server 100Mbps 100Mbps router NIST Net 29 April 2005 ISGC 2005

Transfer measurement with various file size Copy files on WAN emulation 8 files from 10 to 500MB in size Measure as a function of the RTT from 0 to 300ms ‘Scp’ use parallel I/O by default Configure to allocate one thread per 2MB of the file size up to a maximum of 16 threads Measure the bandwidth with iperf set the parallel client threads to 16 29 April 2005 ISGC 2005

Transfer performance with various file size iperf -P 16 Emulated by NIST Net 29 April 2005 ISGC 2005

Transfer performance on WAN (preliminary) BW 100Mbps WAN Emulation (NIST Net) KNU ASCC ANU Krakow IHEP Measured by Scp 29 April 2005 ISGC 2005

Bandwidth The nominal bandwidth between KEK and some sites are same 100Mbps as WAN emulation The actual bandwidth measured by iperf between KEK and other site is lower than WAN emulation e.g. 60Mbps for ASCC with iperf Further investigation is necessary 29 April 2005 ISGC 2005

Conclusions A federation of SRB servers was demonstrated successfully among ANU, KNU, IHEP, ASCC, Krakow and KEK The current results are preliminary and were obtained in the limited time available in the pre-workshop session The functionality of data sharing for analysis was proved to be stable and sufficient to be used by Belle collaboration 29 April 2005 ISGC 2005

Acknowledgements SDSC (San Diego Supercomputer Center) M. Wan ANU (Australia National University) S. J. McMahon University of Melbourne G. Moloney ASCC (Academia Sinica Computing Centre) H. Lin KNU (Kyungpook National University) K. Kwon IHEP (Institute of High Energy Physics) G. Chen Krakow (Institute of Nuclear Physics, Krakow) P. Lason and H. Palka Fujitsu S. Honma and T. Nakajima KEK (High Energy Accelerator Research Organization) Y. Karita, T. Sasaki, S. Y. Suzuki, S. Yashiro and Y. Watase 29 April 2005 ISGC 2005

Backups

‘Sls’ display data objects (file) or collections (directory) ‘SgetD’ display information about SRB data objects ‘SgetR’ display information about SRB resource This is an example. In SRB space, we can get the list of the SRB data object using Sls command. And SgetD command display information about SRB data object. So this is the information of testdata03.txt. We can see the two information with different SRB resources for one object. This is data replication. And user can get the information about the SRB resource using SgetR command. 29 April 2005 ISGC 2005

By the federated MCAT, the SRB zones maintained by each sites can see as a single tree structure directory. 29 April 2005 ISGC 2005