1. iRODS performance test and SRB system at KEK Yoshimi Iida @ KEK Building data grids with iRODS 27 May 2008

2. 27/05/2008Building data grids with iRODS2 Outline  Performance measurement Transfer test between CC-IN2P3 and KEK Scaling test for ICAT Concurrent test for ICAT Compare with iRODS and SRB  SRB/iRODS system at KEK

3. 27/05/2008Building data grids with iRODS3 Transfer between CC-IN2P3 and KEK  1GB data transfer between CC-IN2P3 and KEK Compare with iRODS and bbcp  The network route go through USA The RTT is about 285ms KEK, JP CC-IN2P3, FR NY, USA

4. 27/05/2008Building data grids with iRODS4 System used  iRODS system at CC-IN2P3 ICAT enabled iRODS server on Solaris 10  Thumpers Sun x4500 (AMD processors)  Oracle 10g is on a cluster of dedicated machines  Linux file resource from local disk system  iRODS 0.9  iRODS system at KEK iRODS server on RHEL3  Intel Xeon 3.0GHz ×4  Linux file resource from local disk system  iRODS 0.9

5. 27/05/2008Building data grids with iRODS5 From KEK to CC-IN2P3  1GB data transfer during 24 hours window size 4MB number of parallel streams 16  bbcp often fail to connect

6. 27/05/2008Building data grids with iRODS6 From CC-IN2P3 to KEK  1GB data transfer during 12 hours window size 4MB number of parallel streams 16  iput is better than bbcp

7. 27/05/2008Building data grids with iRODS7 Scaling test  Data Using the same directory at client machine 1000 files of 1000 bytes each  Measurement Ingesting directory and listing the collection Performance measured for every directory operation  1 collection and 1000 files

8. 27/05/2008Building data grids with iRODS8 System used  ICAT based on Oracle system at CC-IN2P3 ICAT enabled iRODS server on Solaris 10  Thumpers Sun x4500 (AMD processors)  Oracle 10g is on a cluster of dedicated machines iRODS resource and client on SL4  Dual AMD Opteron Processor 848  Linux file resource from local disk system  ICAT based on PostgreSQL system at KEK ICAT enabled iRODS server on RHEL3  Dual Intel Xeon 2.8GHz  PostgreSQL 8.2.5 is running on the same machine iRODS resource and client on RHEL3  Dual Intel Xeon 2.8GHz  Linux file resource from local disk system

9. 27/05/2008Building data grids with iRODS9 Ingesting up to 1 million files Running the other process

10. 27/05/2008Building data grids with iRODS10 Nested collection test  Data Registering same directory at client machine 100 files of 100 bytes each  Measurement Ingesting directory and listing the collection Making nested collection every 10 collection Performance measured for every directory operation  1 collection and 100 files

11. 27/05/2008Building data grids with iRODS11 Nested collection test nest-121 : OCI_ERRORcoll-1220: USER_PATH_EXCEEDS_MAX

12. 27/05/2008Building data grids with iRODS12 Concurrent test  Data Ingesting same directory at client machine 1000 files of 1000 bytes each  Measurement Runing multi process at the same time  read operation – ils and iget  write operation – iput and ireg  mixed operation – iput, ireg, ils and iget Performance measured for every directory operation  DB setting Set the number of maximum connections to 200

13. 27/05/2008Building data grids with iRODS13 Concurrent test for reading metadata Error: connectToRhost failed 10 clients200 clients 100 clients300 clients ×

14. 27/05/2008Building data grids with iRODS14 Concurrent test for writing metadata  iput – Oracle ICAT Because of limit of inodes, we cannot put any more files 10 clients100 clients200 clients

15. 27/05/2008Building data grids with iRODS15 Concurrent tests for mix - PostgreSQL 10 clients100 clients200 clients

16. 27/05/2008Building data grids with iRODS16 Concurrent test for mix - Oracle  iput – Oracle ICAT Because of limit of inodes, we cannot put any more files 10 clients100 clients200 clients

17. 27/05/2008Building data grids with iRODS17 Compare with iRODS and SRB  For the scaling test Data  Using the same directory at client machine  1000 files of 1000 bytes each Measurement  Ingesting directory and listing the collection  Performance measured for every directory operation  For the nested collection test Data  Registering same directory at client machine  100 files of 100 bytes each Measurement  Ingesting directory and listing the collection  Making nested collection every 10 collection  Performance measured for every directory operation

18. 27/05/2008Building data grids with iRODS18 System used  iRODS system at KEK ICAT enabled iRODS server on RHEL3  Dual Intel Xeon 2.8GHz  PostgreSQL 8.2.5 is running on the same machine  iRODS 1.0 iRODS resource and client on RHEL3  Dual Intel Xeon 2.8GHz  Linux file resource from local disk system  SRB system at KEK MCAT enabled SRB server on RHEL3  Dual Intel Xeon 2.8GHz  PostgreSQL 8.2.5 is running on the same machine  SRB 3.5.0 SRB resource and client on RHEL3  Dual Intel Xeon 2.8GHz  Linux file resource from local disk system

19. 27/05/2008Building data grids with iRODS19 Scaling test - iRODS and SRB

20. 27/05/2008Building data grids with iRODS20 nested collection – iRODS and SRB coll-1220: USER_PATH_EXCEEDS_MAXnest-47: Error Problem running command

21. 27/05/2008Building data grids with iRODS21 SRB system for Belle at KEK  Both protocols are authorized by GSI  Setting up on GridFTP server grid-mapfile for SRB user SRB configuration file for server and resource  Register LCG user DN on MCAT MelbourneMelbourne NCUNCUASGCASGC KUKU NagoyaNagoya CYFRONETCYFRONET KEK-2KEK-2 KEK-1KEK-1 SINETSINET KEK-LANKEK-LAN GEANT2GEANT2 APANAPAN KEK-DMZKEK-DMZ SRB-DSISRB-DSI KEK FW HSMHSM MCATMCAT Pluggable Extension SRB client SRB server 3.5PB GridFTP SRB NFS Enhanced GridFTP service Belle Net dedicated inside Belle Net dedicated inside LCG user Computing Farm Still not integration with Grid LSF LCG user Belle analysis user

22. 27/05/2008Building data grids with iRODS22 Plan for iRODS system at KEK  Data transfer for J-PARC project Generate huge amount of imaging data at Tokai  About 1PB data in a year in total Store the data at Tokai storage once, then copy to KEK and distribute for collaborators  Storage at Tokai are recycled Bandwidth between 2sites will be 10Gbps KEK Tokai 60km

23. 27/05/2008Building data grids with iRODS23 Acknowledgements  Special thanks to Jean-Yves Nief from CC- IN2P3 for his help to setup iRODS system and supporting at CC-IN2P3  Thanks to Adil Hasan from RAL for his help to these tests

24. 27/05/2008Building data grids with iRODS24 Back up

25. 27/05/2008Building data grids with iRODS25 Bandwidth Lyon-KEK  iperf with some options; -w 4M : TCP window size [Bytes] -P 16 : the number of parallel threads -i 5 : periodic bandwidth reports [sec]

26. 27/05/2008Building data grids with iRODS26 Summery of iRODS Performance  Transfer from KEK to CC-IN2P3 is not stable but iput is better than simple transfer software (bbcp)  iRODS can manage 1 million files stability In case of very nested collection, PostgreSQL ICAT takes long time to register data Better performance than SRB  Oracle ICAT can handle more than 300 clients at the same time, but it takes time according to increase the number of clients