SRB at KEK Yoshimi Iida, Kohki Ishikawa KEK – CC-IN2P3 Meeting on Grids at Lyon September 11-13, 2006

SRB at KEK 2 The Belle experiment at KEK The Belle experiment at KEK is one of several ongoing large experimental collaborations The accumulated data up to now is about 1PB including simulation data for analysis The large amount of data should be analyzed promptly to get statistically improved data for exploring new physics results The computing facility at KEK is so limited that a large portion of the simulation jobs are shared by the collaboration institutes world wide Some of the collaborators requested us to support a GRID environment for data distribution and efficient analysis

SRB at KEK 3 Why SRB? Easy to install and start over Even works with the password authentication Light clients including command line, GUI and the web Server side is also light Only one SQL server is necessary except SRB servers Federation of SRB zones is available Independent operation, but still information is federated

SRB at KEK 4 History 2003 Jun. 06 Start operation of the test system 1 with PostgreSQL HPSS interface 2003 Oct. 22 Test system 2 with DB2 PostgreSQL looks better than DB Apr. 02 Introduction in BELLE Computer system Interface to SONY Peta Site (HSM) via NFS Test with Melbourne 2004 Dec. 06 HEP Data Grid workshop 2004 Together with Michel Wan, SDSC, SRB federation has been established among AU, KR, TW, CN, PL and JP

SRB at KEK 5 The location of the federated zones Krakow, PL SDSC, US KEK, JP KNU, KR IHEP, CN ASCC, TW ANU, AU The latency was measured in December 2004 Destination (from KEK) RTT (msec) Nominal BW (Mbps) IHEP50210 Krakow ANU ASCC33100 KNU

SRB at KEK 6 Transfer rates from KEK Measured by Scp commands that use parallel I/O by default Files ranging from 0.5MB to 100MB were used in the tests as the number of threads that SRB chooses to use is determined by the file size The parallel transfer is effective the improvement of the transfer performance

SRB at KEK 7 Transfer performance on WAN KNU ASCC ANU Krakow IHEP WAN Emulation (NIST Net) The results of WAN emulation are typical data transfer performance for the SRB federation There is a difference between WAN emulation and experimental results The packet loss is not generated in WAN Emulation The latency and quality of the network is the key to obtain better performance and that higher network bandwidth alone is not sufficient.

SRB at KEK 8 History 2003 Jun. 06 Start operation of the test system 1 with PostgreSQL HPSS interface 2003 Oct. 22 Test system 2 with DB2 PostgreSQL looks better than DB Apr. 02 SRB in BELLE Computer system Interface to SONY Peta Site via NFS Test with Melbourne 2004 Dec. 06 HEP Data Grid workshop 2004 Together with Michel Wan, SDSC, SRB federation has been established among AU, KR, TW, CN, PL and JP 2006 Apr. 18 Replace with the new system 2 kinds of interface to HPSS 2006 Jun. 01 Federated with IN2P3

SRB at KEK 9 SRB system srbcert SRB Master PostgreSQL UI SRB Client SRB Server srbcert SRB Server srbcert SRB Server HPSS External Disk Sub-sys HPSS APIHPSS VFS rac01rls10rls09rsr01 “rls10-hpss-vfs”“rls09-hpss-hsi” “rsr01-ufs” GRID NW

SRB at KEK 10 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data SRB Integration with HPSS SRB supports HPSS resource type we can compile HPSS- enabled SRB with HPSS before v4.5, relatively easily But at KEK, A little of change of SRB source code was needed Authentication with Kerberos Non-DCE Linux

SRB at KEK 11 Challenge When HPSS migrates v4.5 to v6.2 HPSS changes Meta-data management & transaction monitoring component Low-level API Meta-data structure definition So, We doubt the source level compatibility of HPSS driver of SRB with HPSS v6.2

SRB at KEK 12 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data Work Around with HPSS VFS Access Work Around for Integration key is “hpssfsd” provide POSIX I/O function into HPSS name space through VFS layer of Linux wrapper of HIS can be exported by NFS we try 2 pattern 1. SRB UNIX Res. Type 2. SRB NFS Res. Type These method are running in early phase, because they does not require any source code modification

SRB at KEK 13 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data HPSS-enabled SRB with HPSS v6.2 Now running with some restriction with v6.2 Modify Source Code After v6.2, HPSS change its low-level API related to some meta-data structure. So, SRB must change its code related to HPSS Some Restriction necessity to renew Kerberos ticket

SRB at KEK 14 Summary of SRB - HPSS HPSS-enabled SRB with HPSS v6.2 is require some modification of source code, but it can be built at first, to use HPSS v6.2 with SRB, POSIX I/O function is useful. performance comparison is a next test subject

SRB at KEK 15 Specification HPSS External Disk Sub-sys rls10rls09rsr01rac01 CPURAM sizeOS (kernel)SRB resource rsr01Intel Xeon MP 3.0GHz x 4 4GBRHEL 3 AS ( ) Storage Subsystem (1TB) is connected with Fiber-Channel rls09Intel Xeon 3.0GHz x 2 1GBRHEL WS 4, ( ) HPSS-enabled SRB Server uses HPSS Client API Library, SRB source code includes HPSS API call. rls10Intel Xeon 3.0GHz x 2 1GBRHEL WS 4, ( ) HPSS VFS service, which offers POSIX VFS gateway toward HPSS Hierarchical Storage Area. rac01Intel Xeon 3.0GHz x 2 1GBRHEL ES 3, ( ) Client GRID LAN

SRB at KEK 16 Transfer performance (preliminary) Data file : 1GB RTT : 0.9ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 GRID LAN ImportExport

SRB at KEK 17 Transfer performance (preliminary) Data file : 1GB RTT : 0.9ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 GRID LAN ImportExport

SRB at KEK 18 Transfer performance (preliminary) Data file : 1GB RTT : 0.08ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs GRID LAN rac01rls10rls09rsr01 ImportExport

SRB at KEK 19 Data file : 1GB RTT : 262ms BW : 17Mbps(iperf) LyonFS3 rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 Internet Transfer performance (preliminary) KEKIN2P3 ImportExport

SRB at KEK 20 Summary We replaced the new SRB system Need to more test for HPSS We could establish federation with IN2P3 and KEK In order to improve the data transfer performance we should investigate the network between France and Japan Federation with RAL

SRB at KEK 21 Back up

SRB at KEK 22 Traceroute from KEK to IN2P3 traceroute to ccbbsn09.in2p3.fr ( ), 30 hops max, 38 byte packets ( ) ms ms ms 2 ns1kb.kek.jp ( ) ms ms ms 3 keksw1-ns.kek.jp ( ) ms ms ms 4 kekgw.kek.jp ( ) ms ms ms 5 kek-S1-2-2.sinet.ad.jp ( ) ms ms ms 6 tokyo-core1-P10-0.sinet.ad.jp ( ) ms ms ms 7 nii-S1-P4-0.sinet.ad.jp ( ) ms ms ms 8 nii-IX1-P2-0.sinet.ad.jp ( ) ms ms ms 9 NYC-gate1-P3-0.sinet.ad.jp ( ) ms ms ms 10 sinet.ny1.ny.geant.net ( ) ms ms ms 11 ny.uk1.uk.geant.net ( ) ms ms ms 12 so rt1.par.fr.geant2.net ( ) ms ms ms 13 renater-gw.rt1.par.fr.geant2.net ( ) ms ms ms 14 lyon-pos6-0.cssi.renater.fr ( ) ms ms ms 15 in2p3-lyon.cssi.renater.fr ( ) ms ms ms 16 lyon-core.in2p3.fr ( ) ms ms ms 17 ccbbsn09.in2p3.fr ( ) ms ms ms

SRB at KEK 23 Slide by

SRB at KEK 24 IN2P3CC-ICEPP iperf Test Kernel 2.6 TCP Congestion Control TCP Reno BIC TCP +PSPacer Precision Software Pacer From AIST GridTCP project Slide by