1. A data Grid test-bed environment in Gigabit WAN with HPSS in Japan A.Manabe, K.Ishikawa +, Y.Itoh +, S.Kawabata, T.Mashimo*, H.Matsumoto*, Y.Morita, H.Sakamoto*, T.Sasaki, H.Sato, J.Tanaka*, I.Ueda*, Y.Watase S.Yamamoto +, S.Yashiro KEK *ICEPP +IBM Japan

2. 2003/3/27CHEP032 Outline zICEPP Computer facility and its roll in ATLAS Japan. zICEPP-KEK Nordu-Grid Testbed. zGridFTP and GSI enabled pftp (GSI-pftp) zData transfer performance.

3. 2003/3/27CHEP033 ICEPP( International Center for Elementary Particle Physics ) zICEPP located in Univ.of Tokyo yCentral computer facility of ATLAS Japan. yICEPP started to install PC farm from 2002 and joined ATLAS DC1 in the last summer. zICEPP PC farm yComputing Nodes x39 nodes PenIII 1.4G x 2cpu x108 nodes Xeon 2.8G x 2cpu will be installed in this month. y2.5TB NFS server x2.2GHz Xeon yLTO Library

5. 2003/3/27CHEP035 ICEPP-KEK Grid Testbed Object zTo configure R&D test bed for ATLAS Japan regional center hierarchy. yICEPP (Tokyo) ATLAS Tier 1 regional center yKEK (Tsukuba) Supposed to be an ATLAS Tier2 local center. zShort term requirement yArchiving ~3 TB data produced at ATLAS DC1 to mass storage and open it to ATLAS members in KEK.

6. 2003/3/27CHEP036 ICEPP-KEK Grid Test Bed Hardware zHardwares yICEPP xComputing Elements 4 nodes with 8cpus. yKEK xComputing Elements 50 nodes with 100 cpus. xHPSS servers 2 disk movers. HPSS system (shared with general users in KEK.)

7. 2003/3/27CHEP037 Test bed – KEK side zFujitsu TS225 50 nodes yPenIII 1GHz x 2CPU z120TB HPSS storage zUsing for basic storage service to general users in KEK z5 Tape movers with 20 3590 drives (share use) z2 disk movers with SSA Raid disks (dedicated use for this test bed)

8. 2003/3/27CHEP038 ICEPP-KEK Grid Testbed zNetwork y1 GbE connection over Super-SINET between ICEPP PC farm, KEK PC farm and HPSS servers in single subnet. yRTT ~ 4ms / quality is quite good.

9. 2003/3/27CHEP039 Japan HEP network backbone on “Super-SINET” Project of Advanced Science and Technologies Super SINET Super SINET : Research 10Gbps Backbone GbE / 10GbE Bridges for peer-connection Connecting major HEP Univ. and institues in Japan. Operation of Optical Cross Connect (OXC) for fiber / wavelength switching Operational from 4th January, 2002 to end of March, 2005 Will be merged in Photonic SINET after April, 2005

10. ICEPP KEK ~60km

11. 2003/3/27CHEP0311 1Gbps 100Mbps ICEPPKEK 100 CPUs 6CPUs HPSS 120TB GRID testbed environment with HPSS through GbE-WAN NorduGrid - grid-manager - gridftp-server Globus-mds Globus-replica PBS server NorduGrid - grid-manager - gridftp-server Globus-mds PBS server PBS clients HPSS servers ~ 60km 0.2TB SECE SE User PCs

12. 2003/3/27CHEP0312 ICEPP-KEK Grid Testbed software zSoftware yGlobus 2.2.2 yNordugrid 0.3.12 + PBS 2.3.16 yHPSS 4.3+ GSI enabled pftp (GSI-pftp)

13. 2003/3/27CHEP0313 NorduGrid zNorduGrid (The Nordic Test bed for Wide Area Computing and Data Handling) yhttp://www.nordugrid.org y“The NorduGrid architecture and tools” presented by A.Waananen et al. @ CHEP03

14. 2003/3/27CHEP0314 Why NorduGrid zNatural Application of GLOBUS toolkit for PBS. yPBS clients do NOT need Globus/NorduGrid installation. èWe installed NG/Globus to just 3 nodes. (ICEPP CE,KEK CE, KEK HPSS SE) but can use more than 60nodes. zSimple, but sufficient functionality. xActually used at ATLAS DC in Nordic states. èGood start for basic regional center functionality test.

15. 2003/3/27CHEP0315 HPSS as NorduGrid Storage Element zHPSS does not speak ‘Globus’. We need something yGridFTP for HPSS xIn design phase at Argonne Lab. xSome are also being developed? (SDSC?) yGSI enabled pftp (GSI-pftp) xdeveloped at LBL. zGSI-pftp is not a GridFTP. But….

16. 2003/3/27CHEP0316 GSI-pftp as NorduGrid SE zBoth Gridftp and GSI-pftp are a kind of ftp, only extended protocols are not common. GridFTPGSI-pftp SPAS,SPOR,ETET, ESTO, SBUF, DCAU PBSZ,PCLO,POPN, PPOR,PROT,PRTR, PSTO AUTH,ADAT and other RFC959

17. 2003/3/27CHEP0317 GSI-pftp as NorduGrid SE zProtocols for parallel transfer and buffer management are different. zDCAU (Data Channel Authentication) is unique for Gridftp. But it is option of user. èGSI-pftpd and Grid-ftp client can successfully communicate each other excepting parallel transfer.

18. 2003/3/27CHEP0318 &(executable=gsim1) (arguments="-d") (inputfiles= (”data.in" "gsiftp://dt05s.cc.kek.jp:2811/hpss/ce/chep/manabe/data2")) (stdout=datafiles.out) (join=true) (maxcputime="36000") (middleware="nordugrid") (jobname="HPSS access test") (stdlog="grid_debug") (ftpThreads=1) Sample XRSL

19. 2003/3/27CHEP0319 Computing Element in ICEPP/KEK HPSS server Disk Mover (Disk Cache) Players In HPSS Disk mover GSIpftp Server Disk mover CE (Gridftp client) x3 Tape: 3590 (14MB/s 40GB) 2CPU Power3 375MHz AIX 4.3 HPSS 4.3 2CPU Power3 375MHz AIX 4.3 HPSS 4.3 Globus 2.0 2CPU PenIII 1GHz RedHat 7.2 Globus 2.2 Tape movers Shared by many users

20. 2003/3/27CHEP0320 Computing Element HPSS Server Disk Mover Possible HPSS Configuration 1 KEK ICEPP 60km SP Switch 150MB/s z Put ‘disk mover (cache)’ near HPSS server. z Cache should be near to consumer but ‘disk mover’ is far from CE. z Get high-performance of SP switch. Super-SINET 1GbE

21. 2003/3/27CHEP0321 Computing Element HPSS Server Disk Mover Possible HPSS Configuration 2 Computing Element KEK ICEPP Super-SINET 1GbE LAN 1GbE zPut ‘remote disk mover(cache)’ near CE. zFast access between CE and cached files. zIf access to the same file from KEK side CE, long detour happen.

22. 2003/3/27CHEP0322 Computing Element HPSS Server Possible HPSS configuration 3 Computing Element HPSS Hierarchy 1 HPSS Hierarchy 2 HPSS Hierarchy 3 KEK ICEPP zTo avoid long access delay for CE in KEK, disk layer can be divided into two hierarchy. But complicated configuration is it.

23. 2003/3/27CHEP0323 Computing Element HPSS Server Disk Mover Possible HPSS Configuration 1 KEK ICEPP 60km Super-SINET 1GbE z Current Setup Computing Element LAN 1GbE LAN WAN

24. 2003/3/27CHEP0324 Performance zBasic Network performance. zHPSS Client API performance. zpftp client - pftp server performance. zGridftp client - pftp server performance.

25. Basic Network Performance z RTT~4ms z packet loss free. z MTU=1500 z CPU/NIC is bottleneck. z Max TCP Buffer Size(256k) in HPSS servers cannot changed. (optimized for IBM SP switch) WAN LAN

26. Basic network performance on Super-sinet z >4 TCP session gets MAX transfer speed. z If enough TCP buffer size ~1 session get almost MAX speed. 0246810 0 200 400 600 # of TCP session Aggregate Tx speed (MBit/s) Network transfer with # of TCP session Buffer size HPSS mover = 256kB ICEPP client KEK HPSS mover Client Buffer size = 100KB Client Buffer size = 1MB WAN

27. Disk mover disk performance z HPSS SSA raw disk performance read/write ~ 35/100 MB/s z PC farm’s disk performance. Read/write ~ 30-40MB/s

28. HPSS Client API HPSS disk CE memory LAN WAN

29. 2003/3/27CHEP0329 HPSS Client API zNW latency impacts to file transfer speed. yMax. raw TCP speed was almost same, but data transfer speed became 1/2 in RTT~4ms WAN. yThe reason is not clear yet. But frequent communication between HPSS core server and HPSS client exists? (every chunk size (=4MB) ?) ywrite overhead at single buffer transfer was bigger than read. y64MB buffer size was enough for RTT=~4ms network.

30. 0246810 0 20 40 60 80 KEK client ICEPP client ICEPP client Pwidth # of file transfer in parallel Transfer speed (MB/s) pftp-pftp HPSS mover disk -> Client to client /dev/null WAN LAN

31. 2003/3/27CHEP0331 pftp-pftp ‘get’ performance zWe compared GSI pftp-pftp transfer with normal kerb- pftp-pftp. Both had equivalent transfer speed. zSame as in Client-API transfer, even with enough buffer size, transfer speed in WAN is 1/2 of that in LAN. zSimultaneous multiple file transfer (>4) gain aggregate transfer bandwidth. We had 2 disk movers with 2 disk paths each (2x2=4) zSingle file transfer with multiple TCP session (pftp function (command=pwidth)) was not effective for RTT=4ms network with enough FTP buffer.

32. Client disk speed @ KEK = 48MB/s Client disk speed @ ICEPP=33MB/s 0246810 0 20 40 60 80 KEK client ( LAN ) ICEPP client( WAN ) # of file transfer in parallel Aggregate Transfer speed (MB/s) pftp-pftp HPSS mover disk Client client disk speed 35-45MB/s to /dev/null to client disk Ftp buffer=64MB

33. 2003/3/27CHEP0333 pftp-pftp get performance (2) zEven if each component (disk, network) has good performance. Total staging performance becomes bad. If access is done in serial way. CPU 100MB/s 640Mbps=80MB/s 40MB/s Total speed = 1/( 1/100 + 1/80 + 1/40) = 21MB/s

34. # of file transfer in parallel HPSS ‘get’ with Tape Library 0246810 0 100 200 300 Elapsed Time (sec) pftp-pftp get performance data was on HPSS mover disk data was in HPSS mover mounted tape data was in HPSS mover unmounted tape tape in drive tape off drive Data in disk cache Data in Tape zThanks to HPSS multi file transfer between tape and disk hierarchy, and enough number of tape drives, we could get speed up in multiple file transfer even if data was in tapes.

35. Aggregate GSI-pftp ‘put’ performance 1 file N files N files 1 file file (pwidth)

36. Gridftp client and GSI-pftp server disk mver (!=pftpd) client pftp-pftpd disk mver (=pftpd) client gridftp-pftpd disk mver (!=pftpd) client gridftp-pftpd

37. 2003/3/27CHEP0337 GSI-pftpd with Gridftp client zIt works ! zBut less secure than Gridftp-Gridftpd (omit data path authentication) zIn our environment, GridFTP parallel TCP transfer is not needed. zWith multiple disk mover, all data transfer go through single pftpd server. (if use with Gridftp client)

38. Path difference pftp - pftpd Gridftp – GSI-pftpd Tape mover Disk mover pftp Server Disk mover CE (pftp client) x3 Tape mover Disk mover pftp Server Disk mover CE (gridftp client) x3

39. 2003/3/27CHEP0339 Summary zICEPP and KEK configured NorduGrid test bed with HPSS storage server over High speed GbE WANetwork. zNetwork latency affected HPSS data transfer speed. z‘GSI-pftpd’ developed by LBL is successfully adopted to the interface between NorduGrid and HPSS. But it has room for performance improvement with multi-disk movers.

40. 2003/3/27CHEP0340 Summary zHPSS parallel mechanism (multi-disk/tape servers) was effective for utilize High-speed middle-range distance network bandwidth.