1. HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK

2. 28/07/2004 Takashi.Sasaki@kek.jp Contents Japanese HENP projects Belle GRID ATLAS-Japan Regional Center Collaboration in Asia Pacific

3. 28/07/2004 Takashi.Sasaki@kek.jp Major HENP projects in Japan KEK Proton Synchrotron K2K etc. KEKB –BELLE experiment 300 members from 54 institutes in10 countries Super-B is planning J-PARC –New facility under construction at Tokai (~2008) –particle physics, material science and life science T2K International collaboration –LHC ATLAS, CERN –HERA ZEUS, DESY –RHIC, BNL –etc.

4. 28/07/2004 Takashi.Sasaki@kek.jp KEK-B Belle Experiment

5. 28/07/2004 Takashi.Sasaki@kek.jp 1km

6. 28/07/2004 Takashi.Sasaki@kek.jp Belle Data 15MB/sec data rate in maximum 200TB/year raw data recorded Equivalent number of Monte Carlo events

7. 28/07/2004 Takashi.Sasaki@kek.jp Belle PC farms

8. 28/07/2004 Takashi.Sasaki@kek.jp Super KEK-B L = 10^35cm^-2s^-1 in 2007? Data rate ~ 250MB/s

9. 28/07/2004 Takashi.Sasaki@kek.jp K2K now upgrading to T2K

10. 28/07/2004 Takashi.Sasaki@kek.jp T2K (Tokai to Kamioka) J-PARC –for particles physics, material and life science –joint project of JAERI and KEK –100 times intense neutrino beam High trigger rate at the near detector –Operational in 2007

11. 28/07/2004 Takashi.Sasaki@kek.jp LHC-ATLAS Japanese contribution –Semiconductor Tracker –Endcap Muon Trigger System –Muon Readout Electronics –Superconducting Solenoid –DAQ system –Regional Analysis Facility ICEPP will the site of the regional center

12. 28/07/2004 Takashi.Sasaki@kek.jp HEP GRID related activities Networking –domestic/international BELLE –data distribution and remote MC production ALTAS Japan –collaboration between ICEPPand KEK Hadron therapy simulation –application of HEP tools into medical field

13. 28/07/2004 Takashi.Sasaki@kek.jp SuperSInet Major HEP sites have a DWDM connection to KEK with 1Gbps

14. 28/07/2004 Takashi.Sasaki@kek.jp NY Chicago To Novosibirsk 0.5 M 10 G 5 G -> 10G ASNET/TANET2 622 M APII 1G SuperSINET APAN TransPAC CERNET 155 M Tsukuba WAN Taiwan China Russia USA SuperSINET Current Network Connections around Japan As of July 2004 SINET 2M Thailand Hawaii 155M Korea

15. 28/07/2004 Takashi.Sasaki@kek.jp BELLE GRID Distributed analysis and data sharing among institutes –Experimental data distribution to remote sites –Monte Carlo production at remote sites and send back events to KEK –remote job submission –etc Testing SRB with GSI now –among Australia, Taiwan and KEK hopefully Korea, soon

16. 28/07/2004 Takashi.Sasaki@kek.jp Why SRB? Easy to install, use and manage –Most of Belle’s collaborating institutes are middle or small size universities poor in man power Useful features –file replication –parallel data transfer –Grid aware: GSI authentication –Command line interface, API, GUI –Fancy Web interface –etc Excellent user support –quick response –seminars Available today! –Solution is needed for Belle

17. 28/07/2004 Takashi.Sasaki@kek.jp SRB(Storage Resource Broker) SRB server SRB server SRB server SRB server Internet disk tape DB nfs MCAT RAID SRB client SRB client Zone

18. 28/07/2004 Takashi.Sasaki@kek.jp SRB zone federation SRB server SRB server SRB server SRB server Internet disk tape DB nfs MCAT RAID disk DB MCAT SRB server SRB server SRB server ZoneA ZoneB

19. 28/07/2004 Takashi.Sasaki@kek.jp SRB Command line interface

20. 28/07/2004 Takashi.Sasaki@kek.jp SRB test bed system HPSS 120TB ZONE:glsrb01 Fire Wall MCAT ZONE:anusf store Internet federation MCAT PostgreSQL gtdmz01 ZONE:gtsrb13 ZONE:kekgt15 KEKAustralia gt13 gl03 gl01 MCAT DB2 MCAT PostgreSQL gt15 RAID 800GB bcs20 NFS (Belle data) BELLE Computer system client DMZ BELLE Secure net

21. 28/07/2004 Takashi.Sasaki@kek.jp Status Zone federation between ANU and KEK has been established –ANU, University of Melbourne and University of Sydney are collaborating for BELLE/ATLAS Grid issue One MCAT running at ANU –Data can store/retrieve on Belle data system and also HPSS in KEK side Zone federation between Academia Sinica, Taiwan and KEK has been established –still need to solve some problems

22. 28/07/2004 Takashi.Sasaki@kek.jp BELLE GRID Future plan Participation of Korean sites Mutual job submission using Globus LCG2+SRB (if they wish) –LCG2 is under testing at KEK also with help of ICEPP. –Because many foreign institutes are working both for BELLE and one of LHC experiments, they want to use LCG rather than vanilla Globus baby “tier-0” at KEK and “tier-1” at ICEPP –SRB and LCG-RLS synchronization will be tried based on Simon Matson’s (Bristol, CMS) work

23. 28/07/2004 Takashi.Sasaki@kek.jp Grid in ATLAS Japan Regional analysis center for ATLAS –ICEPP, the University of Tokyo Joint collaboration between ICEPP and KEK for Grid deployment

24. 28/07/2004 Takashi.Sasaki@kek.jp LCG MW Deployment in Japan RC Pilot Model System@ICEPP –Since 2002 –LCG testbed. Now LCG2_1_1 LCG2 test bed@KEK –baby tier-0 –LCG2_1_1 Regional Center Facility –Will be introduced in JFY2006 –Aiming “Tier1” size resource

25. 28/07/2004 Takashi.Sasaki@kek.jp

26. 28/07/2004 Takashi.Sasaki@kek.jp SuperSInet Performance Measurement “A” setting TCP 479Mbps -P 1 -t 1200 -w 128KB TCP 925Mbps -P 2 -t 1200 -w 128KB TCP 931Mbps -P 4 -t 1200 -w 128KB UDP 953Mbps -b 1000MB -t 1200 -w 128KB "A" setting: 104.9MB/s "B" setting: 110.2MB/s “B” setting (longer window size) TCP 922Mbps -P 1 -t 1200 -w 4096KB UDP 954Mbps -b 1000MB -t 1200 -w 4096KB DWDM

27. 28/07/2004 Takashi.Sasaki@kek.jp 1Gbps 100Mbps ICEPPKEK 100 CPUs 6CPUs HPSS 120TB GRID testbed environment with HPSS through GbE-WAN NorduGrid - grid-manager - gridftp-server Globus-mds Globus-replica PBS server NorduGrid - grid-manager - gridftp-server Globus-mds PBS server PBS clients HPSS servers ~ 60km 0.2TB SECE SE User PCs

28. 28/07/2004 Takashi.Sasaki@kek.jp Client disk speed @ KEK = 48MB/s Client disk speed @ ICEPP=33MB/s 0246810 0 20 40 60 80 KEK client ( LAN ) ICEPP client( WAN ) # of file transfer in parallel Aggregate Transfer speed (MB/s) Pftp→pftp HPSS mover disk Client disk to /dev/null to client disk Ftp buffer=64MB Even 3ms latency affects on results

29. 28/07/2004 Takashi.Sasaki@kek.jp microscopic monitoring on network performance speed=increment of sum of TCP data size in every 10ms window size grows slowly after packet losss longer windows is not perfect when you have packet loss w/o packet loss w/ packet loss

30. 28/07/2004 Takashi.Sasaki@kek.jp FAST http://netlab.caltech.edu/FAST/ Looks nice and we want to try, but we haven’t because of the patent and IP issue. –Some people at KEK afraid that they might have a difficulty to work in the similar topic once they see the source code of it. –Is this true?

31. 28/07/2004 Takashi.Sasaki@kek.jp CA A key issue of GRID ICEPP and KEK will run one CA jointly for BELLE and ATLAS –only for active KEK users to simplify the procedure –current situation ICEPP depends on a foreign CA to join LCG KEK is running a test CA locally CA management costs are not cheap. Any good idea?

32. 28/07/2004 Takashi.Sasaki@kek.jp Storage evaluation SAN solutions –IBM SANfile (aka. StorageTank) Linux+AIX in server side tested AIX, Solaris and Linux clients –Linux clients were fastest in our tests –HP Lustre waiting for beta product delivery

33. 28/07/2004 Takashi.Sasaki@kek.jp Distributed simulation in advanced radio therapy A model on hospitals and computer centers –Hospitals send CT, MRI or PET images (DICOM) with treatment planning data to a computing center as input higher security is required to protect personal data –Full simulation using Geant4 at computing centers parallel simulation –Analysis results and feedbacks in DICOM to hospitals Validation of treatment planning

34. 28/07/2004 Takashi.Sasaki@kek.jp Toward Asia-Pacific collaboration Taka an advantage on working together with people in neighboring time zones HEP population in Asia-Pacific –ATLAS Australia(7), China(15), Japan(45) and Taiwan(5) 72/1446 = 5.0% –CMS China(31), Korea(17) and Taiwan(14) 62/1676 = 3.7% –Alice China(20), Japan(3) and Korea(12) 35/747 = 4.7% –LHC-b China(19) 19/737 = 2.6% –Belle Australia(8),China(11),Japan(122),Korea(15) and Taiwan(13) 169/246 = 67% (excluding Japan 20%)

35. 28/07/2004 Takashi.Sasaki@kek.jp

36. 28/07/2004 Takashi.Sasaki@kek.jp Summary Japan is an unique country in Asia-Pacific region which have large accelerators for HENP –should have a “tier-0” center SRB is under testing for BELLE ICEPP, U-Tokyo and KEK are collaborating together to build the ATLAS regional center in Japan We seek a collaboration among Asia-Pacific countries –BELLE, LHC –More bandwidth is necessary among sites