HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK

28/07/2004 Contents Japanese HENP projects Belle GRID ATLAS-Japan Regional Center Collaboration in Asia Pacific

28/07/2004 Major HENP projects in Japan KEK Proton Synchrotron K2K etc. KEKB –BELLE experiment 300 members from 54 institutes in10 countries Super-B is planning J-PARC –New facility under construction at Tokai (~2008) –particle physics, material science and life science T2K International collaboration –LHC ATLAS, CERN –HERA ZEUS, DESY –RHIC, BNL –etc.

28/07/2004 KEK-B Belle Experiment

28/07/2004 1km

28/07/2004 Belle Data 15MB/sec data rate in maximum 200TB/year raw data recorded Equivalent number of Monte Carlo events

28/07/2004 Belle PC farms

28/07/2004 Super KEK-B L = 10^35cm^-2s^-1 in 2007? Data rate ~ 250MB/s

28/07/2004 K2K now upgrading to T2K

28/07/2004 T2K (Tokai to Kamioka) J-PARC –for particles physics, material and life science –joint project of JAERI and KEK –100 times intense neutrino beam High trigger rate at the near detector –Operational in 2007

28/07/2004 LHC-ATLAS Japanese contribution –Semiconductor Tracker –Endcap Muon Trigger System –Muon Readout Electronics –Superconducting Solenoid –DAQ system –Regional Analysis Facility ICEPP will the site of the regional center

28/07/2004 HEP GRID related activities Networking –domestic/international BELLE –data distribution and remote MC production ALTAS Japan –collaboration between ICEPPand KEK Hadron therapy simulation –application of HEP tools into medical field

28/07/2004 SuperSInet Major HEP sites have a DWDM connection to KEK with 1Gbps

28/07/2004 NY Chicago To Novosibirsk 0.5 M 10 G 5 G -> 10G ASNET/TANET2 622 M APII 1G SuperSINET APAN TransPAC CERNET 155 M Tsukuba WAN Taiwan China Russia USA SuperSINET Current Network Connections around Japan As of July 2004 SINET 2M Thailand Hawaii 155M Korea

28/07/2004 BELLE GRID Distributed analysis and data sharing among institutes –Experimental data distribution to remote sites –Monte Carlo production at remote sites and send back events to KEK –remote job submission –etc Testing SRB with GSI now –among Australia, Taiwan and KEK hopefully Korea, soon

28/07/2004 Why SRB? Easy to install, use and manage –Most of Belle’s collaborating institutes are middle or small size universities poor in man power Useful features –file replication –parallel data transfer –Grid aware: GSI authentication –Command line interface, API, GUI –Fancy Web interface –etc Excellent user support –quick response –seminars Available today! –Solution is needed for Belle

28/07/2004 SRB(Storage Resource Broker) SRB server SRB server SRB server SRB server Internet disk tape DB nfs MCAT RAID SRB client SRB client Zone

28/07/2004 SRB zone federation SRB server SRB server SRB server SRB server Internet disk tape DB nfs MCAT RAID disk DB MCAT SRB server SRB server SRB server ZoneA ZoneB

28/07/2004 SRB Command line interface

28/07/2004 SRB test bed system HPSS 120TB ZONE:glsrb01 Fire Wall MCAT ZONE:anusf store Internet federation MCAT PostgreSQL gtdmz01 ZONE:gtsrb13 ZONE:kekgt15 KEKAustralia gt13 gl03 gl01 MCAT DB2 MCAT PostgreSQL gt15 RAID 800GB bcs20 NFS (Belle data) BELLE Computer system client DMZ BELLE Secure net

28/07/2004 Status Zone federation between ANU and KEK has been established –ANU, University of Melbourne and University of Sydney are collaborating for BELLE/ATLAS Grid issue One MCAT running at ANU –Data can store/retrieve on Belle data system and also HPSS in KEK side Zone federation between Academia Sinica, Taiwan and KEK has been established –still need to solve some problems

28/07/2004 BELLE GRID Future plan Participation of Korean sites Mutual job submission using Globus LCG2+SRB (if they wish) –LCG2 is under testing at KEK also with help of ICEPP. –Because many foreign institutes are working both for BELLE and one of LHC experiments, they want to use LCG rather than vanilla Globus baby “tier-0” at KEK and “tier-1” at ICEPP –SRB and LCG-RLS synchronization will be tried based on Simon Matson’s (Bristol, CMS) work

28/07/2004 Grid in ATLAS Japan Regional analysis center for ATLAS –ICEPP, the University of Tokyo Joint collaboration between ICEPP and KEK for Grid deployment

28/07/2004 LCG MW Deployment in Japan RC Pilot Model –Since 2002 –LCG testbed. Now LCG2_1_1 LCG2 test –baby tier-0 –LCG2_1_1 Regional Center Facility –Will be introduced in JFY2006 –Aiming “Tier1” size resource

28/07/2004

28/07/2004 SuperSInet Performance Measurement “A” setting TCP 479Mbps -P 1 -t w 128KB TCP 925Mbps -P 2 -t w 128KB TCP 931Mbps -P 4 -t w 128KB UDP 953Mbps -b 1000MB -t w 128KB "A" setting: 104.9MB/s "B" setting: 110.2MB/s “B” setting (longer window size) TCP 922Mbps -P 1 -t w 4096KB UDP 954Mbps -b 1000MB -t w 4096KB DWDM

28/07/2004 1Gbps 100Mbps ICEPPKEK 100 CPUs 6CPUs HPSS 120TB GRID testbed environment with HPSS through GbE-WAN NorduGrid - grid-manager - gridftp-server Globus-mds Globus-replica PBS server NorduGrid - grid-manager - gridftp-server Globus-mds PBS server PBS clients HPSS servers ~ 60km 0.2TB SECE SE User PCs

28/07/2004 Client disk KEK = 48MB/s Client disk ICEPP=33MB/s KEK client ( LAN ) ICEPP client( WAN ) # of file transfer in parallel Aggregate Transfer speed (MB/s) Pftp→pftp HPSS mover disk Client disk to /dev/null to client disk Ftp buffer=64MB Even 3ms latency affects on results

28/07/2004 microscopic monitoring on network performance speed=increment of sum of TCP data size in every 10ms window size grows slowly after packet losss longer windows is not perfect when you have packet loss w/o packet loss w/ packet loss

28/07/2004 FAST Looks nice and we want to try, but we haven’t because of the patent and IP issue. –Some people at KEK afraid that they might have a difficulty to work in the similar topic once they see the source code of it. –Is this true?

28/07/2004 CA A key issue of GRID ICEPP and KEK will run one CA jointly for BELLE and ATLAS –only for active KEK users to simplify the procedure –current situation ICEPP depends on a foreign CA to join LCG KEK is running a test CA locally CA management costs are not cheap. Any good idea?

28/07/2004 Storage evaluation SAN solutions –IBM SANfile (aka. StorageTank) Linux+AIX in server side tested AIX, Solaris and Linux clients –Linux clients were fastest in our tests –HP Lustre waiting for beta product delivery

28/07/2004 Distributed simulation in advanced radio therapy A model on hospitals and computer centers –Hospitals send CT, MRI or PET images (DICOM) with treatment planning data to a computing center as input higher security is required to protect personal data –Full simulation using Geant4 at computing centers parallel simulation –Analysis results and feedbacks in DICOM to hospitals Validation of treatment planning

28/07/2004 Toward Asia-Pacific collaboration Taka an advantage on working together with people in neighboring time zones HEP population in Asia-Pacific –ATLAS Australia(7), China(15), Japan(45) and Taiwan(5) 72/1446 = 5.0% –CMS China(31), Korea(17) and Taiwan(14) 62/1676 = 3.7% –Alice China(20), Japan(3) and Korea(12) 35/747 = 4.7% –LHC-b China(19) 19/737 = 2.6% –Belle Australia(8),China(11),Japan(122),Korea(15) and Taiwan(13) 169/246 = 67% (excluding Japan 20%)

28/07/2004

28/07/2004 Summary Japan is an unique country in Asia-Pacific region which have large accelerators for HENP –should have a “tier-0” center SRB is under testing for BELLE ICEPP, U-Tokyo and KEK are collaborating together to build the ATLAS regional center in Japan We seek a collaboration among Asia-Pacific countries –BELLE, LHC –More bandwidth is necessary among sites