Status and Prospective in Japan KEK High Energy Accelerator Research Organization Status and Prospective in Japan Atsushi Manabe Computing Research Center (CRC) KEK Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Outline Grid projects (budget) in Japan academic region. HEP Data Grid activities in Japan. KEK Needs in KEK Strategy of KEK CRC Pacifi-Grid Gfarm project Grid/HPSS test bed on SupserSINET. ICEPP: ATLAS Japan Regional Center Int'l HEP Data Grid Work Shop in Daegu

Grid Projects(budget) in Japan (academic area only) MEXT(Ministry of Education, sports …. and technology) projects SuperSINET; Gigabit backbone for academic researchers in Japan MEXT Informatics Project A05: Application Grid projects for science ~ US$ 6M /5 y Astronomy (Virtual Observatory), io-Infomatics/Genome (BioGrid) …… Networking supercomputers: ITBL ~US$ 100M /5y National Research Grid Initiative ~US$ 300M/5y focused in Nano-tech application (LHC/ATLAS(ICEPP,KEK) METI (Ministry of Economy, Trade and Industry) project AIST GTRC(Grid Technology Research Center) ~ US$ 140M Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Gigabit network Super SINET 10 Gbps MEXT DWDM nationwide network About 40 universities and labs US$ 60M/yr 5 application areas: HEP, Nanotech, Bioinformatics, Astronomy, Grid Computing Tsukuba-WAN 330Gbps DWDM ring in the Tsukuba area Tens of national labs 6 supercomputers in TOP100 Testbed for Grid infrastructure and applications Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu IP 10 Gbps WDM path NIFS IMS IP router OXC Nagoya U NIG Nagoya Hub Hokkaido Tohoku U Osaka Hub Tokyo Hub KEK Osaka U Tsukuba US，EU NII Chiba Kyoto U Korea NII Kyushu　U ISAS U Tokyo ICR Kyoto-U Internet TITEC NAO IMS Waseda U-Tokyo Network Topology of SuperSINET Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu SuperSINET Backbone KEK Titech ICEPP Int'l HEP Data Grid Work Shop in Daegu

Network Infrastructure TEIN NII- NY TRANSPAC Taiwan-US Japan (NII) -NY : 2.4G x 2 Jan. 2003 Japan – US: 622M x 2 (TRANSPAC) Korea – US: 45 M Korea – Japan: 2.4G Jan. 2003 Korea – Europe: 2 M (TEIN) China(IHEP) – Japan(KEK): 128 kbps (HEP) China – US: 10 M Taiwan – Japan: 155 M Taiwan – US: 622 M (Dec. 2002) Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu MEXT Grid Projects MEXT “Informatics” Project : A05 “Grid” Area Lead by Shinji Shimozo(Osaka U.) & Satoshi Matsuoka(Titech) 11 groups of universities and labs Research and development on application, middleware, and infrastructure Support the Virtual Observatory(astronomy) and the BIO-Grid(bio-Infomatics/Genome) projects Tokyo Institute of Technology “Titech Campus Grid” (US$ 2M) Lead by Satoshi Matsuoka(Titech) Total of 800 CPU PC clusters at 13 locations in a 30km range connected by Super TITANET(1-4Gbps) backbone Pilot project for virtual parallel computer infrastructure for high end computational e-Science Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu ITBL ITBL(IT-based Laboratory) To Connect Super-Computers in Government Labs: NAL, RIKEN, NIED, NIMS, JST, JAERI Project period: 2001-2005 (3-stage project) with total of US$ 105M funding Software development project. Applications: mechanical simulation, computational biology, material science, environment, earthquake engineering Step 1: Supercomputer centers of government lab are networked via SuperSINET Step 2: “Virtual Research Environment”: Grid-enabling laboratory applications Step 3: Sharing information among researchers from widely distributed disciplines and institutions Int'l HEP Data Grid Work Shop in Daegu

NAREGI (National Research Grid Initiative) NII (The National Institute of Informatics) under MEXT project. Making Center of Grid Research & Development Development of Grid Lower and Middleware (Grid scheduler,VM, RPC, MPI, GUI tools…) Networking, Security and User management (Routing, robust TCP/IP protocol, Traffic management) Applications for Nano-science and technology. Joint with Institute for Molecular Science. http://www.grid.nii.ac.jp/ Int'l HEP Data Grid Work Shop in Daegu

Grid Technology Research Center Part of AIST under METI (Ministry of Economy, Trade and Industry) GTRC (Grid Technology Research Center) Established in Jan. 2002 Director: Satoshi Sekiguchi US$140M for building, equipment, R&D and operations AIST super server; PC cluster 2004-2Q 11TFLOPS/ AMD Operon x 2116CPUs To be central institute for Grid research in Japan Not just R&D, but serve as central CA, ApGrid’s principal site Collaboration with institutions and other Grid projects of both domestic and international scale Organizing GRID Consortium Japan http://unit.aist.go.jp/grid/ Int'l HEP Data Grid Work Shop in Daegu

Grid related Activity at KEK Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Major projects in KEK KEK sponsors projects which use accelerators and also projects in related fields KEKB accelerator BELLE experiment Proton synchrotron nuclear and elementary particle physics medical Synchrotron Radiation Facility bio, material science and so on Also many theorists in related fields Support International HEP collaboration ATLAS Japan Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu KEK Future projects J-PARC will be built in Tokai div. of Japan Atomic Energy Research Institute (JAERI) where ~50km north from KEK start the operation in JFY2005 50 GeV high-intensity proton synchrotron nuclear physics and elementary particle physics Super B factory x30 Luminosity(require x10 CPU power). Under discussion. Linear Collider discussion is still under way Int'l HEP Data Grid Work Shop in Daegu

Computing Facilities at KEK Computing Research Center offers group oriented computing services Exclusive computing resource for main groups Networking, storage and computing resource User group owns their own computer systems PC farms Int'l HEP Data Grid Work Shop in Daegu

Main computing facilities KEKCC computer system for experiments at PS, linear collider R&D, theory, ATLAS and so on IBM SP3, 120TB HPSS storage KEKB computer system for Belle Sun WSs and PC farms. 500TB storage. KEKSC: Super-computer system for mainly lattice QCD G. Hitachi SR-8000 100 nodes; almost 1TFLOPTS Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Our needs in Grid Era ATLAS Regional Center (Grid is premise) Being set up at ICEPP (Int’l Center for Particle Physics, U. Tokyo) Still KEK have large portion of ATLAS Japan members and they will make data analysis using computing resource at KEK. We will support them. And BELLE; international collaboration and require more and more CPU and storage. J-PARC; joint project of other Inst. regionally separated. Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu BELLE BELLE is the major project at KEK They have many collaborators world wide They always needs additional CPU power because acc. luminosity improving and improving. They seek a solution of their problems related computing issue using GRID. We interviewed the BELLE computing coordinator and got the wish list shown in the following slides Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Wish lists from BELLE What BELLE wants: A way to integrate CPU powers and way to distribute data among collaborators’ Univ.s and Inst.s for their data production. Automatic way to submit analysis jobs in the distributed environment. once analysis codes have placed somewhere, process for all of experimental data and Monte Carlo should be done automatically and simply users gain the result Automatic Monte Carlo production Automatic version control for libraries, constants and others on all of the computer system in BELLE Automatic update on reconstructed data or Monte Carlo if libraries or constants are once updated Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Wish lists from BELLE 2 It is not easy to achieve their requests technically simplest solution will be hire more Post-Docs or find more grad-students, however We have decomposed their requests should be done in application level (BELLE) or can be solved by GRID Int'l HEP Data Grid Work Shop in Daegu

What they need related GRID An efficient way to distribute jobs and way to manage/share/distribute data for the jobs. effective data catalogue physical to logical mapping Seamless and unique file tree query file names, contents and others works on the Internet but secure efficient and easy to distribute data to remote sites easy to use web interface and GUI also Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu continued mass storage management system not only tapes but also large amount of disks robust and fast transfer easy integration with other component We already have them partially on LAN, but we need to expand them to the Internet. We hope GRID will solve them. Int'l HEP Data Grid Work Shop in Daegu

KEK CRC strategy towards GRID Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Done or On going. Install GT2(Globus toolkit ver.2) and applications (LSF) to two CRC Systems (KEKCC & KEKB) ) and will open to some limited users. (remote Job submission) Build temporal CA with single Virtual Organization(VO). Build a inter-institute wide Grid test bed on SuperSINET and make a performance test. (talk later) Trial use of SRB with Belle data in a test system. Start to touch LCG tools by ATLAS Japan people. watch technical trends carefully. Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu SRB SRB is available today and looks useful even in HEP ( http://www.npaci.edu/DICE/SRB/ ) developed at San Diego Super Computer Center seamless solution to HPSS good for the KEKCC (central computer) users Data catalogue and replication SLAC is also testing Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Until early 2004 Deploy GT3, evaluate LCG tools and other something new technology. Build own VO at each sites and integrate them. File sharing by SRB (or ?) and seamless job submission between KEKCC and KEKB Pacifi-Grid test environment; Pan-Pacific collaboration (see later) KNU (Korea), Univ. of Melbourne (Australia), SLAC and KEK will test SRB Start to develop higher level software (experiment side) on the top of Globus and SRB. Message passing with GSI authentication ? Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu ~2004 KEKB and KEKCC (5yeas rental )system replacement (2005-4Q) . GRID will become concrete until this stage or die Final specification have to be decided Decision of middle ware (Grid or else?) CA Security human resource trained engineer on site Int'l HEP Data Grid Work Shop in Daegu

Other topics discussed internally CA nation wide, institution wide or else? KEK certify all Japanese HEP user? Rules for certification of users. PKI secure enough? Safety private key management (IC card?) management costs? seamless integration with local authentication Network How we can overcome latency? better connectivity to the Internet, routing and throughput middleware Globus, Avaki or something else SRB, LCG or other data replication solution Storage SAN StorageTank (IBM) security vs usability Int'l HEP Data Grid Work Shop in Daegu

Pacifi-Grid: Proposal on Open Pacific Grid collaboration Takashi Sasaki (KEK) Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu charter Give research institutions and IT vendors opportunity of testing Grid technologies under inter-state environment. Emphasis on Data Grid Open to everybody who wants to join Grid is a newly emerging technology and more experiences are necessary before deploying at the large scale production systems Test on high latency network robustness tests interoperability tests among sites and vendors security Int'l HEP Data Grid Work Shop in Daegu

Collaboration (under discussion) Korea U of Melbourne, Australia SLAC, USA JAPAN ICEPP (Univ. of Tokyo) KEK Some venders Under discussing with Fujitsu, IBM Japan and so on Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Gfarm R&D Development R&D Grid middleware of Cluster File System with job execution. Int'l HEP Data Grid Work Shop in Daegu

R&D of Gfarm (Grid data farm) Development by CS scientist of AIST, Titech and KEK since 2000 AIST :Advanced Inst. for Science and Technology Architecture PC farm with large local disk/node as a GRID fabric Large data file is divided into fragments and stored in the disks Data I/O by parallel file system Affinity scheduling of process and storage for data residence Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Data Access Locality computing node storage high speed switch job manager A B user job job manager A B user job computing node storage element Exploit the data access locality as much as possible Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Gfarm http://datafarm.apgrid.org/ Grid Data Farm : Beta 3 release is now available. Parallel I/O: exploit the data access locality, store and access files by “fragments” Parallel Job: program runs on the nodes where the file fragments reside: “owner computes” Programs and file fragments, job history, data checksum, file replication are managed with MetaDB Fragments are replicated for backup and load balancing User sees the fragmented files as a single Gfarm URL Provide system call hooks for open(), close(), read(), write() etc Authentication: Globus GSI and/or Shared Private Key Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu FADS/Goofy in ATLAS Used in sub-detector software developments and physics validations Detector modules also run in ATLAS mainstream framework (Athena) Testbed for ROOT I/O Bandwidth Challenge in SC2002 Generated 106 fully simulated higss  4 events in 2 days with ~ 400 CPU Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu Gfarm achievement “File affinity scheduling” provides parallel processing capability for both I/O-bound and CPU-intensive jobs. FADS/Goofy was successfully implemented on Gfarm architecture. In the process, ROOT I/O module works with Gfarm parallel file system using the system call hook, without a changes to the ROOT package. ROOT I/O files has been successfully replicated at 2.286 Gbps using the SC2002 Bandwidth Challenge testbed with 12 nodes ~ 190Mbps/node. Please refer to CHEP03 reports by Y.Morita and O.Tatebe Int'l HEP Data Grid Work Shop in Daegu

Status of ATLAS Japan RC (ICEPP U. of Tokyo) R&D is ongoing Test-buy of PC farm and storage 1U P3, P4 servers ( >150nodes and more) , LTO (10TB) with CASTOR and >20TB disk servers. Nordu GRID was tested first Data transfer tests between U. Tokyo and KEK was done ATLAS data challenges LCG 0 and 1 Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu

LCG-0 Deployment Status These sites deployed the LCG-0 pilot system and will be the first sites to deploy LCG-1 Site Scheduled Status Tier 1 CERN 15/2/03 Done 1 CNAF 28/2/03 2 RAL 3 FNAL 30/3/03 4 Taipei 15/4/03 5 FZK 30/4/03 6 IN2P3 7/5/03 In prep. 7 BNL 15/5/03 8 Russia (Moscow) 21/5/03 9 Tokyo Tier 2 10 Legnaro (INFN) After CNAF Int'l HEP Data Grid Work Shop in Daegu

LCG Resource Commitments – 1Q04   CPU (kSI2K) Disk TB Support FTE Tape CERN 700 160 10.0 1000 Czech Republic 60 5 2.5 France 420 81 10.2 540 Germany 207 40 9.0 62 Holland 124 3 4.0 12 Italy 507 16.0 100 Japan 220 45 5.0 Poland 86 9 28 Russia 120 30 Taiwan Spain 150 Sweden 179 2.0 Switzerland 26 UK 1656 226 17.3 295 USA 801 176 15.5 1741 Total 5600 1169 120.0 4223 Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu

Status of ATLAS Japan RC 2 Budget requests have been submitted to the funding agency still no decision has been made and no decision will be made within this FY All of national universities and some national laboratories including KEK will be non-nationalized (not privatized, but will be a kind of independent governmental agencies) Int'l HEP Data Grid Work Shop in Daegu

Summary of KEK Grid activity KEK had a step forward for the Grid computing People at ATLAS Japan RC are working on LCG Realistic environment is necessary to test middleware and applications We have started the discussion among Pan-Pacific sites to collaborate each other We want to exchange knowledge and experiences Int'l HEP Data Grid Work Shop in Daegu

Int'l HEP Data Grid Work Shop in Daegu End of my presentation Thank you Int'l HEP Data Grid Work Shop in Daegu