1. 3/27/2007Grid Efforts in Belle1 Hideyuki Nakazawa (National Central University, Taiwan), Belle Collaboration, KEK

2. 3/27/2007Grid Efforts in Belle2 Out Line Belle experiment Computing system in Belle LCG at KEK and Belle VO status Introduction of SRB Summary

3. 3/27/2007Grid Efforts in Belle3 KEKB Accelerator Asymmetric e+e- collider 3.5 GeV on 8 GeV 3 km circumference 22mrad Crossing Angle Continuous Injection Belle Detector Generic purpose 7 sub-detectors “B factory” experiment at KEK (Japan). Belle KEKB Linac 3km Mt. Tsukuba Belle Experiment

4. 3/27/2007Grid Efforts in Belle4 Belle Collaboration 13 countries, 57 institutes, ~400 collaborators IHEP, Vienna ITEP Kanagawa U. KEK Korea U. Krakow Inst. of Nucl. Phys. Kyoto U. Kyungpook Nat’l U. EPF Lausanne Jozef Stefan Inst. / U. of Ljubljana / U. of Maribor U. of Melbourne Aomori U. BINP Chiba U. Chonnam Nat’l U. U. of Cincinnati Ewha Womans U. Frankfurt U. Gyeongsang Nat’l U. U. of Hawaii Hiroshima Tech. IHEP, Beijing IHEP, Moscow Nagoya U. Nara Women’s U. National Central U. Nat’l Kaoshiung Normal U. National Taiwan U. National United U. Nihon Dental College Niigata U. Osaka U. Osaka City U. Panjab U. Peking U. U. of Pittsburgh Princeton U. Riken Saga U. USTC Seoul National U. Shinshu U. Sungkyunkwan U. U. of Sydney Tata Institute Toho U. Tohoku U. Tohuku Gakuin U. U. of Tokyo Tokyo Inst. of Tech. Tokyo Metropolitan U. Tokyo U. of Agri. and Tech. Toyama Nat’l College U. of Tsukuba Utkal U. VPI Yonsei U. Lots of contribution from Taiwan

5. 3/27/2007Grid Efforts in Belle5 Luminosity Produce large amount of B mesons!! peak luminosity 1.7118 × 10 34 cm -2 s -1 710 fb - 1 1 fb - 1 ~ 10 6 BB Integrated Luminosity ( fb - 1 ) ● Crab Cavity installed, being tuned now. being tuned now. Luminosity doubled? Luminosity doubled? Integrated Luminosity 1 fb -1 ~ 1TB / day

6. 3/27/2007Grid Efforts in Belle6 History of Belle computing system Performance1997- 4 years 2001- 5 years 2006- 6 years Computing Server [SPECint2000 rate] ~100 (WS) ~1,250 (WS+PC) ~42,500 (PC) Disk Capacity [TB] ~4~91000 Tape Library Capacity [TB] 1606203,500 Work Group Server [# of hosts] 3+(9)1180+16FS User Workstation [# of hosts] 25WS +68X 23WS +100PC 128PC

7. 3/27/2007Grid Efforts in Belle7 Overview of the B Computer Storage Computing Servers Workgroup Servers reserved for Grid On-line Reconstruction Farm

8. 3/27/2007Grid Efforts in Belle8 Belle System Computing Server: ~42,500 SPECint2K Storage System (DISK): 1PB Storage System (HSM): 3.5PB

9. 3/27/2007Grid Efforts in Belle9 Data Production at Belle online reconstruction farm “MDST” data (four vector, PID info etc.) (four vector, PID info etc.) rawdata + “DST” data production Users' analyes hadron 120TB + others ~ 1PB MC Generation and Detector Simulation 2.5THz (to finish in 6 months) 2THz (to finish in 2 months) HSM non-HSM Loose Selection Criteria @500/fb

10. 3/27/2007Grid Efforts in Belle10 Why Grid in Belle? No urgent requirement No urgent requirement Belle shifts to precise and exotic measurement Belle shifts to precise and exotic measurement More MC statistics necessary for precise measurement More MC statistics necessary for precise measurement New skim for exotic process New skim for exotic process Lesson in de facto standard Lesson in de facto standard Maybe we should start considering about Grid Just my feeling

11. 3/27/2007Grid Efforts in Belle11 Grid Introduction Strategy Strong support from KEK CRC Strong support from KEK CRC Starting with MC production and accumulating experiences, gradually shift to handle experimental data Starting with MC production and accumulating experiences, gradually shift to handle experimental data Recruitment Recruitment Some collaborators who have running LCG are preparing to join the Belle VO Some collaborators who have running LCG are preparing to join the Belle VO Experiencing Grid potential may change Experiencing Grid potential may change Belle’s recognition ?

12. 3/27/2007Grid Efforts in Belle12 LCG Deployment at KEK JP-KEK-CRC-01JP-KEK-CRC-01 Since Nov. 2005. Since Nov. 2005. Registered to GOC, in operation as WLCG Registered to GOC, in operation as WLCG Site Role: Site Role: practice for production system JP-KEK-CRC-02. practice for production system JP-KEK-CRC-02. test use among university groups in Japan. test use among university groups in Japan. Resource and Component: Resource and Component: SL-3.0.5 w/ gLite-3.0 later SL-3.0.5 w/ gLite-3.0 later CPU: 14, Storage: ~1.5TB CPU: 14, Storage: ~1.5TB FTS, FTA, RB, MON, BDII, LFC, CE, SE FTS, FTA, RB, MON, BDII, LFC, CE, SE Supporting VOs: Supporting VOs: belle, apdg, g4med, ppj, dteam, ops and ail belle, apdg, g4med, ppj, dteam, ops and ail JP-KEK-CRC-02JP-KEK-CRC-02 Since early 2006. Since early 2006. Registered to GOC, in operation as WLCG Registered to GOC, in operation as WLCG Site Role: Site Role: More stable services based on KEK-1 experiences. More stable services based on KEK-1 experiences. Resource and Component: Resource and Component: SL or SLC w/ gLite-3.0 later SL or SLC w/ gLite-3.0 later CPU: 48, Storage: ~1TB (w/o HPSS) CPU: 48, Storage: ~1TB (w/o HPSS) Full components Full components Supporting VOs: Supporting VOs: belle, apdg, g4med, atlasj, ppj, ilc, dteam, ops and ail belle, apdg, g4med, atlasj, ppj, ilc, dteam, ops and ail Operation is supported by great efforts by APROC members in ASGC.

13. 3/27/2007Grid Efforts in Belle13 Belle VO 9 sites Belle software are installed to 3 sites (KEK x2, ASGC) ~60 CPUs 2TB storage MC production ongoing Installation manual ready GFAL with Belle software

14. 3/27/2007Grid Efforts in Belle14 Total Number of Jobs at KEK in 2006 JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-02JP-KEK-CRC-02 200 700 400 1,000 1,400 BelleBelleBelleBelle

15. 3/27/2007Grid Efforts in Belle15 Total CPU Time at KEK in 2006 (Normalized by 1kSI2K) JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-02JP-KEK-CRC-02 4,000 3,000 1,000 [hrs kSI2K] 12,000 10,000 4,000 BelleBelleBelleBelle

16. 3/27/2007Grid Efforts in Belle16 Logical Site Overview KEK Firewall SuperSINETSuperSINET HSMHSM Grid LAN KEK-2202.13.197.0/24KEK-2202.13.197.0/24 KEK-DMZKEK-DMZ MCAT172.22.28.0/24MCAT172.22.28.0/24 130.87.224.0/21SRB172.22.28.0/24130.87.224.0/21SRB172.22.28.0/24 SRB-DSI130.87.104.0/22SRB-DSI130.87.104.0/22 KEK-CC KEK-1130.87.208.0/22KEK-1130.87.208.0/22 $ scp output Belle: $ scp input Grid: Local files CPUs WS

17. 3/27/2007Grid Efforts in Belle17 SRB Introduction Schedule Construction Planning Grid Grid Belle Operation Belle Operation Networking Networking KEKCC/IBM KEKCC/IBM Construction Planning Grid Grid Belle Operation Belle Operation Networking Networking KEKCC/IBM KEKCC/IBM MCATMCAT SRBSRB FWFW SRB-DSISRB-DSI TestTest ConnectionConnection Start Operation Preparation

18. 3/27/2007Grid Efforts in Belle18 Belle Grid Deployment Future Plan Federate with Japanese universities. Federate with Japanese universities. KEK hosts the Belle experiment and behaves as Tier-0. KEK hosts the Belle experiment and behaves as Tier-0. Univ. with reasonable resources: full LCG (Tier-1) Univ. with reasonable resources: full LCG (Tier-1) Univ. without resources: UI Univ. without resources: UI The central services such as VOMS, LFC and FTS are provided by KEK. The central services such as VOMS, LFC and FTS are provided by KEK. KEK also covers web Information and support service. KEK also covers web Information and support service. Grid operation is co- operated with 1~2 staffs in each full LCG site. Grid operation is co- operated with 1~2 staffs in each full LCG site. JP-KEK-CRC-02 JP-KEK-CRC-03 University UI University UI University UI University UI University UI University UI University UI University UI University UI Tier-0 Tier-1 deploy in the future preliminary design

19. 3/27/2007Grid Efforts in Belle19 Summary Belle VO launched Belle software are installed to 3 sites KEK sites are mainly used by Belle MC production ongoing SRB is being introduced

20. 3/27/2007Grid Efforts in Belle20 Additonal (Belle's) Resources We now obtain high-performance computer system; but we didn't suddenly switch to the “less expensive” system. 350TB disks 1.5PB tapes 934 CPUs 20units/20TB We have been testing such system for several years. ● ● Linux based PC clusters ● ● S-ATA disk based RAID drives ● ● S-AIT tape drives 1000TB disks 3.5PB tapes 2280 CPUs B computer for comparison These resources have been essential for Belle (production/analysis)

21. 3/27/2007Grid Efforts in Belle21 Belle Grid Deployment Plan We are planning a 2-phased deployment for BELLE experiments. We are planning a 2-phased deployment for BELLE experiments. Phase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with other VOs. Phase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with other VOs. JP-KEK-CRC-02 consists of “Central Computing System” maintained by IBM corporation. JP-KEK-CRC-02 consists of “Central Computing System” maintained by IBM corporation. Available resources: Available resources:  CPU: 72 processors (opteron), SE: 200TB (with HPSS) Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production System Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production System JP-KEK-CRC-03 uses a part of “B Factory Computer System” resources. JP-KEK-CRC-03 uses a part of “B Factory Computer System” resources. Available resources (maximum estimation) Available resources (maximum estimation)  CPU: 2200 CPU, SE: 1PB (disk), 3.5 PB (HSM) This system will be maintained by CRC and NetOne corporation. This system will be maintained by CRC and NetOne corporation.

22. 3/27/2007Grid Efforts in Belle22 Computing Servers ● ● DELL Power Edge 1855 Xeon 3.6GHz x2 Memory 1GB ● ● Made in Taiwan [Quanta] ● ● WG: 80 servers (for login) Linux (RHEL) ● ● CS: 1128 servers Linux (CentOS) ● ● total: 45662 SPEC CINT 2000 Rate. equivalent to 8.7THz CPU will be increased by x2.5 (i.e. to 110000 SPEC CINT 2000 Rate) in 2009. 1 enclosure = 10 nodes / 7U space 1 rack = 50 nodes

23. 3/27/2007Grid Efforts in Belle23 Storage System (Disk) ● ● Total 1PB with 42 file servers (1.5PB in 2009) ● ● SATAII 500GB disk x ~2000 (~1.8 failure/day ?) ● ● 3 types of RAID (to avoid problems) ● ● HSM = 370 TB non-HSM = 630 TB ADTX ArrayMasStor LP 15drive/3U/7.5TB Nexan SATA Beast 42drive/4U/21TB SystemWorks MASTER RAID B1230 16drive/3U/8TB (made in Taiwan)

24. 3/27/2007Grid Efforts in Belle24 Storage System (Tape) ● ● Backup ● ● 90TB + 12drv + 3srv ● ● LTO3 400GB/volume ● ● NetVault ● ● HSM: PetaSite (SONY) ● ● 3.5PB + 60drv + 13srv ● ● SAIT 500GB/volume ● ● 30MB/s drive ● ● Petaserve