1. Ichiro Adachi ACAT03, 2003.Dec.021 Ichiro Adachi KEK representing for computing & DST/MC production group ACAT03, KEK, 2003.Dec.02 Belle Computing System

2. Ichiro Adachi ACAT03, 2003.Dec.022 Outline Introduction Software Computing Model DST/MC Processing Data Management Plan & Summary

3. Ichiro Adachi ACAT03, 2003.Dec.023 Introduction

4. Ichiro Adachi ACAT03, 2003.Dec.024 Belle Experiment B-factory experiment at KEK KEKB ring Asymmetric energy e+e- collider: KEKB Explore CP violation and flavor physics in B mesons Belle detector

5. Ichiro Adachi ACAT03, 2003.Dec.025 Belle Detector Silicon Vertex Detector 3 layers of DSSD for vertexing Central Drift Chamber tracking & dE/dx ToF counter P-ID Aerogel Chrenkov Counter π/K seperation CsI(Tl) Calorimeter photon and electrons KLM muon & KL catcher Superconduncting solenoid of 1.5T

6. Ichiro Adachi ACAT03, 2003.Dec.026 Belle’s Achievements 2001: Observation of large CP violation in B meson system 2002: Evidence of CP-violating asymmetries in B 0  π + π - 2003: Indication of new physics from B 0  Ks computing environment Network Storage system CPU power Management lots of B mesons

7. Ichiro Adachi ACAT03, 2003.Dec.027 More data KEKB has achieved design luminosity of 10 34 cm -2 sec -1 –10 B meson pairs/sec Accumulated more than 160fb -1 of data so far Largest B meson data sample at  energy region 160fb -1

8. Ichiro Adachi ACAT03, 2003.Dec.028 Basic Requirements Beam data should be available for physics analyses in a couple of months – Software updates can be reflected onto physics analysis immediately – Physics outputs in a timely manner MC sample at least 3 times larger than beam data – Analysis technique gets matured with a large statistics of beam data – Systematic study needs more statistics in MC sample

9. Ichiro Adachi ACAT03, 2003.Dec.029 Basic Requirements(cont’d) Software –stable & robust Computing Model –efficient –expandable Performance –data availability for analyses

10. Ichiro Adachi ACAT03, 2003.Dec.0210 Software

11. Ichiro Adachi ACAT03, 2003.Dec.0211 Software Tools Input with panther Output with panther unpacking calibration tracking vertexing clustering particle ID diagnosis B.A.S.F. Event flow module loaded dynamically shared object Home-made kits –“B.A.S.F.” for framework Belle AnalySis Framework unique framework for any step of event processing event-by-event parallel processing on SMP –“Panther” for I/O package unique data format from DAQ to user analysis bank system with zlib compression –reconstruction & simulation library written in C++ Other utilities –CERNLIB/CLHEP… –PostgreSQL for database

12. Ichiro Adachi ACAT03, 2003.Dec.0212 Computing Model

13. Ichiro Adachi ACAT03, 2003.Dec.0213 Super-SINET to Univ’s 2002 Computing Model Overview Fast network (1~10Gbps) 4Gbps KEKB computers from 2001 Feb Disk Storage (60TB) 2003 Mar User PC farms 10 login servers for User PC Farms 2003 Jun

14. Ichiro Adachi ACAT03, 2003.Dec.0214 KEKB Computers University resources User analysis & storage system Computing network for batch jobs and DST/MC production Tokyo Nagoya Tohoku super-sinet 1Gbps Sun computing server 500MHz*4 GbE switch Compaq 38 hosts online tape server PC farms tape library 500TB Fujitsu HSM server HSM library 120TB disk 4TB file server 8TB 500MHz*4 9 hosts user PC 1GHz 100hosts work group server GbE switch GbE switch

15. Ichiro Adachi ACAT03, 2003.Dec.0215 PC farms -heterogeneous system from various vendors -cost effectiveness -3 types of CPU(Pen3/Xeon/Athlon) Compaq 60PC’s 0.7GHz@Xeon Fujitsu 127PC’s 1.26GHz@P3 NEC 84PC’s 2.8GHz@Xeon processor clock speed 19992002 Dell 36PC’s 0.5GHz@P3 20012003 Appro 113PC’s “1.67GHz”@Athlon

16. Ichiro Adachi ACAT03, 2003.Dec.0216 CPU & Disk Storage Sun CPU –9 servers(0.5GHz*4CPU) –38 computing servers(ibid.) tape drives(2 each for 20hosts) Linux CPU –60 Compaq servers(Intel Xeon, 0.7GHz*4CPU) –127 Fujitsu servers(P3, 1.26GHz dual) –113 Appro servers(Athlon, 1.67GHz dual) –84 NEC servers(P3, 2.8GHz dual) Disk servers & storage -Tape library DTF2 tape(200GB), 24MB/s IO 500TB total 40 tape drives –8TB NFS file servers –120TB HSM servers 4TB staging disk –2 servers for 60TB disk

17. Ichiro Adachi ACAT03, 2003.Dec.0217 User PC farm & Disk Storage Login servers PC farm LSF scheduler Disk storage public beam data MC data debugging user code job data notice Local disk 6TB user data histograms 84 PC’s with dual Xeon 2.8GHz CPUs CPU utilization

18. Ichiro Adachi ACAT03, 2003.Dec.0218 Super-SINET at Belle Disks located at Nagoya (~350km away from KEK) are NFS-mounted to the KEK host Directly write data onto such disks from batch jobs running at KEK computer superSINET also used for copying a various type of data hadronic sample J/  inclusive bsbs D*sD*s full recon KEK Nagoya 350km

19. Ichiro Adachi ACAT03, 2003.Dec.0219 DST/MC Processing

20. Ichiro Adachi ACAT03, 2003.Dec.0220 DST Production & Skimming Scheme PC farm raw data DST data disk DST data histograms log files disk or HSM skims such as hadronic data sample Sun disk data transfer Sun 1. Production(reproduction) 2. Skimming histograms log files user analysis

21. Ichiro Adachi ACAT03, 2003.Dec.0221 Processing Power & Failure Rate Processing power –Processing ~1fb -1 per day with 180GHz Allocate 40 PC hosts(0.7GHzx4CPU) for daily production to catch up with DAQ –2.5fb -1 per day possible Processing speed(in case of MC) with 1GHz one CPU –Reconstruction: 3.4sec –Geant simulation: 2.3sec Failure rate for one B meson pair module crashnegligible tape I/O error1% process communication error3% network trouble/system errornegligible

22. Ichiro Adachi ACAT03, 2003.Dec.0222 Reprocessing Snapshot L processed /day(pb-1) exp25 exp27 gap: waiting for constants

23. Ichiro Adachi ACAT03, 2003.Dec.0223 Performance:beam data processing All data including a final bit of beam data have been always processed and been used for analyses 2001 summer 30fb -1 2.5months 3months 2002 summer 78fb -1 2003 summer 159fb -1

24. Ichiro Adachi ACAT03, 2003.Dec.0224 MC Production MC sample –3 times bigger statistics –Run dependence taken into account beam data file Run# xxx B 0 MC data B + B - MC data charm MC data min. set of generic MC Run# xxx light quark MC run-dependent background IP profile 3 files

25. Ichiro Adachi ACAT03, 2003.Dec.0225 MC Production(cont’d) PC farms at KEK shared with DST processing Switching to MC production can be made easily MC samples for 159 fb -1 has been completed in November 2003 2002 2003 2.2 billion events Library major update Library minor update Library minor update

26. Ichiro Adachi ACAT03, 2003.Dec.0226 MC Production at Remote Sites Total CPU resources at remote sites amounts to ~600GHz 14% of MC events has been produced outside KEK All data have been transferred to KEK via network 14% at remote sites ~600GHz GHz

27. Ichiro Adachi ACAT03, 2003.Dec.0227 Data Management

28. Ichiro Adachi ACAT03, 2003.Dec.0228 Data Management 20K files for beam runs 240K files for run-dependent MC data User has to go through those to get final results Web based interface File information are stored in postgreSQL database ~”meta data” SQL database command in batch job access answer inquire data files inquireread job submit user

29. Ichiro Adachi ACAT03, 2003.Dec.0229 Data Management(cont’d) File info centralized and uniquely managed Easy to change if necessary –Disk failure etc SQL database trouble administrator

30. Ichiro Adachi ACAT03, 2003.Dec.0230 Plan & Summary

31. Ichiro Adachi ACAT03, 2003.Dec.0231 Software Plan for 2004 Belle detector upgraded this summer –Silicon vertex detector 4 layers( from 3 layers ) DSSD smaller inner radius & wider acceptance –New inner chamber Cathode part replaced into new chamber –Real-time processing Refer to talk by Itoh san(Dec/4 session2) Need to update reconstruction software –Calibration constants newly determined –Tuning is underway Reprocess Belle phase-I data of 159fb -1 –Under discussion

32. Ichiro Adachi ACAT03, 2003.Dec.0232 Prospects We are in the stage of O(PB) scale –More than 2500 DTF2 tapes –Will record another 100fb -1 by 2004 summer –Obviously increase data –Super B-factory project? Data storage as well as management can be a big issue

33. Ichiro Adachi ACAT03, 2003.Dec.0233 Summary Our computing system has been working well –Processing beam data as well as MC data have been successfully done –Proven up to 160fb -1

34. Ichiro Adachi ACAT03, 2003.Dec.0234 Data size Raw data35KB/event DST data58KB/event mini-DST data 12KB/event Total raw data ~120TB for 120fb -1 1000 DTF2(200GB) tapes for hadronic event