1. CHEP03 P5, 3/24/03, R.Itoh 1 Upgrade of Belle DAQ System Ryosuke Itoh, KEK representing Belle DAQ group 1. Introduction 2. Requirements 3. Design 4. Software Trigger and Event Reduction 5. Summary

2. CHEP03 P5, 3/24/03, R.Itoh 1 1. Introduction Belle Experiment : B-factory experiment at KEK in Japan - Study of CP violation in B meson system - Precise measurement of CKM matrix element 11 22 33 Unitarity Triangle V cd V cb * V ud V ub * V td V tb *    requires very high statistics of B meson decays Latest result: sin2  1 =0.82±0.12±0.05 (ICHEP02) (obtained with 78/fb = 85M BB)

3. CHEP03 P5, 3/24/03, R.Itoh 1 Peak luminosity : 8.3  10 33 /cm 2 /sec Recorded luminosity / day : 457.8/pb Total recorded luminosity : >120/fb - all world records!!

4. CHEP03 P5, 3/24/03, R.Itoh 1 However, we need more luminosity for * precise determination of     and V ub * search for new physics in rare B decays through penguin diagram need > 3000/fb (cf. ~120/fb up to now, took 3 years) Drastic upgrade of accelerator (KEKB) is necessary : Super KEKB Target Luminosity : >10 35 /cm 2 /sec (current maximum: ~10 34 ) Upgrade of Detector and DAQ is necessary writing LoI aiming at the upgrade in 2006

5. CHEP03 P5, 3/24/03, R.Itoh 1 2. Requirements Physics trigger rate 100Hz 1KHz Maximum trigger rate 500Hz 10(-30)KHz Event size at L1 40KB/ev 200-300KB/ev Data flow rate at L1 20MB/s > 2 GB/sec (>5GB/sec w/o SVD trg.) Data flow at storage 10MB/s 240MB/sec Reduction factor in DAQ 2 8-40 Belle SuperKEKB(10 35 ) How to achieve higher event reduction factor  Key of DAQ upgrade

6. CHEP03 P5, 3/24/03, R.Itoh 1 Upgrade Strategy Pipeline based readout is essential  Gate and Delay method(Belle)  deadtime = 100% at 10KHz Use of common electronics as possible  CoPPER (Common readout platform) * unified handling of pipeline readout using on-board PC module * detail will be covered in tomorrow's P5 session (Higuchi) and poster session (Igarashi) Scalability  to keep up with gradual increase in luminosity

7. CHEP03 P5, 3/24/03, R.Itoh 1 Expected event size at L1 Pixel - ~100KB SVD 15KB ~30KB CDC 6KB ~10KB PID 3KB(TOF/ACC) ~20KB ECL 8KB ~100KB KLM 3KB ~3KB TRG/others 3KB ~3KB Belle SuperKEKB * Pixel: event size compression possible * ECL : wave form sampling to obtain required resolution (~10 buckets/hit*12bit) -> can be reduced to 1/5 by feature extraction * Other: event size compression using word-packing/"zip" Event Processing on CoPPER ~40KB ~300KB ~100KB/ev

8. CHEP03 P5, 3/24/03, R.Itoh 1 Current Belle's Event Builder 3. Design..... ~1000 CoPPERs ~50 Readout PCs..... ~10 Event Building Farms ~10 L3 Farms L2.5 L2/size reduction L3 ~1GB/sec ~500MB/sec ~250MB/sec 200-300KB/ev -> 100KB/ev Transfer Network mass storage

9. CHEP03 P5, 3/24/03, R.Itoh 1 Event Building Current system at Belle : switchless event building farm - based on point-to-point 100base-TX/GbE connection - working very stably in current experimental condition - Use this system as a "unit" - Have multiple units operated in parallel

10. CHEP03 P5, 3/24/03, R.Itoh 1 Event transfer network (~10 100base-TX ports + 1000base-T port) CDC 1 CDC 2 CDC Event Builder Units (up to ~10).... (up to ~10) 1000base-T 100base-TX Network Switch CDC 1000base-T * Belle's event builder assumes event fragment from one detector is fed into one NIC on layer 1 PC (1 readout subsystem/detector at Belle  p2p) * Upgraded system : event fragment from one detector is provided from many readout PC's ( up to ~10 /detector) No big network switch  cost effective

11. CHEP03 P5, 3/24/03, R.Itoh 1 ~30 servers GbE FE Switch PC server (2~4 CPUs; P4@4GHz) Reconstruction Farm More processing power is required to have more reduction  Real-time event reconstruction for L3 trigger layer1 layer2 layer3/ input distributor ~ 65MB/sec ~125MB/sec/unit 100KB/ev 1.25KHz/unit 100KB/ev ~600Hz 5MB/sec/node ~3MB/sec/node (inc. DST data) 1 Unit  processing power for L=10 34 /cm 2 /sec

12. CHEP03 P5, 3/24/03, R.Itoh 1 Belle Event Building Farm 28 x 1U 2 Athron MP servers 2xP3 + 80GB*2 2xP3 + 80GB*2 GbE-SX 3com Switch 4400 100base- TX contol PC DAQnet GbE-LX Planex FMG-226SX 3com Switch 4400 GbE-SX placed in “server room” Comp. Center works as disk/memory cache development in progress Test Bench at Belle

13. CHEP03 P5, 3/24/03, R.Itoh 1 Storage Belle: Currently using high speed tape device w/ robot(DTF/PetaSite) * SONY gave up to release faster DTF drives * market is small  expensive - Recent disks are much faster than tape drive * ex. Dell/EMC CX600 / Fujitsu ETERNUS - 200MB/sec (2Gbps FiberChannel I/F) * preliminary test of prototype (borrowed from some company) shows >70MB/sec read/write speed using 1Gbps FC cf. DTF : 24MB/sec Record data on disk directly. Parallel data streams.  R&D has been started with Computing people. - will be tested in Belle Environment in next FY

14. CHEP03 P5, 3/24/03, R.Itoh 1 4. Software trigger / Event reduction 1) Level 2 trigger (on CoPPER modules) - event trigger after pipeline readout - trigger signal is genarated by dedicated hardware (ex. SVD trg) -> latency < ~50  sec (cf. L1 latency : ~10  sec) - trigger signal is distributed to CoPPER via timing logic with event tag - software running on CoPPER CPU rejects the event by looking at the trigger event tag - Event Reduction is very important in high-intensity experiments to keep mass-storage manageable. - We need a versatile and powerful software trigger /event size reduction scheme to obtain reduction factor of <1/10 after L1 trigger. Trigger rate reduction : ~1/3-1/5 (30~50KHz -> 10KHz) Event size reduction : 1 (~200-300 KB)

15. CHEP03 P5, 3/24/03, R.Itoh 1 2) Event processing on CoPPER/Readout PC - Software Data sparcification * Feature extraction for wave-form sampling * Event size compression by various method (bit-squeezing, zip, etc.) - Raw Data Data Formatting (to Panther / ROOT I/O (?)) Trigger rate reduction : 1 (10KHz) Event size reduction : 1/3 (~300 KB->100KB) CoPPER : linux-operated PC on board  possibility of versatile event data processing

16. CHEP03 P5, 3/24/03, R.Itoh 1 3) Level 2.5 trigger - Software trigger using partially-built event data (data from one subdetector/several related subdetectors) - Current Belle's "L3" scheme can be used Trigger rate reduction : 1/2 (10KHz->5KHz) Event size reduction : 1 * Fast Tracking + Hardware trigger information (Belle)

17. CHEP03 P5, 3/24/03, R.Itoh 1 Power of event reduction by "physics skim" at Belle Fraction in events after L2.5 Hadronic 14.2%     /2photon 9.6% Monitor(=e + e -,    ,etc) ~1% (can be scaled) Trigger rate reduction : 1/4 (~2KHz) Event size reduction : 1 (+ reconstruction info(~100KB/ev)) * Data flow rate will increase by a factor of 2 if we leave reconst. info together on storage  requires more multiplicity in storage 4) Level 3 trigger - Software trigger using fully-built and fully-reconstructed data - Trigger at a level of "Physics Skim" * hadronic event selection * selection of specially-interested events

18. CHEP03 P5, 3/24/03, R.Itoh 1 5. Summary a Upgrade of B-factory at KEK (KEKB) is being planned to achieve >10 times luminosity increase hopefully in 2006. Design of new DAQ system to cope with >10KHz trigger rate with an event size of 300KB is in progress based on the system currently used at Belle. The key issue in the design is the reduction of data flow at storage less than a factor of 1/10 of that at L1. The reduction is feasible by the use of widely-distributed processing from readout modules to reconstruction farm. - Other R&D's (timing-distribution, data monitoring, etc.) are also going on. Stay Tuned!

19. CHEP03 P5, 3/24/03, R.Itoh 1 Backup Slides

20. CHEP03 P5, 3/24/03, R.Itoh 1 Detector Electronics Quick Summary - SVD : CMS APV25 chip -> promising! - Pixel : candidate = ALICE/BTeV chip too slow! large data size : 4KB(binary)/30KB(8bit analog) + 90KB position - CDC : 3 approaches 1) ADC with waveform sampling (10bit@200MHz) 2) TDC only with charge to time conversion (ASDQ) 3) TDC + FADC (TMC+12bitFADC@20MHz) - ECL : wave form sampling needed to avoid pileup effect (12bit FADC@2MHz for barrel, 20MHz(?) for pure CsI) - TOP/RICH : need to manage pixel photo-detector * time stretcher(Varner)/AMT(Arai), analog pipeline(Ikeda) - KLM : readout scheme is not so much different from Belle's regardless of choice of detection device (RPC/Sci. Tile) * TDC based multiplexing + on-board data compression all electronics will be equipped as "FINESSE" implemented as daughter board on CoPPER.

21. CHEP03 P5, 3/24/03, R.Itoh 1 Pipeline and CoPPER L1 pipeline ADC TDC Free running clock Gate Trigger FIFO CPU Network/Serial bus “FINESSE” “CoPPER” Common Readout Platform This part is supplied by each subdetector group as "daughter cards" * Parallel Session 5 Tuesday: 15:00- * Poster Session

22. CHEP03 P5, 3/24/03, R.Itoh 1 Readout PC - Linux-operated PC with NIC/Serial-bus - Collects event fragments from 1 CoPPER crate (containing ~20 CoPPER boards) - CoPPER : PCI (PMC) based CPU board is built-in -> various choice in the connection to Readout PC Choice : 100BaseTX, GbE, USB2.0, IEEE1394, etc. * possible choice - point-to-point connection using 100Base-TX (no switch) - multi-port 100Base-TX NIC on PC (4 ports x 5 cards) <- we have experiences in current event building farm - Sub-eventbuilding is performed on Readout PC

23. CHEP03 P5, 3/24/03, R.Itoh 1 DAQ upgrade schedule ● Target year of SuperKEKB upgrade : 2006 ● 1 full prototype (CoPPER to storage) : 2004 200220032004 2005 2006 Copper prototype full testbench by end of FY2003 Copper crate readout TDC/ADC FINESSE reference prototype of subdet. FINESSE test in a crate TTRX prototype system test Recon Farm/Storage prototype Recon farm operation in Belle backend testbench mass production (LoI/fall) (MU06) EFC with CoPPER readout CDC with CoPPER full upgrade SEQ/TTD

24. CHEP03 P5, 3/24/03, R.Itoh 1