1. Takeo Higuchi Institute of Particle and Nuclear Studies, KEK; On behalf of COPPER working group Jan.20, 2004 Hawaii, USA Super B Factory Workshop Readout Electronics for the SuperKEKB

2. Contents Introduction Conceptual design The COPPER-II Readout Scheme COPPER-II Performance Schedule Summary

3. Requirements for Readout Electronics Trigger System Readout Elec. Event Builder Storage Detector DAQ Trigger rate > 10 kHz Data size ~ 300 kB/ev Readout ~ 50 k channels Trigger rate > 10 kHz Data size ~ 300 kB/ev Readout ~ 50 k channels SuperKEKB Also good for J-PARC, K2K …

4. Conceptual Design What are keys in designing the readout system?

5. System Design Event buffers for pipelined DAQ Data size reduction by processor on data path Processor Readout FIFO Digitization Data path Consequent design of readout electronics from detector to EB

6. Design from Cost View Flexible system – Common platform + user-defined ADC/TDC. – Implementation of ADC/TDC and CPU as modules. Compact design – Less #-of-crates to house platform boards. – Smaller room of electronics hut. Commodity usage – Good products at low price in computer market. – Easy to follow/import newest technology.

7. Design Overview PMC CPU local bus PCI bus Detector signals mezzanine (add-on) module Network IF Trigger Module ADC/TDC Module Bridge PMC FIFO

8. Conceptual Design (Front-end I/F) User-defined ADC/TDC module sampling clock Detector signal 0xaa55 0x0246 0xf3b7 … Readout FIFO ADC/TDC L1 trigger pipeline trigger signal Add-on type Module flexibility compactness

9. Conceptual Design (L1 Trigger Pipeline) Digitized data in Data out (to Readout FIFO) Always WRITE busy Data READ Trigger HOLD switch Data READ Always WRITE Data READ switch EMPTY L1 triggerReady pipeline

10. Conceptual Design (FIFO) Detector Local bus Bridge PCI bus ADC/TDC Module trigger signal DMA 32 pipeline

11. Conceptual Design (Back-end) Data path = PCI bus PCI mezzanine modules (PMC) Bridge CPU Trigger Network EB PCI bus Data suppression DMA PCI = Up to 133 MB/s trigger signal Local bus online data reduction flexibility commodity use compactness

12. Conceptual Design (Processor) Modular CPU – Modularity = easy to upgrade. – Intel x86 series: most familiar architecture. Operating System = Linux – Easy to develop device driver / software. – Low price. flexibility commodity use

13. PCI Ethernet card PMC Ethernet card What’s ‘PMC’? - PCI Mezzanine Card Standard –PMC is 100% compliant with the PCI. –PMC is suitable for high density applications. –Many commercial products are available: Ethernet cards, GbE cards, memory modules, CPUs, etc. 74×149 mm 2

14. PMC Size CPU RadiSys EPC-6315 – Equipped with Intel PentiumIII 800 MHz. – RedHat Linux 7.3 runs. – 32-bit 33/66 MHz PCI bus interface. You can login this CPU like your PC Even physics-analysis codes run on it! Contains Linux image

15. Whole Design of the DAQ Platform PMC CPU local bus PCI bus Detector signals mezzanine (add-on) module Network IF Trigger Module VME 9U sized board ADC/TDC Module Bridge PMC FIFO

16. The COPPER-II This is our readout platform for the Super B factory.

17. The COPPER-II PMC Processor Trigger Generic PMC slot ADC/TDC VME-9U sized board 4 ADC/TDC slots 1 MB × 4 FIFO 32-bit local bus 3 PMC slots 32-bit 33MHz PCI bus Local-PCI bridge VME I/F 4 ADC/TDC slots 1 MB × 4 FIFO 32-bit local bus 3 PMC slots 32-bit 33MHz PCI bus Local-PCI bridge VME I/F COmmon Pipelined Platform for Electronics Readout On-board Ether

18. ADC/TDC Module Local bus I/F (for control) Digitized signal output Trigger busy Clock, L1 trigger Clock = 43MHz, Reduced clock = depends on sub-system 32 Photo of prototype card 76×180 mm 2

19. Readout Scheme How it works?

20. Readout Scheme (1) L1 Trigger comes… – Trigger module distributes trigger signal to the ADC/TDC modules. – Digitized data are held on the ADC/TDC module. – Data transfer to the COPPER-II starts. COPPER-II FIFOs are filled… – The COPPER-II’s FPGA formats data (header and footer). – The COPPER-II’s FPGA counts each event size in FIFOs. – A PCI interrupt is generated when a total data size in the FIFOs exceed threshold to initiate the DMA.

21. Readout Scheme (2) PCI interrupt is generated… – The CPU checks the size of stored data counted by the COPPER-II. – The CPU starts data transfer from the FIFOs to CPU’s main memory using DMA. A user process on the CPU is woke up after DMA… – The user process reads data and performs the data size reduction. The data are transferred to the event builder…

22. COPPER-II Performance Up to what input rate does COPPER-II work?

23. Study Setup ADC proto. Vacant Trigger CPU Ether RX The COPPER-II Generates virtual ‘ADC’ data. Data size / event is variable. Self trigger. Trigger rate is variable.

24. Maximum Trigger Rate Typical trigger rate Required trigger rate @ 416 bytes/ev/ADC-module The COPPER-II works > 30 kHz input rate Belle’s TDC data 16 bytes / ch. (drift chamber) 400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module. Belle’s TDC data 16 bytes / ch. (drift chamber) 400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module.

25. Profile of CPU Usage User time: ~2% System time: ~20% Idle time: ~78% = CPU power that is equivalent to P3 ~600MHz is still available Large idle time fraction indicates the PCI bus works at the full performance. – 416 bytes / ADC-module / ev × 40 kHz × 4 ADC modules = 67 MB/s. @ 416 bytes/ev/ADC-module

26. Performance Degradation by Network Use @ 416 bytes/ev/ADC-module CPU FIFO Read FIFO Read FIFO Read FIFO Read Network Transfer proc.A proc.B RX Ethernet 11MB/s CPU user time: ~2% CPU system time: ~20% CPU idle time: ~78% Maximum accepted trigger rate: 40 kHz CPU user time: ~2% CPU system time: ~20% CPU idle time: ~78% Maximum accepted trigger rate: 40 kHz CPU user time: ~5% CPU system time: ~31% CPU idle time: ~64% Maximum accepted trigger rate: 32 kHz CPU user time: ~5% CPU system time: ~31% CPU idle time: ~64% Maximum accepted trigger rate: 32 kHz Still works well.

27. Schedule What are we planning?

28. Schedule in Near Future The COPPER-II prototype is being debugged. We start production of the COPPER-II from the begging of Feb.2004. We are planning to replace some Belle-DAQ parts with COPPER in 2005 summer.

29. Long Term Plan COPPER Proto. Upgrade Integrated Test Goal Performance Study ASIC Development ADC Module Design TDC Module Design Mass CPU Control User I/F Framework Event Building Farm L2/L3 Triggers PlatformHardware Applications Software Applications ADC/TDC Proto. We are here.

30. Summary We have developed new readout electronics, COPPER-II, for the Super B factory. Design keywords are: Pipeline, Online data reduction, Flexibility, Compactness, and Commodity usage. At the typical data size, the COPPER-II works up to 40 kHz trigger rate, which satisfies the requirements.

31. Backup Slides Please help me…

32. Introduction What is a readout electronics? What is Super B factory requiring for it?

33. Introduction to Readout Electronics Trigger System Readout Elec. Event Builder Storage Detector Role Signal digitization Event buffering Data suppression Data transfer to EB Role Signal digitization Event buffering Data suppression Data transfer to EB DAQ

34. Keywords in the System Design Five keywords, again… pipeline online data reduction flexibility compactness commodity use

35. Add-on-structure module FIFO chip Bridge chip PMC size CPU PMC size network interface All Components on a 9U-Size Board ADC/TDC Module Bridge CPU Network compactness

36. Level 1 Trigger Trigger Module ADC/TDC Module trigger busy FINESSE TTRX TRIGGER from the trigger module. BUSY response by ADC/TDC module. Data hold. Data-transfer to FIFO.

37. ADC/TDC Module  Readout FIFO Data transfer to readout FIFO – 32-bit FIFO per ADC/TDC module. – Data size of each event is counted by FPGA and is stored to FIFO-word-counter FIFO. ADC/TDC Module Readout FIFO (1 MB) 32-bit data FIFO WENA FIFO WCLK spied FIFO-word- counter FIFO-word- counter FIFO-word-counter FIFO To CPU FPGA

38. Readout FIFO  CPU: DMA Local-PCI Bridge Local-PCI Bridge CPU PCI bus RadiSys EPC-6315PLX-9054 Local bus Readout FIFO FIFO-word- counter FIFO Readout FIFO filled PCI interrupt Check event size Initiate DMA DMA data transfer DMA over PCI interrupt

39. COPPER Device Driver interrupt_handler() do_DMA() while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size; }; start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size ); while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size; }; start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size ); interrupt_handler() PCI interrupt Readout FIFO ready (every event) PCI interrupt DMA over DMA buffer user_read() do_DMA() Read new data (if exist) after last do_DMA(). Return to user code. Read all available data in FIFO (mostly one event)

40. CPU  Event Builder EB Ethernet CPU Network PCI bus serial line for debug – Are event-header/event-footer correct? – Are data contents consistent with pre-defined ones? – Generator module’s event counters = event tag from trigger module? – Are all event counters from 4 generator modules consistent? – Do event counters increase by 1 correctly? – Are event-header/event-footer correct? – Are data contents consistent with pre-defined ones? – Generator module’s event counters = event tag from trigger module? – Are all event counters from 4 generator modules consistent? – Do event counters increase by 1 correctly?

41. Maximum Trigger Rate vs. Data Size Typical data size

42. Data Transfer Speed on PCI vs. Data Size Typical data size