1. Data Acquisition of the PXD Takeo Higuchi (KEK) PXD follow up meeting on Nov.7,2010 « DISCLAIMER» Materials prepared for and presented in PXD follow up meeting on Nov.7 th,2010 May be slightly behind the latest.

2. Contents Overview of data acquisition (DAQ) system Data handling hybrid (DHH) ATCA readout option PC readout option High level trigger (HLT) system Schedule Summary

3. Belle II DAQ Overview Illustration by M.Nakao PXD DAQ options ATCA board + self tracking:Challenge ATCA board + HLT tracking:Baseline PC readout + HLT tracking:Backup

4. Requirements to the PXD DAQ Occupancy2% Data size / event800 kB Maximum L1 trigger rate30 kHz Data are sent out through 40 DHH links Throughout / DHH link600 MB/s Trivially the occupancy depends on the BG. But as a preliminary number to start the design, we agreed to use occupancy=2%.

5. Design of the PXD DAQ PXD DAQ coverage PXD coverage DAQ coverage

6. DHH [1] Functions of data handling hybrid (DHH) – Main data stream Receive 4 data blocks from PXD. Receive global system clock. Receive the level-1 trigger decision. Provide the level1 trigger pipeline (4x1GB/s) Output the data to the ATCA/PC via an optical serial link (AURORA/RocketIO). – Peripherals Handle slow control signals from the CDAQ. Distribute JTAG signals. K.Igor (MUT) et al.

7. DHH [2] PXD  DHH 30-50 cm Kapton Flat cable 10 m Patch Panel No radiation worry DHH Passive ? Active I: signal conditioner for serial links Active II: radiation hard laser drivers and optical transceivers

8. DHH [3] DHH Trigger timing system FPGA USB, Ethernet RJ45 ATCA readout board (or PC) Power Supplies DHH Controller DHH  PXD DAQ (option A) – Single communication port with the TT system.

9. DHH [4] DHH ATCA readout board (or PC) Power Supplies FPGA SFP RJ45 Ladder’s flat cable connector PIGGY-BACK Power Module POWER CONNECTOR JTAG Slow Control Trigger timing signals Slow control Trigger timing system directly communicates with the DHH. DHH  PXD DAQ (option B) – Less variations of the electronics. Option choice will be made after a prototype DHH becomes ready.

10. DHH [5] Intelligent level-1 pipeline – In case of the second trigger during the DHP output, the DHP stretches the DHP output frame and inserts the second header to it. Trigger DHP trigger one frame HEADER TRAILER

11. ATCA Readout [1] Functions of ATCA board – Receive the PXD data from DHH over the AURORA/RocketIO through up to 8 SFP+/SFP connectors. – Perform sub-event building. – Receive region-of-interest information on the PXD (i.e. track-associated region) from the HLT over the Ethernet. – Hold the PXD data for average HLT latency (5secs at most). – Perform data size reduction using the RoI information by an onboard-FPGA. – Output the final data to the global event builder PC. S.Lange (Gießen), Z.Liu (IHEP) et al.

12. ATCA Readout [2] Snapshot of the ATCA board ATCA full mesh neighbor link x5 GbE x8 SFP x10 flash x4 processor FPGA Switch FPGA

13. ATCA Readout [3] Block diagram

14. ATCA Readout [4] System test / performance study – Throughput. Data generator  optical fiber  SFP  onboard DDR  PC ~150 hours, ~25 MB/min (limited by PC), 0 error. No packet loss at ~1.6Gbps of optical link. – GbE network. ~0.25Gbps (external PC  FPGA). Ethernet on PowerPC on the onboard FPGA. – P2P communication on the ATCA backplane. ATCA slot #1  slot #14; the longest line in full mesh backplane Only few jitter and overshoots are observed.

15. ATCA Readout [5] Grand challenge: RoI finding by the onboard-FPGA – Study on data size reduction algorithm – Heller (MPI) The algorithm utilizes Hough transformation of SVD hits and wedge-shape-sectored detector volume. Tests with physics background look very promising. 99 % track finding efficiency down to 300 MeV/c is realized. Achieved average reduction factor is 10. – Study on Helix finding by FPGA – Münchow (Gießen) Algorithm is tuned to fit in the FPGA: use of lookup table of trigonometric functions, no recursive macro call etc. Fast Hough transformation algorithm is implemented on FPGA simulator: 2 (10) clock cycles per transformation for 8x8 (64x64) Hough plane bins.

16. ATCA Readout [6] R&D of new ATCA board – Because of the requirements for DDR memory 4GB/FPGA(20GB/CN) for the HLT latency. 6 Gbps/FPGA optical link. – Design: motherboard + 4 x daughter cards Whether the requirements were really required was discussed in the WS. Vertex5 2x2GB DDR2 2x3Gbps PCB layouts of the daughter card are ready.

17. PC Readout [1] Data flow in case of PC option RocketIO  PCIe card for PXD data RX is one of the key R&D issues in this option. TH et al.

18. PC Readout [2] Rough sketch of the RocketIO  PCIe card Xilinx FPGA XC5VFX70T-2? Xilinx FPGA XC5VFX70T-2? Clocking Crystal (312MHz) Buffer Optical link x8 PCIe (Gen1) x4 PCIe (Gen2) >6.25Gbps Buffer full indicating signal LVDS/RJ45 AURORA on RocketIO

19. PC Readout [3] Pre-study of the PC readout option – Verify that the data transfer speed from the SFP+ through the PCIe server via an FPGA exceeds 6.25Gbps (before the 8b10b encoding) using a prototype RocketIO  PCIe card. – Measure the CPU load of the PCIe server at when it receives the data at 6.25Gbps. – We are surveying the company to make contract. We pay mainly for the R&D of the FPGA firmware to interface RocketIO and the PCIe.

20. Hardware platform of the pre-study – EK-V6-ML605-G-J: \273,000. Virtex6 FPGA evaluation kit provided by Xilinx. FPGA = XC6VLX240T. Capability of PCIe Gen2 (x4). x2 FMC connectors interfacing to optional daughter card. – TD-BD-FMC-OPT4BOARD: \500,000. FMC daughter card with x4 SFP+ (AVAGO). PC Readout [4]

21. Hardware configuration of the pre-study PC Readout [5] Vertex6 LX240T PCIe FMC ML605 TD-BD -FMC................................ Loopback optical link PCIe server Virtex-6’s internal memory  FMC  daughter card’s SFP+ (TX-only)  optical cable  loopback  daughter card’s SFP+ (RX-only)  FMC  Virtex-6’s internal memory (for PCIe block buffer)  PCIe on ML605 (out)  PCIe server (in)  TX/RX data comparison  discarded Data path

22. PC Readout [6] Firmware/software for the pre-study – IP core for PCIe TX/RX and DMA controller “SYPCIE” … product of Japanese IP vendor (SYSTEC). Capable of PCIe Gen2 (x4) in combination with Virtex6. Free of charge (for evaluation). – Free-of-charge version runs for up to 120mins. – The IP is re-enabled by cold start of the FPGA. – PCIe device driver SYSTEC’s device driver provided as a compiled object. Free of charge (for evaluation).

23. HLT [1] R.Itoh et al.

24. HLT [2] Claimed HLT latency – 5 seconds have been claimed so far, which corresponds to 3GB/DHH at 30kHz level-1 rate for HLT pipeline. 600MB/s/DHH x 5secs = 3GB/s – 3GB/DHH is one of bottlenecks to take the ATCA option. Do we really need such a huge memory? Estimation from the Belle RFARM Processing-time distribution taken from 5k events of luminosity run (exp.57). Most of events take «5 seconds. Average latency is ~500ms. Requirement on the HLT pipeline size can be relaxed to 300MB/DHH. (then, 2GB/DHH is well sufficient.)

25. HLT [3] HLT  RoI matching

26. HLT [4] HLT output derandomizer – The HLT output is not event-number sequential. The HLT latency depends on the event. As soon as the HLT decision made, the HLT farm outputs the RoI information. – In the TDR, event sequence derandomizer is required, but after detailed investigation, we find it is not needed. As far as the manager of the HLT latency buffer in the ATCA/PC is intelligent.

27. HLT [5] Event building as randomized event order

28. Short Term Schedule Next PXD DAQ workshop [1] – Date and place Around the end of April, 2011 before DEPFET meeting; in Ringberg, or any other enjoyable place in Germany. – Key agenda Make decision of the ATCA option or PC option.

29. Short Term Schedule [2] Next PXD DAQ workshop [2] – Items to be presented by each option group ATCA option … baseline option – Demonstration of the following items: » One ATCA can receive the RoI from the HLT, and » The ATCA can produce an output based on that. PC option … backup option – Report of the “RocketIO  PCIe  PC” system performance. – Quantitative strategy to implement the RoI matching software. – Same criteria as the ATCA option should be demonstrated. – Deadline = the next PXD DAQ WS, absolutely Funding issue of Gießen Univ. has been positively cleared.

30. Long Term Schedule In Apr.2011ATCA/PC option decision. By Jun.2011ATCA/PC prototype electronics finalize. By Sep.2012PXD DAQ integration to HLT. By Mar.2013PXD and HLT integration to EVB2. Local DAQ test (other sub-detectors than PXD). By Dec.2013PXD integration to the actual readout system. Sub-detector integration to CDAQ one by one. From 2014Global-DAQ system test.

31. Summary DHH – Two designs to interface the trigger timing system/clock system exist. Option choice will be made after a DHH prototype becomes ready. ATCA/PC – Baseline option and backup option are running in parallel. Option choice will be made in Apr.,2011. HLT – Required HLT pipeline size on the ATCA/PC can be relaxed than the original: 3GB/DHH  300MB/DHH (2GB/DHH for margin is very sufficient).