Development of PCI Bus Based DAQ Platform for Higher Luminosity Experiments T.Higuchi, 1 H.Fujii, 1 M.Ikeno, 1 Y.Igarashi, 1 E.Inoue, 1 R.Itoh, 1 H.Kodama, 1 T.Murakami, 1 Y.Nagasaka, 2 M.Nakao, 1 K.Nakayoshi, 1 M.Saitoh, 1 S.Shimazaki, 1 S.Y.Suzuki, 1 M.Tanaka, 1 K.Tauchi, 1 G.Varner, 3 M.Yamauchi, 1 Y.Yasu, 1 T.Katayama, 4 K.Watanabe, 4 M.Ishizuka, 5 S.Onozawa, 5 and C.J.Li, 5 1 High Energy Accelerator Research Organization (KEK) 2 Hiroshima Institute of Technology 3 University of Hawaii 4 Densan Co. Ltd. 5 Designtech Co. Ltd.

Takeo HIGUCHI (KEK), CHEP2003 Future High Energy Experiments Next B-factory (Super KEB) – Higher luminosity L > cm -2 s -1  trigger rate > 10 kHz. – Dedicated Super Belle detector  data size ~ 300 kB / event. J-PARC (Japan Proton Accelerator Research Complex) – Trigger rate > 10 kHz. – Requirements vary widely depending on experiments. Linac construction 50 GeV synchrotron building KEK-B bird view Belle detector

Takeo HIGUCHI (KEK), CHEP2003 Call for a New DAQ Platform The Belle DAQ was implemented without pipeline. It operates with ~10% deadtime at 500 Hz trigger rate. In future experiments, the trigger rate will be > 10 kHz. A pipelined DAQ for deadtime-less system is necessary. Further more, the complicated detector system may output > 10 times larger data as current one. High density platform is favored. To catch up with the future setup, we have developed a new DAQ platform along with the following boundary conditions….

Boundary Conditions in the Development Keep system flexible – Separate detector dependent parts (ADC, TDC, etc) from the system. Reduce bandwidth usage – Put data reduction CPU on the data stream. Increase trigger efficiency – Put event buffer on the stream to reduce deadtime. Make module compact – Increase module density to be fit in the room. Keep cost lower – Import computer market trends/techniques as possible. New

Takeo HIGUCHI (KEK), CHEP2003 Conceptual Design (Front-end) Mezzanine-ADC or Mezzanine-TDC Module sampling clock Detector 0xaa55 0x0246 0xf3b7 … User-defined ADC/TDC with add-on structure  Less dependence of the platform on the detectors. Mezzanine structure  Higher platform density. User-defined ADC/TDC with add-on structure  Less dependence of the platform on the detectors. Mezzanine structure  Higher platform density. Readout FIFO ADC/TDC L1 trigger pipeline trigger signal

Takeo HIGUCHI (KEK), CHEP2003 TDC Design Example TDC module layout –Input = 24ch LVDS, Output = 24ch. –Resolution = 0.78 ns/bit 40 MHz clock). –Trigger buffer depth = 8 words. –Enough space to put required functions. FPGA AMT  186 mm 2 to COPPER connector AMT-2 = TDC with L1 buffer designed by ATLAS Under design data/24chdigitized output busy clock, trigger

Takeo HIGUCHI (KEK), CHEP2003 Conceptual Design (Back-end) PCI bus based platform PCI provides up to 133MB/s data transfer. PCI is one of PC market standards.  We need less worry about device development.  Newest technology can be always purchased. PCI has mezzanine standard (PMC).  Platform density can be increased. PCI provides up to 133MB/s data transfer. PCI is one of PC market standards.  We need less worry about device development.  Newest technology can be always purchased. PCI has mezzanine standard (PMC).  Platform density can be increased. PCI9054 CPU Trigger Network EB PCI bus Local bus Data compression Readout FIFO deadtime reduction DMA 0xaa55 0x0246 0xf3b7 …

Schematic of the New DAQ Platform Memory Network local busPCI bus detector signals ADC/TDC trigger, clock mezzanine (add-on) module PMC Bridge FIFO trigger interrupt CPU Bus CTRL

Takeo HIGUCHI (KEK), CHEP2003 The COPPER VME-9U size board ADC/TDC CPU/PMC PMC COmmon Pipelined Platform for Electronics Readout VME Rear Board (SPIGOT)

Takeo HIGUCHI (KEK), CHEP2003 COPPER Components 4 ADC/TDC (FINESSE) slots –~100 ch (24 ch  4) / COPPER board 2 MB readout FIFOs –512 kB for each FINESSE slot: IDT72V295 (IDT)  2  4 slots. Data transfer buses –A32/D32/33MHz local bus. –FPGA for Local bus and FIFO control. –Local-PCI bus bridge: PCI9054 (PLX), which has DMA facility. –A32/D32/33MHz PCI bus. –PCI-PCI bus bridge: (Intel)  2. 5 PMC slots (2 are on the SPIGOT) –1 for CPU, 1 for link device to the event builder, and 3 for generic use (intended for additional memory.) VME interface –PCI and VME are connected via dual-port memory for reset/debug use.

Takeo HIGUCHI (KEK), CHEP2003 The FINESSE Front-end INstrumentation Entity for Sub-detector Specific Electronics 186 mm 71 mm User defined ADC/TDC module ~24 input ch. Local bus I/F (for control) Up to 16 bit. output Trigger busyClock, L1 trigger

PCI Ethernet card PMC Ethernet card PCI Mezzanine Card Standard - PMC –PMC is 100% compliant with the PCI. –Many applications are available: Ethernet card, Gigabit Ethernet card, memory module, CPU, etc.

Takeo HIGUCHI (KEK), CHEP2003 PMC Processor RadiSys EPC-6315 – Equipped with Intel PentiumIII 800 MHz. – 512 kB secondary cache. – 256 MB SDRAM with ECC. – Chipset: RadiSys – Bootable from CompactFlash. ­ RedHat Linux 7.3 is running. – 33-bit 33/66 MHz PCI bus interface. CompactFlash socket (boot) RJ-45 Ethernet port (slow control)

Takeo HIGUCHI (KEK), CHEP2003 Test Setup FINESSE COPPER SPIGOT CPU Serial Ethernet NIC (Tx) Trigger Vacant PCLinux PC (Rx) hub 10 Hz trigger Ethernet Thermostatic oven (humidity = 30%) Prototype FINESSEs

Takeo HIGUCHI (KEK), CHEP2003 Stability Test Check list at Rx PC. – Is event-header/event-footer marker correct? – Are data contents same as pre-defined one? – FINESSE local counter == Event tag from trigger system? – Are 4 local counters from FINESSE same? – Does FINESSE local counters increase by 1 correctly? Thermal condition was varied during the test. 1h 20ºC temp. time 10ºC, 24h 1h 40ºC, 24h 50ºC, 24h 15ºC, 1h + room-temp. 48 hours 77 hours Total data transfer = 3.4 GB. No error!

Takeo HIGUCHI (KEK), CHEP2003 Performance Study Data transfer speed – The data transfer speed from the readout FIFOs to the CPU main memory using DMA is measured ~125 MB/s (w/o DMA overhead.) 94 % of full performance. Performance study is proceeding…

Takeo HIGUCHI (KEK), CHEP2003 Status and Prospects COPPER – The COPPER is under final tuning and the design will be fixed by this May. FINESSE – Design of TDC FINESSE will be finalized soon and start production. – Design of ADC FINESSE is being designed. Integrated system test – We replace a part of current Belle DAQ system with COPPERs in the next summer for the integrated system test.

Takeo HIGUCHI (KEK), CHEP2003 Summary We are constructing a pipelined DAQ system for future HEP experiments by developing a PCI-bus based DAQ platform that has mezzanine structure for density increase. The platform equips processor module for data reduction. The platform showed good stability after 125h test. The data-transfer performance of the platform was measured to be close to full performance (94%) in DMA mode (without DMA overhead.) The mezzanine-ADC/TDC modules are being designed. We start integrated system test from next summer using current Belle DAQ system.

Takeo HIGUCHI (KEK), CHEP2003 Current DAQ Platform at Belle FASTBUS crate TDCs Proc. I/F VME crate Trigger dual port mems Q-to-T converted signal Tx VxWorks Trigger input G-Link to event builder

Takeo HIGUCHI (KEK), CHEP2003 The AMT mm Block diagram of the AMT-2

Takeo HIGUCHI (KEK), CHEP2003 Operating System Linux RedHat 7.3 –It costs very low. –It has established a stringent reliability. ­Linux has been used on the Intel platform widely. –Many excellent development tools are provided. –Editor, compiler, debugger, PCI diagnostics tools… –It requires little skill. ­e.g.: Only experts can use VxWorks.