Off-detector & Software development of DAQ System for the EUDET calorimeters Tao Wu On behalf of CALICE-UK Collaboration

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ2 EUDET DAQ Targets Maximizing to use of off-the-shelf commercial components, cheap, scalable and maintainable Provide well-defined interfaces between DAQ components to allow for minimizing costs and development cycles; A control system to easily integrate the rest of sub-systems of detectors Software to build events from bunch train data and disparate sources into single event data Manage network and data storage

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ3 DAQ Mechanical Design targets Targeting EUDET Module and ILC/CALICE; Aim to develop a generic system e.g. ECAL/HCAL Triggerless DAQ system: use C&C instead All data are sent off detector within a bunch train Use bunch struct. as advantages: 1ms in bunch train, read out data within 200ms of inter-train gap; DAQ system will also control power cycling of readout ASICs A funnel-like DAQ to collect, wrap and transmit data in stages before sending to central storage

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ4 DAQ Architecture Overview Slab hosts VFE chips (ASICs) Slab hosts VFE chips (ASICs) Detector Interface Detector Interface connected to Slabs LDA servicing DIFs Link/Data Aggregator Link/Data Aggregator LDAs read out by ODR via opto-links Off-Detector Receiver Off-Detector Receiver PC hosts ODR PCI-Express driver software Local Software DAQ Full blown Software DAQ LDA PC e.g. ECAL Slab DIF ODR Driver Opto Opto Ccc- & Data-link

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ5 DAQ Architecture Hierarchy LDA Host PC PCIe ODR Host PC PCIe ODR Detector Unit DIF C&C Detector Unit DIF Detector Unit DIF Detector Unit DIF Storage 1-3Gb Fibre Mbps HDMI cabling m 0.1-1m Detector Counting Room Covered by this talk Covered by Valeria

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ6 Current Architecture: ODR Assemble data into a usable form for processing Logical tasks: Receive from LDA Process e.g. event building Store Current input is Ethernet for testing performance Internal FPGA test data generator; External data stream via network DAQ PC LDA CCC-link ODR Data-link DIF ASICs DIF ASICs DIF ASICs … Clock / FastControl FE ODR Store Machine clock

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ7 Off Detector Receiver (ODR) Hardware: Using commercial FPGA dev-board: PLDA XPressFX100 Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with expansion board) SFPs for optic link Expansion (e.g. 3xSFP) Receives modularized data from LDA Realized as PCI-Express card, hosted in DAQ PC. 1-4 Opti-links/card (Gigabit), 4 LDAs/card, 1-2 cards/PC Buffers and transfers to store as fast as possible Sends controls and configs to LDA for distribution to DIFs Performance studies & optimisation on-going B.G., RHUL

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ8 ODR Data Access Rate transfer the data from ODR memory to the user-program memory via 4 links ~700 MByte/sec by 25 DMA buffers B.G., RHUL All measurements: single requester thread, no disk write, data copied to the host memory.

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ9 Clock & Control (C&C) board C&C unit provides machine clock and fast signals to 8x ODR/LDA. A low jitter and fixed latency; Logic control (FPGA, connected via USB) Link Data Aggregator provides next stage fanout to Detector Interfaces Eg C&C unit  8 LDAs  8 DIFs = 64 DUs. Signalling over same HDMI type cables Facility to generate optical link clock (~ MHz from ~50MHz machine clock) Board is already designed, will be build soon LDA Host PC PCIe ODR C&C Host PC PCIe ODR Machine Run- Control M.P., UCL busy

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ10 EUDET DAQ software: State State = Dead Transition = PowerUp suceedfailed State = Ready State = Running State = Configured State = InBunchTrain Transition = PowerDown Transition = StartRun Transition = EndRun Trans. = StartConfiguration Trans. = EndConfiguration Trans. = BunchTrainStart Trans. = BunchTrainEnd Preparing Run number & type Ending/Finishing Checks status & communications Distributes configurations Starts the data taking

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ11 Transition: StartRun DAQPCDAQPCDAQPC RCFCConfDB file file file read system status send run number & type send configurations file file Files to be written for book keeping: system status by DAQ PC run info by RC PC system status by FC

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ12 Data Storage Scenario I DAQPC localdisk Centralstore Scenario II DAQPC Centralstore Scenario III DAQ PC file in memory Centralstore RAID Array Which scenario to choose depending on the bandwidth with which the data gets produced: (1) up to 200Mbit/sec, (2) up to ~1600Mbit/sec, (3) from there on Estimation of the data rate for the EUDET DAQ prototype has to cope with ~400Mbit/s, however it depends on the detector and the choice of VFE.

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ13 DAQ Software: DOOCS framework Software development and code base Computer Infrastructure Device layer Middle layer Communication Application layer Sun/Linux Cluster Software Libs

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ14 DAQ Software of DOOCS Fast Linac ImageSlow ADC BM DS DS ML FC SC DAQ server EVB Operator GUI Local GUI Remote GUI dCache DCCPDCCP DISK RC DB RC GUI ROOT storagestorage Data streams FC/SC: Fast/Slow Collector BM: Buffer Manager EVB: Event Builder DS: DAQ Server

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ15 Adapting DOOCS to EUDET DAQ Modeling hardware card via device server ODR/LDA/DIF/ASICs Existing: VME, Sedac, Profi-Bus; ODR/LDA/DIF/ASICs Equipment Name Server Equipment Name Server (ENS): Facility(F) / Device(D) / Location(L) / Property (P) e.g. CALICE.ECAL/ODR/ODR1/Status ► F: CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL ► D: ODR, LDA, DIF, ASICs; ► L: ODR1,ODR2,ODRX; LDA1,LDA2,LDAX; DIF1,DIF2,DIFX; ► Property: X X X ? Build interface talking to hardware (ODR) as a starting point; Classify the properties and functionalities of each device for our EUDET DAQ system (DIF, LDA, ODR, etc.)

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ16 Future plans Classify the necessary properties and functions of all hardware components; Continue their building & testing, debugging & improving meanwhile; Continue to develop interface to hardware (ODR) talking to DOOCS; Investigate & define the DIF  LDA  ODR links How to deal with (a)synchronization for C&C? Putting (all) components together, test…

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ17 Summary Off-detector Receiver has been built for receiving, event building & data storage; its performance has been testing; Clock & Control instead of triggers, the board will be built soon; DOOCS framework is reusable & suitable for our DAQ system. DAQ software is in designing phase…

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ18 DAQ architecture Link Data Aggregator (LDA) Detector Interface (DIF) Detector Unit ASICs Off Detector Receiver (ODR) P. Göttlicher, DESY DAQ software

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ19 Current ODR Prototype Host System EthernetInterface Test Data Generator Source Selector Buffer pointer ManagerBufferMemory Used FIFO DMAEngine PCI Express Buses Local Storage RAID Array Gigabit NIC Test source External data stream Optical fibre data stream Address data stream Access pointers Event extracts Read pointers Test Data Generator Control Off-Detector-Receiver Card driver code DMA engine ODR local software

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ20 Requester Current ODR Prototype “Requester”: multithreaded application, data copied to the host memory ODR VHDL code Requester FIFO Dup. Data Buff DMA FIFO to User buffer Write pointer Read pointer DMA data to user buffer(s) (host memory) DMA transfer in maximum 4 k (page size) blocks. Supervisor Multiple instances of the “requester” thread. Each accessing different ODR card or different channel.

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ21 Establishing optimum number of user-defined DMA buffers. No performance improvement above 20 allocate buffers, Comparison of all channels. Number of data buffers: 20 Performance (Int.Data Gen)

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ22 Improvements in performance when accessing more then one channel. The maximum performance achieved for 3-channels access. Adding additional Channel (4-channel access) does not yield improvement in transfer rate. Performance (continuation)

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ23 Equipment Name Server (ENS) NN+1N+2 N+2,1N+1,1N,1N+1,2 N+2,1,1N+1,1,1N+1,2,1N+2,1,2 …,1…,2…,1 …,2…,3 KK+1K+2K+3K+4 Facilities Devices Locations Properties Entries Select between devices of the same type in different facilities in on-line addressing of the control system. Provide the resolution of the names of the devices used in the control system with one entry per device server. Define user authentication parameters which are used by the control software to restrict the access to the specified control devices

ILC/ECFA 2008 WarsawEUDET/CALICE UK DAQ24 ENS Client programs ENS Facility/Device/Location/Property Query device name Equipment device Addresses of equipment server actual data request to the equipment server Facility:CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL Device:ODR, LDA, DIF, ASIC1, ASIC2, ASIC3 Location: ODR1, ODR2, ODRX; LDA1, LDA2, LDAX; DIF1, DIF2, DIFX Property:X X X ? signal connections ENS can signal connections by additional properties, e.g. for device DIF: CALICE.ECAL/DIF/DIF1/ODR_CON CALICE.ECAL/DIF/DIF1/LDA_CON CALICE.ECAL/DIF/DIF1/DEBUG_MODE