First release of Data Acquisition Backbone Core J. Adamczewski-Musch, H.G. Essel, N. Kurz, S. Linev GSI Darmstadt, Germany Experiment Electronics: Data processing group

FAIR accelerators FAIR baseline* SIS100 SIS300 p-linac HESR Super- FRS p-bar target p-linac Super- FRS SIS100 SIS300 HESR CR RESR Unilac SIS 100 NESR Antiproton Prod. Target SIS18 Upgrade SIS 300 Accelerator SuperFRS PANDA Atomic Phys. Plasma Phys. Low Energy Exp. High Energy Exp. NESR Exp. FLAIR Atomic Physics HADES & CBM Experiment FAIR baseline* Facility for Antiproton and Ion Research *FAIR Monthly, April 2009 18.03.2010

CBM experiment at FAIR Transition Radiation Electro- Detectors Dipole magnet Ring Imaging Cherenkov Detector Transition Radiation Detectors Resistive Plate Chambers (TOF) Electro- magnetic Calorimeter Silicon Tracking System Projectile Spectator (Calorimeter) Micro Vertex One of experiments in FAIR will CBM – Condensed Barion Matter experiment Free-running electronic Precise timing for all detectors 18.03.2010

DAQ architectures Conventional DAQ CBM DAQ HLT HLT FLES L1 trigger HLT HLT FLES CBM DAQ Self-triggered Front-end all hits shipped to DAQ. Data push architecture Detector FEE buffer Readout buffer Switch Processor farm Storage Fast links Readout buffer outside radiation area. Many Gbyte storage easily possible. Allows L1 decision times of 10-100 ms High-throughput Event building One cannot make filter (trigger) decision before complete event is reconstructed. Event reconstruction based on time stamps (time slices building) ~1 TB/s of primary data rates up to event builders First event selection done in processor farm. 18.03.2010

InfiniBand performance studies > 500 MBytes/s bidirectional + 25% InfiniBand is probable candidate for use in event building. Has low-latency and high bandwidth, implements zero-copy concept Scales with number of nodes, low CPU usage Meet CBM requirements thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz 18.03.2010

Hardware tests nXYTER, 128-channel self-triggered readout chip with few ns time resolution, DEFNI collaboration Beside big plans for final system now we need to perform a lot of detectors tests We think, that DAQ software components should be used also at that point for better integration into the final system. For instance, we start investigation of self-triggered electronics with nXYTER chip. The n-XYTER is a 128 channel asynchronous, self- triggered, self-sparcifying readout ASIC developed as a front-end for neutron scattering detector applications. In FAIR experiments like CBM it is used as prototype and test-bed of front-end electronics for several kinds of detectors prototypes. The CBM readout controller (ROC), developed in Kirchhoff Institut für Physik, Heidelberg, is an FPGA-based board, designed to read time-stamped data from n-XYTER and transfer that data via optical link to PC. As alternative option, Ethernet can be used for data transfer. CBM Readout controller (ROC), Kirchhoff Institut für Physik, Heidelberg 18.03.2010

Future DAQ requirements Flexible – adopt different kinds of soft- and hardware Compact – use only necessary code Scalable – from small detector tests to 100-nodes clusters Performing – low framework overhead Multiprocessing & multithreading I show only plans for CBM DAQ, but many other future experiments will demand for new features which are not provided by with our current DAQ system. Therefore we start development of new framework, which should fulfill such a new requirements. First problem which should be solved by DAQ is appropriate usage of multithreading capabilities of modern CPUs. 18.03.2010

Multithreading Deadlock! Thread1 Thread2 runs runs Thread3 Thread4 void MainLoop() { while (IsWorking()) { … writeSomething(); } runs void MainLoop() { while (IsWorking()) { … doSomething(); } runs Deadlock! Thread3 Thread4 void MainLoop() { while (IsWorking()) { … calcSomething(); } runs void MainLoop() { while (IsWorking()) { … readSomething(); } runs Now it is unavoidable to use multi-CPU/multi-core machines Different tasks should run as different threads Threads is just place of code, but not code itself In simple case it is just kind of main loops, which running any code. As long as such threads are not communicate with each other, there is no problems. But if they should communicate, it often leads to different problems. For instance, dead locks. To avoid them, you should really count locks, check their hierarchy and so on. As a result, development of multithreading applications can become very complicated. To avoid such problem in DABC, another approach was taken. 18.03.2010

Multithreading in DABC Processor Events queues Events Timeouts Commands runs Processor Events queues Events Timeouts Commands runs ev1 Thread3 Thread4 Processor Events queues Events Timeouts Commands runs Processor Events queues Events Timeouts Commands runs Code, which runs in threads, organized in form of processors There are several processors in thread Processor process events from the thread event queues. In thread processors executed sequentially (according priority of coming events), therefore no locking required if two such processors communicate with each other. Threads also by request provide timeouts to the processors. Of course, processors can communicate (fire events) with processors from other threads. Such methods are fully thread-safe. Typically one extends such calls and provides more data with event id – buffers, commands, anything else. In such approach code in processors runs absolutely independent from each other and cannot block (in form of blocked mutexes). Any requests to other threads converted to the events and send to other. Other thread can send a response event. Another important problem is memory management. ev2 ev3 18.03.2010

Memory management dabc::MemoryPool dabc::Buffer dabc::Buffer References dabc::Buffer References Memory blocks dabc::Buffer References Concept of memory pools – pre-allocated blocks of memory. Once allocated, layout of memory pool can be fixed, which can avoid expensive (in terms of time) reallocation operation during run. One can reference different peaces of memory pool via buffer objects Many references on the same memory region allowed via reference counter. Scatter-gather lists are supported Only asynchronous (non-blocking) requests to memory pool are supported. Main aim – support of zero-copy concept across complete set of components Multiple references on resources (blocks) Copy of reference without copy of data Zero copy transfer: PCI/DMA → buffer → IB/DMA 18.03.2010

User code - modules dabc::Module dabc::Module provides: dabc::PoolHandle dabc::Timer dabc::Port dabc::Parameter dabc::Module provides: I/O ports for communications Pools handles to request memory Timers for timeouts processing Configuration & monitoring parameters Main place for user-specific code is module. Module has: i/o ports for communication with other components (or other modules) pools handles to request memory timers for timeout processing configuration parameters 18.03.2010

Two kind of modules: sync & async UserModule “Pool” “Input” “Output0” “Output1” Constructors mainly the same CreatePoolHandle("Pool", 2048, 1); CreateInput("Input", Pool(), 5); CreateOutput("Output0", Pool(), 5); CreateOutput("Output1", Pool(), 5); fCnt = 0; Explicit loop in ModuleSync Event processing in ModuleAsync Actual user code should be implemented as subclass of ModuleSync or ModuleAsync classes. Constructor of user module looks mostly the same for both kinds, difference is in implementing code. Sync: explicit main loop, linear code, blocks when requires resources (buffers), suited for simple communication patterns, requires explicit thread to run Async: event-based processing, more complex for implementation, more flexible in functionality, can share thread with other modules or devices Just look on simple example of user module, with one input and two outputs. The only task of such module is sending data consequently from input to both outputs. Probably, for such simple case explicit main loop can be a good solution, but if one starts to add more inputs or outputs and requires complex rules which input and when should deliver to some output, liner code becoming not so easy to implement. For complex tasks usually second approach of events processing with async module is used. That’s about data handling inside user code. Whats happens with buffers outside? void UserModule::MainLoop() { while (ModuleWorking()) { dabc::Buffer* buf = Recv(Input()); if (fCnt++ % 2 == 0) Send(Output(0), buf); else Send(Output(1), buf); } void UserModule::ProcessIOEvent(dabc::Port*) { while (Input()->CanRecv() && Output(fCnt % 2)->CanSend()) { dabc::Buffer* buf = Input()->Recv(); Output(fCnt++ % 2)->Send(buf); } 18.03.2010

Transport & device – dataflow node1 File UserModule “Pool” “Input” “Output0” “Output1” UserModule “Pool” “Input” “Output0” “Output1” Transport Transport Transport Transport Transport DataServer PCIeDevice NetworkDevice Transport: manages buffers queue runs in own thread decouples user code from actual transport functionality Device: represents hardware items manages several transports node2 UserModule “Pool” “Input” “Output0” “Output1” Transport connects ports between modules or with data source/drains. Device represents different hard- or software components, which can provide or consume data Transport connects device with module ports, one device can handle several transports Typical use is network I/O, file I/O, hardware board communications Transport manages its own buffer queue – such queue is sheared between modules if transport connect them locally. Typically transport runs its own thread, but it may also use same thread as module. Main aim of transport is to decouple user code from actual I/O code of framework Transport Transport NetworkDevice File 18.03.2010

Many other features Central dabc::Manager class Configuration with XML files Controlling interface and GUI dabc::Commands Factories (plugins) architecture Application and state machine Event building network (BNET) All this included in version 1 of DABC 18.03.2010

Download via http://dabc.gsi.de DABC release 1.0 Download via http://dabc.gsi.de DABC Controls Plugins Applications Core Slim (batch) DIM socket IB verbs* bnet mbs bnet-mbs ROC/udp ROC/ABB* core-test net-test bnet-test mbs bnet-mbs ROC Java GUI DABC itself is just small core library Useful functionality organized in form of plugins, which can be attached when they need There are already number of such plugins Socket handling with socket-specific thread and processor classes to implement event-based socket handling (via select() function) InfiniBand verbs plugin to utilize full functionality of that fast network technology with low CPU consumption Plugin for MBS (Multi Branch System), current standard DAQ in GSI. Provides set of useful components to connect to running DAQ, work with data formats, read/write data files Event building framework (bnet) – set of DABC-based components to implement NxM event building network, designed to decouple user data handling from actual networking Integration of CBM readout controller – data access either via Ethernet socket or via PCIx card with optical link to board. DABC will be used not only as DAQ for that board but also as device access library, which decouples user code from actual access mode All plugins just a libraries, which can be loaded by DABC at startup time. User can always select which libraries are required. Together with plugins there are number of applications, which are provided. It is either just a configuration file for some ready-to-use plugins (like MBS data combiner module) or implementation of subclasses with user code Plugins: Implementation of device/transports and experiment specific data handling code (programmers) Applications: Mainly setup or testing programs (users) ROC: CBM Readout Controller board ABB: Active buffer board (PCIe) IB: InfiniBand MBS: Multi Branch System (GSI DAQ) * external packages needed 18.03.2010

DABC in CBM test beam roc::Readout STS GEM online go4 monitor Combiner Sorter raw data calibrated data Input0 Input1 Input2 Output0 Output1 roc::Readout GEM First beam tests of CBM detector prototypes. First usage of DABC preliminary version for taking, sorting and storing data. online go4 monitor 18.03.2010

DABC as access layer to ROC ROC/udp plugin ROC/PCIe User access layer DABC Ethernet Users scripts, GUIs file I/O online monitor optic Readout controller can be connected either via Ethernet or by optic While DABC provides a lot of services as thread, memory, configuration management, It becomes a good basis for developing common access layer to the ROC functionality. As a result, same code used in tests setups and in DAQ applications, where many ROCs are connected PCIe 18.03.2010

Planned CBM beam tests Readout boards Combiner boards Detectors FEE Commodity PCs More test beams are planned In next beam it is planned to use many readout nodes and perform event building over IB network. Copper Optic InfiniBand 18.03.2010

Event building network test Readouts Senders InfiniBand Receivers Event builders Libs for all applications Defaults for all workers Explicit config for PCIe board Worker1 PCIe Rnd Worker2 Worker3 Rnd Controller Example of Bnet configuration of 4x4 net. Worker4 Rnd Rnd Rnd Rnd 18.03.2010

Conclusion DABC is general-purpose DAQ framework Can be used for different purposes – from small detectors tests to multi-nodes application Through its plugin architecture can be easy extended to specific needs Open for improvements and new ideas 18.03.2010

Thanks! 18.03.2010