Developments and applications of DAQ framework DABC v2 Jörn Adamczewski-Musch, Nikolaus Kurz, Sergey Linev GSI / Experiment Electronic CHEP

DABC core functionality  Multi-thread, multi-node applications  Zero-copy transport approach full support for InfiniBand VERBS advanced sockets treatment with select()  Flexible plugin interface MBS, HADAQ, ROOT, Go4, EPICS, DIM, FESA  http for slow control JavaScript ROOT as web GUI  In development since 2007 v1.0 in 2009, now v Sergey Linev, DABC v2, dabc.gsi.de2

Local communication Sergey Linev, DABC v2, dabc.gsi.de3 ReceiverSender memory pool buffer File, socket, PCIe,... Consumer  Pre-allocated memory pool  Reference counter for each buffer  Buffers queue between modules  No memcpy between sender and receiver queue

Remote communication Sergey Linev, DABC v2, dabc.gsi.de4 Transport Sender memory pool buffer Producer Receiver Transport memory pool buffer Consumer node1 node2  Same approach as in local  No memcpy for InfiniBand  Different communication patterns: 1  1, 1  N, N  1, N x M queue

InfiniBand tests with DABC  ½ fat tree topology  ~800 nodes with QDR IB  12 spine and 34 leaf switches Sergey Linev, DABC v2, dabc.gsi.de5 ibspine01 ibswitch ibswitch ibswitch ibswitch ibswitch ibswitch ibswitch ibswitch ibswitch ibswitch ibspine02ibspine03ibspine04 ibswitch ibswitch  all-to-all traffic pattern  1.25 TB/s or 80% of available bandwidth Results from May 2011

PEXOR/KINPEX plug-ins Sergey Linev, DABC v2, dabc.gsi.de6 PEXOR (KINPEX): Pci-EXpress Optical Receiver Lattice (Kintex) FPGA 4 lane PCIe 4 high speed SFP connectors FEBEX (ADC), POLAND (QFW), TAMEX (TDC) various frontends Detector GOSIP protocol, optical chains pexorplugin::Device InitDaq() UserReadout() RequestToken() StartAcquisition()... poland::Device InitDaq() UserReadout() explodertest::Device InitDaq() UserReadout() subclasses DABC hardware libpexor.so user library pexor.ko Linux kernel module gosipcmd control tool

PEXOR/KINPEX plug-ins Sergey Linev, DABC v2, dabc.gsi.de7  Use generic pexor kernel module and library  Device base class: common functionality (gosip, DMA, interrupts) interface methods (InitDAQ(), UserReadout() )  Device subclasses: Frontend specific, may implement “user” code  Readout modes: Free running (polling request) Request after trigger interrupt (emulates MBS DAQ) “automatic” within trigger interrupt handler

in DABC  Access to different data in DAQ application  JavaScript ROOT as user interface  One could integrate and access data from different sources: DABC, ROOT, Go4, MBS, EPICS, DIM, FESA,...  Same data could be accessed by Go4 and displayed using native ROOT graphics  One also could use command-line tools to access DAQ/analysis data via http or socket Sergey Linev, DABC v2, dabc.gsi.de8

Heterogeneous distributed applications Sergey Linev, DABC v2, dabc.gsi.de9 MBS DAQ shell> mbs -dabc mbs> open file test.lmd mbs> sta acq cmd Go4 analysis $ go4analysis -stream localhost GO4- *> Reading library: libGo4UserAnalysis GO4-*> Close of TUserSource GO4-*> Create default analysis with processor class TFirstStepProcessor GO4-*> Use class TStreamEvent as output event GO4-*> Welcome to Go4 Analysis Framework Release v4.6.3 (build 40603) ! GO4-*> Analysis: Added analysis step Analysis GO4-*> Main: starting analysis in batch mode... GO4-!> Fail to open AutoSave file Go4AutoSave.root GO4-*> Analysis LoadObjects: Failed to load from file Go4AutoSave.root f_evt_get_open for STREAM: port=6002 timeout=1 GO4-*> Factory: Create input event for MBS GO4-*> Event MbsEvent101 has source localhost class: TGo4MbsStream GO4-*> Factory: Create event processor Processor GO4-*> Create TFirstStepProcessor Processor GO4-*> Executing ROOT script first.C GO4-*> Factory: Create output event OutputEvent GO4-*> Event OutputEvent has source Processor class: TFirstStepProcessor GO4-*> Analysis -- Initializing EventClasses done. TGo4Sniffer socket Publisher HADAQ Publisher socket EPICS DABC DAQ DABC master dabc::Publisher collects data from remote build global hierarchies version control http socket mbs plugin socket DABC agents log status

MBS/GOSIP web gui Sergey Linev, DABC v2, dabc.gsi.de10

DABC/PEXOR/GOSIP web gui Sergey Linev, DABC v2, dabc.gsi.de11

POLAND frontend web gui Sergey Linev, DABC v2, dabc.gsi.de12

HADES detector HADES is a versatile detector for a precise spectroscopy of e + e - pairs (di-electrons) and charged hadrons produced in proton, pion and heavy ion induced reactions in a 1÷3.5 GeV kinetic beam energy region Sergey Linev, DABC v2, dabc.gsi.de13

IOC HADES DAQ before Sergey Linev, DABC v2, dabc.gsi.de14 TRB small hld hld IOC EPICS GUI Event builder server /shm HYDRA2  ~30 TRB frontend boards  ~10 KHz, max 300 MB/s  simple round-robin approach  C-based, single thread code  EPICS IOCs for DAQ slow control  file-based quality monitoring UDP client

DABC HADES DAQ now Sergey Linev, DABC v2, dabc.gsi.de15 TRB hld Combiner output stream  DABC on all event builders  All infrastructure remains as is  Socket data stream to online analysis  Used in production beams 2014 http DABC hld Combiner output stream http DABC hld Combiner output stream http DABC hld Combiner output stream http EPICS GUI /shm server client HYDRA2 UDP

HADES DAQ upgrade  New RICH and ECAL electronic more than 100 TRBs (30 up to now)  Use in many places FPGA TDCs need for special calibration procedure  Potentially free-running readout for RICH need to produce subevents afterwards  HLD data format should be preserved Sergey Linev, DABC v2, dabc.gsi.de16

Change networking topology Sergey Linev, DABC v2, dabc.gsi.de17 TRB DABC TRB DABC 10Gb Ethernet All data from single TRB always send to same DAQ node Every DAQ node can perform TDC calibration and event selection immediately Event building done afterwards with TCP/IP or IB UDP TCP

Next plans  FPGA TDCs calibrations in DABC already done, including web control GUI  Event building network (BNet) for HADES planned as v3 release, end of 2015  Accommodate C++11 features native threads, references probably as v4 in ~ Sergey Linev, DABC v2, dabc.gsi.de18

Conclusion  Flexible framework for different DAQ tasks  Web and JSROOT as user interface  Used in different test beams  Production DAQ for HADES experiment Sergey Linev, DABC v2, dabc.gsi.de19