DABCDABC ROC-based DAQ: latest developments and perspectives Jörn Adamczewski-Musch, Hans G. Essel, Sergey Linev GSI, Experiment Electronics: Data Processing group Optics ROClib version 2 Preparation for beamtime in June 2010 BNet demonstrator Throttling

DABCDABC S.Linev ROC-based DAQ: status and perspectives 2 ROClib - status end 2009 Only nXYTER frontend was existing and supported Connection between ROC and PC via Ethernet ROC GUI was disabled rocutil was the only tool for configuration of the ROC DABC DAQ application for readout of several ROCs go4monitor for simple online/offline analysis of data, taken during beamtime Worked reliable during beamtime 2009

DABCDABC S.Linev ROC-based DAQ: status and perspectives 3 Optic developments Three kind of traffic in one media: –data –control –DLM Required extra hardware: –AVNet board Required extra software (included in ROClib): –PCI driver (root account) –mprace library –abbdaemon application Lot of tests last several month: –stable work in fixed configuration, bis 250 MB/s datarate –problems with simultaneous usage of both ports –already supported in ROClib, some improvements required

DABCDABC S.Linev ROC-based DAQ: status and perspectives 4 New functionality in ROClib v2.0 Goal for v 2.0 – optics as mainstream, Ethernet as fallback solution New functionality in ROC FPGA code: –support of new frontend (FEET) –programmable commands lists –new message format (little endian) –reorganization of control address space (breaks compatibility) On the PC side: –support of the new FPGA features –device/transport classes for optic –new roc::Message format – support both old and new formats –roc::Iterator class for access data from different data sources –rocGui for ROC/nXYTER/FEET configurations –go4monitor can be connected directly to the ROC (without DAQ app)

DABCDABC S.Linev ROC-based DAQ: status and perspectives 5 New roc::Message and roc::Iterator classes roc::Message is replacement for old nxyter::Data class Supports both old (v1.x) and new (v2.x) message formats roc::Iterator class allows to get data: –from ROC via Ethernet (via roc::Board class) –from ROC via Optic (also via roc::Board class) –from LMD files (replacement for fileapi) roc::Board* brd = roc::Board::Connect(“optic://abb0”); brd->startDaq(); roc::Iterator iter(brd); int cnt = 0; while (iter.next() && (cnt++<1000)) { iter.msg().printData(); } brd->stopDat(); roc::Board::Close(brd);

DABCDABC S.Linev ROC-based DAQ: status and perspectives 6 rocGui Qt4-based Only for configurations –DAQ with go4monitor Can configure: –ROC itself –nXYTER frontend –FEET frontend Works with optic and Ethernet Able to produce scripts for rocutil Modular design, easy to extend

DABCDABC S.Linev ROC-based DAQ: status and perspectives 7 Migration from v1.x to 2.0 Version 2.0 is not backward compatible with 1.x Old LMD files are supported in v2.0 Final version 2.0 will be announced together with exact instruction for upgrade Exact sequence to upgrade firmware/software on the ROC: –checkout new ROClib in separate location; –with old rocupload program update first firmware (not forget jumpers): rocupload cbmtest01 –fw ~/roclib2/firmware/Release/FEET _ETHERNET bit –with old rocupload program update PowerPC software: rocupload cbmtest01 –sw ~/roclib2/sw-host/release/image.bin –reboot ROC –compile new ROClib and try to use it with the ROC rocGui cbmtest01 –For more info see README.txt and cbm-wiki.gsi.de

DABCDABC S.Linev ROC-based DAQ: status and perspectives 8 Beamtime - plans end 2008 Data combiner board SFP FEB4nx SiCBM01B2 ROC Eth A B SYNC-S AUX FEB4nx SiCBM01B2 ROC Eth A B SYNC-S AUX FEB4nx SiCBM01B2 ROC SFP A B AUX FEB1nx Gen BEAM AUX... ROC SFP A B AUX ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B AUX FEB4nx SiDem1 CBM STS Demo 1 ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B SYNC-S AUX FEB4nx SiDem1 ROC SFP A B AUX FEB4nx SiDem1 CBM STS Demo 1 Data combiner board SFP Data combiner board SFP Eth MBS DABC node1 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe DABC node2 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe DABC node3 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe DABC node4 Eth Readout Event building Filtering File store IB/ PCIe InfiniBand switch

DABCDABC S.Linev ROC-based DAQ: status and perspectives 9 Beamtime in June 2010 – realistic plan FEB4nx SiCBM01B2 ROC Eth A B SYNC-S AUX FEB4nx SiCBM01B2 ROC Eth A B SYNC-S AUX FEB4nx SiCBM01B2 ROC SFP A B SyncS AUX FEB1nx Gen BEAM AUX... ROC SFP A B SyncM AUX ROC SFP A B SYNC-S AUX ROC SFP A B SyncS AUX ROC SFP A B SYNC-S AUX ROC SFP A B SyncS AUX DABC node1 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe DABC node2 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe DABC node3 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe InfiniBand switch DABC node4 SFP/ PCIe Readout Event building Filtering File store IB/ PCIe Any other frontend

DABCDABC S.Linev ROC-based DAQ: status and perspectives 10 Beamtime in June 2010 Necessary components: –Up to 8 ROCs, 4 ABBs –SyncM/SyncS cabling –4x Opteron-IB DAQ cluster with 4 TB storage –DABC with ROC-ABB and ROC-BNet plugin –Several PCs for monitoring Still to do: –reliable work of ABB with both SFPs –modifications in ROC-ABB plugin –Intensive ROC-BNet tests –can be done in next two months Fallback solution (if optic fails) –Same electronics, but readout via Ethernet –Reduced datarates –With or without BNet

DABCDABC S.Linev ROC-based DAQ: status and perspectives 11 CBM DAQ perspectives - dataflow FEE Readout Buffering BNet FLES SLES, storage Switch ~1 TB/s of raw data over network Provides combined time slices to the FLES Throttle data in case of buffers/networks overflow

DABCDABC S.Linev ROC-based DAQ: status and perspectives 12 BNet simulations with SystemC (2005)

DABCDABC S.Linev ROC-based DAQ: status and perspectives 13 InfiniBand performance tests ( )

DABCDABC S.Linev ROC-based DAQ: status and perspectives 14 DABC development (since 2007) General purpose DAQ framework Data-flow core –threads, device & transports –user modules, application Configuration, control, monitoring, GUI Provides components for custom event building network - BNet Real data sources – MBS, ROC/UDP, ROC/Optic, generic PCIe board driver Used as DAQ for CBM 2008/2009 beamtimes Mid- and short-term plans: –Improved command interface (deadlocks recognition) –Support of other kinds of control system (EPICS) –Connection to cluster/job management system

DABCDABC S.Linev ROC-based DAQ: status and perspectives 15 BNet demonstrator Since 2007: –better understanding of traffic/datarates –more realistic trigger algorithms were proposed –DABC was established One can launch activity for new BNet demonstrator: –with more realistic traffic generators –with ~ nodes InfiniBand cluster (~20% of final CBM) –use DABC-based infrastructure –emulation of FLES activity (CPU consumption) Aim of demonstrator: –where InfiniBand technology now, that is near perspectives? –that kind of traffic shaping we need, if any? –is time-scheduled transport gives us significant benefits? –reliability and errors recovery issues?

DABCDABC S.Linev ROC-based DAQ: status and perspectives 16 CBM DAQ perspectives - throttling FEE Readout Buffering BNet FLES SLES, storage No backpressure until BNet. How to throttle data when data rates higher than output link performance?

DABCDABC S.Linev ROC-based DAQ: status and perspectives 17 Throttling algorithm While all messages are time-stamped, one can provide very simple throttle algorithm, based on absolute epoch number. For instance: –drop every 8-th epoch, if not enough –drop every 4-th epoch, if not enough –drop every 2-nd epoch, if not enough –drop everything, but first epoch Any other dropping pattern can be implemented Algorithm required memory for at least last 8 epoch –no problems in readout buffer or BNet What to do in FEE, where buffer capacity is limited? E4E3E2E1E8E7E6E5

DABCDABC S.Linev ROC-based DAQ: status and perspectives 18 Throttling in FEE Doing same as in readout buffers and Bnet – requires large buffer space and complex arbitration logic, implemented in hardware => too complicated As proposed by W. Müller on last DAQ meeting, set busy bit (flip-flop) if FIFO full and reject any new data until next epoch. Efficiency of such algorithm is highly depends from relation between FIFO buffer space, epoch length (E) and output link performance (RATE): –can reasonably work only when FIFO << E * RATE, but potentially high data lost –if FIFO ≈ E * RATE, it is about 50% probability if data of even or odd epoch will be excluded by the FEE chip => completely unusable –if FIFO >> E * RATE, id of dropped epochs will vary from chip to chip, no any real correlation, also unusable The only way to drop data in the FEE – introduce message counter per epoch. If amount of data in current epoch exceeds limit, all consequent messages in current epoch will be dropped. Guarantees, that at least in the epochs begin all data are preserved. Open question – per-channel or per-chip logic?