D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU RP6 project JRA1 FutureDAQ RII3-CT Data Acquisition Backbone Core J.Adamczewski, H.G.Essel, N.Kurz, S.Linev GSI, Experiment Electronics, Data Processing group Motivation Data-flow engine, control Event building network Performance To do

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 2 CBM data acquisition data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher switching network BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution W.F.J.Müller, 2004

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 3 CBM DAQ features summary Self-triggered time stamped data channels. Complex trigger algorithms  transport until filter farm FPGA controlled data flows Event building on full data rate ~1TB/s Event builder network BNet: ~1000 nodes, high-speed interconnections Linux may run on most DAQ nodes (even FPGAs)

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 4 PCIe FE Front end board: sampling ADCs, clock distribution Active Buffer board *: PCI express card PC ABB PCIe 2.5 GBit/s bi-directional (optical) link: data, clock Use case example: Frontend components test The goal: Detector tests FEE tests Data flow tests FE: Frontend board ABB: Active Buffer board * A.Kugel, G.Marcus, W.Gao, Mannheim University Informatics V

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 5 GE switch PC ABB PCIe DCB FE IB switch FE DC Front end board: sampling ADCs, clock distribution Data combiner boards, clock distribution to FE Active Buffer board: PCI express card 8-20 PCs dual/quad PC ABB PCIe 88 44 Scales up to 10k channels, 160 CPUs ~625 Mb/s bi-directional (optical) link: data, clock ~2.5 Gb/s data links The goal: Investigate critical technology Detector tests Replace existing DAQ FE: Frontend board DC: Data combiner board ABB: Active Buffer board GE: Gigabit Ethernet IB: InfiniBand MBS: Multi Branch System MBS Use case example: middle-size setup

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 6 Driving forces and motivation for DABC 2004 → EU RP6 project JRA1 FutureDAQ* 2004 → CBM FutureDAQ for FAIR 2005 → FOPI DAQ upgrade (skipped) 2007 → NUSTAR DAQ * RII3-CT → MBS future 50 installations at GSI, 50 external Detector tests FE equipment tests Data transport Time distribution Switched event building Software evaluation MBS event builder General purpose DAQ Data Acquisition Backbone Core Intermediate demonstrator Requirements connect (nearly) any front-ends handle triggered or self-trigger front-ends process time stamped data streams build events over fast networks provide data flow control (to front-ends) interfaces to plug in application codes connect MBS readout or collector nodes be controllable by several controls frameworks

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 7 DABC key components Data-flow engine –memory and buffers management –threads and events management –data processing modules with ports, parameters, timers –transport and device classes, file I/O –back pressure mechanism Slow control and configuration –components setup on each node –parameters monitoring/changing –commands execution –state machine logic –user interface

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 8 Data flow engine DABC Module port DABC Module port process Local transport A module processes data of one or several data streams. Data streams propagated through ports, which are connected by transports Queue

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 9 Data flow engine DABC Module port DABC Module port process Device Net transport Device Net transport Queue Network A module processes data of one or several data streams. Data streams propagated through ports, which are connected by transports and devices

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 10 Memory management Main features: All memory to be used in modules for transport organized in memory pools Memory pool consists of one or several blocks of memory, divided on equal peaces - buffers Each buffer in memory pool can be referenced once for writing and any times for reading Several references may be combined in gather list Use for transport: Only data from memory pools can be transported via ports Each module port associated with only memory pool Transport between two modules in same application done via pointer Zero-copy network transport where supported (InfiniBand) Support of gather lists for all kind of transports

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 11 Threads and event management DABC Module A processInput processCommand Event manager Thread with modules Device thread Commands are synchronized with the data flow also through the Event manager Data ready event A device, i.e. socket device controls several ports (transports). Once a queue buffer is filled, the transport signals the Event manager, which in turn calls the processInput function of the associated module. Transport Queue DABC Module B Manager thread Command event

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 12 Back pressure mechanism Basic idea: sender allowed to send packets only after receiver confirms (with special acknowledge message) that it has enough resources to receive these packets Implemented on transport layer (not visible for user) Impact on module code: No additional efforts is required Can be enabled/disabled for any port To block connection - just do not read packets from it Pro: easy method for traffic control in small system Con: easy way to block complete network when single node is hanging

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 13 Class diagram as currently implemented Socket InfiniBand PCIboard Port Bnet Module

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 14 Slow control Tools for control Configuration via XML files State machines, Infospace, message/error loggers, monitoring Communication: Webserver, SOAP, DIM Connectivity through DIM to: LabView, EPICS, Java, any DIM client/server Java with NetBeans (Matisse GUI builder) maybe soon in Eclipse Front-end controls? Mix of cooperating control systems First LabView and Java GUIs operable

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 15 Controls & monitoring communication XDAQ Executive (process, address space) Web server (SOAP) DIM client: Java GUI LabView GUI EPICS GUI SOAP client: Java GUI Infospace* XDAQ Application Modules, command queue DABC data-flow DIM server State machine Web browser GRIDCC * Infospace – remotely accessible parameters

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 16 LabView-DIM Control GUI* Generic construction of fixed parameter table from DIM servers * Dietrich Beck, EE

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 17 Java-DIM Control Generic construction from commands and parameters offered by DABC DIM servers Applications create the commands and parameters and publish Ratemeter, trending, statistics

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 18 Generic Java DIM GUI controls

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 19 Event building network (BNet) data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher switching network BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 20 collecting sorting tagging building filtering analysis collecting sorting tagging building filtering analysis IB Linux GE analysis archive Linux GE: Gigabit Ethernet IB: InfiniBand frontend DataDispatcher frontend other frontend MBS readout Sender BNet dataflow – bidirectional approach DABC Receiver

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 21 plugin input BNet – modules view plugin* Readout plugin SubeventCombiner DataReceiver DataSender Network NxM plugin EventBuilder plugin* EventFilter M sender nodes 6 different modules N receiver nodes N outputs M inputs Analisys / Storage * optional

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 22 BNet components BNet modules: Readout – collecting packets from readout channels (MBS, PCI-board, other) SubeventCombiner – search and combines data, belonging to same event or time frame DataSender – sends subevent data over network to correspondent event building node DataReceiver – forwards packets from network to event builder EventBuilder – build complete event EventFilter – performs first event filtering BNet user plugins: finds data tags (event id or time stamp) in raw packet resort data for sending over net build event from received subevent packets implements filtering algorithm for events

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 23 InfiniBand – testbench for BNet InfiniBand - reliable low-latency zero-copy high-speed data transport Aim: Prove of principal as event building network candidate for CBM Tests last year: GSI cluster - 4 nodes, SDR Forschungszentrum Karlsruhe* (March 2007) – 23 nodes DDR UNI Mainz** (August 2007) nodes DDR DDR – double data rate (up to 20 Gb/s), SDR – single data rate (10 Gb/s) SDR (GSI)DDR (Mainz) Unidirectional0.98 GB/s1.65 GB/s Bidirectional0.95 GB/s1.3 GB/s Point-to-point tests Rate per node Gb Ethernet~100 MB/s InfiniBand~800 MB/s BNet prototype (GSI, 4 Nodes) * thanks to Frank Schmitz, Ivan Kondov and Project CampusGrid in FZK ** thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 24 Scaling of asynchronous traffic

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 25 Chaotic (async.) versus scheduled (sync.)

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 26 DABC further tasks Achieved: Control infrastructure –setup, configure –communication –very first Java GUI Data flow engine –multi-threading –InfiniBand –Sockets (Gigabit Ethernet) –first PCIexpress board –good performance –back pressure To do: Data formats Error handling/recovery MBS event building Time-stamped data Final API definitions Documentation