Presentation is loading. Please wait.

Presentation is loading. Please wait.

D ata A cquisition B ackbone C ore DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work.

Published byCalvin Gardner Modified over 8 years ago

Similar presentations

Presentation on theme: "D ata A cquisition B ackbone C ore DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work."— Presentation transcript:

1 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU RP6 project JRA1 FutureDAQ RII3-CT-2004-506078 Again: motivation What we have achieved since January (Legnaro) Performance measurements To do Data Acquisition Backbone Core J.Adamczewski, H.G.Essel, N.Kurz, S.Linev GSI, Experiment Electronics: Data Processing group

2 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 2 Driving forces for DABC 2004 → EU RP6 project JRA1 FutureDAQ* 2004 → CBM FutureDAQ for FAIR 2005 → FOPI DAQ upgrade (skipped) * RII3-CT-2004-506078 1996 → MultiBranchSystem MBS future 50 installations at GSI, 50 external http://daq.gsi.de Detector tests FE equipment tests Data transport Time distribution Switched event building Software evaluation MBS event builder General purpose DAQ Data Acquisition Backbone Core Intermediate demonstrator Requirements build events over fast networks (Gbyte/s) handle triggered or self-trigger front-ends process time stamped data streams provide data flow control (to front-ends) connect (nearly) any front-ends provide interfaces to plug in application codes connect MBS readout or collector nodes be controllable by several controls frameworks

3 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 3 PCIe FE Front end board: sampling ADCs, clock distribution Active Buffer board *: PCI express card PC ABB PCIe 2.5 GBit/s bi-directional (optical) link: data, clock Use case example: Frontend components test The goal: Detector tests FEE tests Data flow FE: Frontend board ABB: Active Buffer board * A.Kugel, G.Marcus, W.Gao, Mannheim University Informatics V

4 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 4 GE switch PC ABB PCIe DCB FE IB switch FE DC Front end board: sampling ADCs (4*4), clock distribution Data collector boards, clock distribution to FE Active Buffer board: PCI express card 8-20 PCs dual quad PC ABB PCIe *8 *4 Scales up to 10k channels, 160 CPUs bidirectional event building Gbyte/s/line 2.5 GBit/s bi-directional (optical) link: data, clock 2.5 GBit/s data links Gigabit Ethernet The goal: Investigate critical technology Detector tests Replace existing DAQ FE: Frontend board DC: Data combiner board ABB: Active Buffer board GE: Gigabit Ethernet IB: InfiniBand MBS: Multi Branch System MBS Use case example: Frontend components test

5 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 5 MBS topologies Readout 1 Readout n Triggerbus TCP Eventbuilder 1 Eventbuilder m Readout Eventbuilder Timebus TCP Timesorter Eventbuilder Readout Triggerbus Eventbuilder Readout Eventbuilder Monitor (analysis) Archive, RFIO Memory mapped bus n x m multiple EBs Time stamped (20 ns) EB hierarchy EB data transport ControlNode TCP < 20 MByte/s

6 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 6 data input sorting tagging filter analysis data input sorting tagging filter analysis IB Linux GE analysis archive Linux GE: Gigabit Ethernet IB: InfiniBand frontend: ActiveBuffer frontend: other frontend: MBS readout scheduler DABC overview: Event building over switched network DABC

7 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 7 DABC: Software objects of data flow engine DABC Module port Data manager DABC Module port process Data queue Device Transport Device TransportData queue Network Module manager locally A module processes data of one or several data streams. Data streams propagate through ports, which are connected by transports and devices

8 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 8 Node XDAQ Executive (process, addressspace) Web server (SOAP), DIM** server XDAQ Application (thread), infospace DABC Core (threads) XDAQ* Infrastructure * Standard DAQ framework for LHC CMS experiment Orsini, Gutleber http://xdaqwiki.cern.chhttp://xdaqwiki.cern.ch ** DIM: Distributed Information Management System, CERN

9 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 9 Transport & Queue DABC: Synchronization DABC Module A processInput processCommand ActionEvent manager Thread with 1-n modules Command Device thread Transport & Data queue A device, i.e. socket device controls several ports (transports). Once a queue buffer is filled, the transport signals the ActionEvent manager, which in turn calls the processInput function of the associated module. Commands are synchronized with the data flow through the ActionEvent manager Data ready

10 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 10 plugin input DABC: Example with module chain plugin Readout plugin combiner receiver sender Network plugin builder plugin analysis Node 6 different modules

11 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 11 Class diagram as currently implemented Ethernet InfiniBand PCIboard Port 6 Modules

12 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 12 Implementation of a PCI board dabc::Transport dabc::Device dabc::PCIDevice dabc::PCITransport dabc::PCIWorkingThread dabc::AbbDevice Driver components

13 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 13 Controls Tools for controls Configuration via XML files State machines, Infospace (global parameters), message/error loggers, monitoring Communication: Webserver, SOAP, DIM Connectivity through DIM to: LabView, EPICS, Java, any DIM client/server Java with NetBeans (Matisse GUI builder) maybe soon in Eclipse Front-end controls? Mix of cooperating control systems First LabView and Java GUIs operable

14 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 14 Controls & monitoring communication XDAQ Executive (process, addressspace) Web server (SOAP) DIM client: Java GUI LabView GUI EPICS GUI SOAP client: Java GUI Infospace* XDAQ Application Modules command queue DABC Core DIM server State machine Web browser GRIDCC * Infospace: remotely accessible parameters

15 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 15 Java-DIM control study Generic construction from commands and parameters offered by DABC DIM servers Applications create the commands and parameters and publish Rate meter, trending, statistics

16 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 16 InfiniBand test cluster at GSI 4 nodes: –Double Opteron 2.2 GHz, 2GB RAM –Mellanox MHES18-XT host adapter (PCIe) –2x Gigabit Ethernet host adapters –SuSE Linux 9.3, x64bit version Mellanox MTS2400 24X InfiniBand switch InfiniBand switch: TS2400 InfiniBand HCA: MHES18-XT Gigabit Ethernet switch

17 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 17 Network test performance 4x4 bidirectional

18 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 18 Optimized transfer (23 nodes at FZK), all to all Synchronization never overloads receivers, but gains 25% only! We thank Frank Schmitz, Ivan Kondov and Project CampusGrid at the Institute for Scientific Computing, Forschungszentrum Karlsruhe for providing resources and support for the large-scale measurements.

19 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 19 Scaling of performance (FZK) Degration only with large buffers

20 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 20 plugin input DABC: Full event building example measurement 4*Readout Data generators combiner receiver sender Network builderanalysis 4 nodes, all to all 6 different modules Node

21 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 21 Dataflow engine and event building rate 4x4 nodes

22 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 22 DABC further tasks Achieved Infrastructure –Setup, configure –Communication lines –Multi-threading Data flow engine –InfiniBand –Sockets (Gigabit Ethernet) –first PCIexpress board –very high performance –back pressure Very first (0.1) Java GUI Todo Data formats Combiner/Time sort MBS event building Event building Final Programming Interface definitions Documentation

23 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 23 Resources and time People of data processing group H.G.Essel J.Adamczewski N.Kurz S.Linev People of controls group maybe one FTE People from CBM hopefully

Download ppt "D ata A cquisition B ackbone C ore DABCDABC 19.7.07, Huelva J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work."

Similar presentations

NSS 2008, Dresden, High Performance Event Building over InfiniBand Networks GSI Helmholtzzentrum für Schwerionenforschung GmbH 20.10.081.

NSS 2008, Dresden, High Performance Event Building over InfiniBand Networks GSI Helmholtzzentrum für Schwerionenforschung GmbH
June 19, 2002 A Software Skeleton for the Full Front-End Crate Test at BNL Goal: to provide a working data acquisition (DAQ) system for the coming full.

June 19, 2002 A Software Skeleton for the Full Front-End Crate Test at BNL Goal: to provide a working data acquisition (DAQ) system for the coming full.
FEE/DAQ Demonstrator Walter F.J. Müller, GSI, Darmstadt for the CBM Collaboration 3 rd FutureDAQ Workshop GSI, Darmstadt, October 11, 2005.

FEE/DAQ Demonstrator Walter F.J. Müller, GSI, Darmstadt for the CBM Collaboration 3 rd FutureDAQ Workshop GSI, Darmstadt, October 11, 2005.
Remigius K Mommsen Fermilab A New Event Builder for CMS Run II A New Event Builder for CMS Run II on behalf of the CMS DAQ group.

Remigius K Mommsen Fermilab A New Event Builder for CMS Run II A New Event Builder for CMS Run II on behalf of the CMS DAQ group.
CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/20 New Experiences with the ALICE High Level Trigger Data Transport.

CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/20 New Experiences with the ALICE High Level Trigger Data Transport.
Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 1 New Test Results for the ALICE High Level Trigger.

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK New Test Results for the ALICE High Level Trigger.
VC Sept 2005Jean-Sébastien Graulich Report on DAQ Workshop Jean-Sebastien Graulich, Univ. Genève o Introduction o Monitoring and Control o Detector DAQ.

VC Sept 2005Jean-Sébastien Graulich Report on DAQ Workshop Jean-Sebastien Graulich, Univ. Genève o Introduction o Monitoring and Control o Detector DAQ.
DABCDABC D ata A cquisition B ackbone C ore RT2010 J.Adamczewski-Musch, H.G.Essel, S.Linev 1 Data Acquisition Backbone Core Framework.

DABCDABC D ata A cquisition B ackbone C ore RT2010 J.Adamczewski-Musch, H.G.Essel, S.Linev 1 Data Acquisition Backbone Core Framework.
Data Acquisition Software for CMS HCAL Testbeams Jeremiah Mans Princeton University CHEP2003 San Diego, CA.

Data Acquisition Software for CMS HCAL Testbeams Jeremiah Mans Princeton University CHEP2003 San Diego, CA.
Use of ROOT in the D0 Online Event Monitoring System Joel Snow, D0 Collaboration, February 2000.

Use of ROOT in the D0 Online Event Monitoring System Joel Snow, D0 Collaboration, February 2000.
CERN - European Laboratory for Particle Physics HEP Computer Farms Frédéric Hemmer CERN Information Technology Division Physics Data processing Group.

CERN - European Laboratory for Particle Physics HEP Computer Farms Frédéric Hemmer CERN Information Technology Division Physics Data processing Group.
MSS, ALICE week, 21/9/041 A part of ALICE-DAQ for the Forward Detectors University of Athens Physics Department Annie BELOGIANNI, Paraskevi GANOTI, Filimon.

MSS, ALICE week, 21/9/041 A part of ALICE-DAQ for the Forward Detectors University of Athens Physics Department Annie BELOGIANNI, Paraskevi GANOTI, Filimon.
Data Acquisition Backbone Core DABC J. Adamczewski, H.G. Essel, N. Kurz, S. Linev GSI, Darmstadt The new Facility for Antiproton and Ion Research at GSI.

Data Acquisition Backbone Core DABC J. Adamczewski, H.G. Essel, N. Kurz, S. Linev GSI, Darmstadt The new Facility for Antiproton and Ion Research at GSI.
D ata A cquisition B ackbone C ore 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU.

D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU.
Artdaq Introduction artdaq is a toolkit for creating the event building and filtering portions of a DAQ. A set of ready-to-use components along with hooks.

Artdaq Introduction artdaq is a toolkit for creating the event building and filtering portions of a DAQ. A set of ready-to-use components along with hooks.
A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research.

A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research.
Boosting Event Building Performance Using Infiniband FDR for CMS Upgrade Andrew Forrest – CERN (PH/CMD) Technology and Instrumentation in Particle Physics.

Boosting Event Building Performance Using Infiniband FDR for CMS Upgrade Andrew Forrest – CERN (PH/CMD) Technology and Instrumentation in Particle Physics.
TRIGGER-LESS AND RECONFIGURABLE DATA ACQUISITION SYSTEM FOR POSITRON EMISSION TOMOGRAPHY Grzegorz Korcyl 2013.

TRIGGER-LESS AND RECONFIGURABLE DATA ACQUISITION SYSTEM FOR POSITRON EMISSION TOMOGRAPHY Grzegorz Korcyl 2013.
Cluster Computers. Introduction Cluster computing –Standard PCs or workstations connected by a fast network –Good price/performance ratio –Exploit existing.

Cluster Computers. Introduction Cluster computing –Standard PCs or workstations connected by a fast network –Good price/performance ratio –Exploit existing.
GBT Interface Card for a Linux Computer Carson Teale 1.

GBT Interface Card for a Linux Computer Carson Teale 1.

Similar presentations

Ads by Google