Dynamic configuration of the CMS Data Acquisition Cluster Hannes Sakulin, CERN/PH On behalf of the CMS DAQ group Part 1: Configuring the CMS DAQ Cluster Part 1: Configuring the CMS DAQ Cluster Part 2: Creating DAQ Configurations Part 2: Creating DAQ Configurations

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 2 The Compact Muon Solenoid Experiment Drift-Tube chambers Cathode Strip Chambers Resistive Plate Chambers Iron Yoke 4 T Superconducting Coil Tracker Si StripSi Strip Si PixelSi Pixel Electromagnetic Calorimeter Hadronic Calorimeter LHC p-p collisions, E CM =14 TeV Bunch crossing frequency 40 MHz CMS Multi-purpose detector, broad physics programme 55 Mio. Channels to read out, 1 MB event size after zero suppression 100 kHz Level-1 accept rate = DAQ input rate  High-Level Trigger implemented in software running on the DAQ cluster  DAQ input throughput 100 GB / s

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 3 CMS Central DAQ System...Front-End Drivers 0 ~500 ~650 Readout Links FMM …… Trigger Throttling System … FMM Switch Fabric … DATA FROM DETECTOR Myrinet … Level-1 Trigger Input Stage Custom Electronics controlled by PCs through Compact PCI Event Builder Slices Software on PC farm & Gigabit Ethernet … 8x Super-Fragment Builder Myrinet 8x8 071

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 4 Event Builder Slice EVMRU Gigabit Ethernet Switch BF 01 ~71 01 ~ RU 2 SM Builder / Filter Unit 8 cores 1 builder process 7 filter (=high level trigger) processes Readout Unit Event Manager Storage Manager multiple rails super - fragments mass storage Myrinet to Tier-0

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 5 Event Builder Slice EVMRU Gigabit Ethernet Switch BF 01 ~71 01 ~ RU 2 SM Builder / Filter Unit 8 cores 1 builder process 7 filter (=high level trigger) processes Readout Unit Event Manager Storage Manager multiple rails super - fragments mass storage Myrinet to Tier-0 Software components on all hosts based on the XDAQ infrastructure

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 6 CMS Central DAQ Online Software DAQ online software is based on a common infrastructure: XDAQ  XDAQ executives (= processes)  XDAQ applications one or more per executive  Data Transport Protocols  Hardware access Library  highly configurable through XML documents XDAQ executive Host XML XDAQ application XDAQ: see talk by J. Gutleber at the start of the session custom hardware XDAQ application Data Transport … XML document determines role of the executive libraries to be loaded applications & parameters network connections collaborating applications

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 7 Dynamic configuration of hosts Resource Service (Oracle database) jobcontrol service Run Control XDAQ executive Host in the CMS DAQ cluster application libraries are installed locally on all PCs XML Java / Tomcat XDAQ application XDAQ executive XML SOAP JDBC XDAQ configuration determines configuration of custom hardware configuration of Super-Fragment Builder event data flow topology in Event Builders

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 8 Configuration Structure in Resource Service FM FM 2 FM 1 XDAQ executive configuration document Run Control Function Manager Hierarchy job control executive Application 2 Application 1 XML executive Application 2 Application 1 XML executive Application 2 Application 1 XML executive Application XML … … executive Application 2 Application 1 XML application properties XDAQ I2O & SOAP connections Relational Database model, Java Object model control connections Function managers are loadable Java modules that control parts of the DAQ System url, source url, parameters

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 9 Resource Service Organization Resource Service Configurations are organized in a tree. Tree Structure Configurations are versioned XML configuration documents stored in relational scheme Serialized Java Objects stored in the database for performance reasons Resource Service & Run Control used by all CMS Subsystems

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 10 … Dynamic Configuration of the DAQ Cluster Run Control XML Typical Central DAQ Configuration XML System startup time: 35 seconds (further optimizations on the way) loading of the configuration from resource service loading and creation of function managers start of all XDAQ executives and applications on all hosts Resource Service database XML 5Run Control tomcats 50Function Managers 1500hosts 6500executives (processes) 10000applications

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 11 Part 2: How do we fill the Resource Service?

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 12 The problem: Configuration needs to be flexible The system shown is an ideal system  In real life, always hardware problems in a small fraction of the system  Need to quickly adapt the configuration … ✗ ✗✗

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 13 The problem: Configuration needs to be flexible The system shown is an ideal system  In real life, always hardware problems in a small fraction of the system  Need to quickly adapt the configuration Not always using the full system  Staged deployment  Maintenance of part of the cluster  Parallel test runs / partitioned operation  Need to scale the configuration May also change roles of hosts …

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 14 The problem: Configuration needs to be flexible The system shown is an ideal system  In real life, always hardware problems in a small fraction of the system  Need to quickly adapt the configuration Not always using the full system  Staged deployment  Maintenance of part of the cluster  Parallel test runs / partitioned operation  Need to scale the configuration May also change roles of hosts …

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 15 The problem: Configuration needs to be flexible The system shown is an ideal system  In real life, always hardware problems in a small fraction of the system  Need to quickly adapt the configuration Not always using the full system  Staged deployment  Maintenance of part of the cluster  Parallel test runs / partitioned operation  Need to scale the configuration May also change roles of hosts …

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 16 The problem: Configuration needs to be flexible The system shown is an ideal system  In real life, always hardware problems in a small fraction of the system  Need to quickly adapt the configuration Not always using the full system  Staged deployment  Maintenance of part of the cluster  Parallel test runs / partitioned operation  Need to scale the configuration May also change roles of hosts Software keeps evolving, parameters are optimized  Super-fragment composition  Number of network rails, buffer sizes, …  Need to adjust configuration … frequently need to change configuration

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 17 Parameters of the custom electronics need to be adjusted  what Readout Links and Merging Modules to include (crate, slot)  what channels to include / mask out Routing of the Myrinet Super-fragment Builder changes  what input addresses send data to what output addresses  respect constraints of switch fabric in order to be non-blocking Networks Connections in the event builder vary  need to specify the collaborating hosts for each host  need to specify the network to use for each connection Multiplicity, Location and Connectivity of Functional Units vary frequently Quick analysis: How do configurations differ ? Control structure largely constant  some parts need to be duplicated for each Event Builder Slice Composition of functional units (e.g. Readout Unit) largely constant  slow evolution with new software releases Configuration Template algorithmic computation High Level Configuration Layout

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 18 The solution: DAQ Configurator DAQ Configurator High-Level DAQ Configuration Layout Configuration Template Hardware & Configuration DB Configuration Template DB Standalone Java web-start application Resource Service DB XML DAQ Configuration

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 19 The solution: DAQ Configurator DAQ Configurator High-Level DAQ Configuration Layout Configuration Template Hardware & Configuration DB Configuration Template DB Standalone Java web-start application Resource Service DB XML DAQ Configuration

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 20 Configuration Template DB Template FM Template FM 2 Template FM 1 Application 1 Template Control Structure Hierarchy of Run Control Function Managers Template Unit Template Connections I2O / SOAP single / multi rail global / local to slice Template XDAQ Applications Template Unit Application 2 Application 1 Configuration Templates are organized in a tree Template Application parameters placeholders filled from high-level configuration layout or computation Template Functional Units map to XDAQ executives

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 21 Configuration templates are parameterizable Variables substitution  ${varName}, ${ arithmeticExpression } Three levels (sets of variables)  Site settings (production system, test beds) location of libraries addresses of services, etc.  Account Settings run control installation port ranges, etc.  User input before creation of the configuration dataset label test options (e.g. drop data at a certain stage)  Can use the same configuration template for test setups and for the real DAQ system

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 22 Typical CMS DAQ Software Template DAQ FM SLICE FM FEDBuilder FM TTS FM RUBuilder FM HLTS FM FRL Controller Front-end Readout Link Crate FBO EVM pheaps ptATCP Readout Unit BU ptATCP Builder Unit FMM Controller Fast Merging Module Crate Event Processor Filter Unit Storage Manager Template Control Structure Template Functional Units ptATCP Template Connections FBO RU pheaps ptATCP Resource Broker Event Manager duplicated per slice

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 23 DAQ Configurator workflow DAQ Configurator High-Level DAQ Configuration Layout Configuration Template Hardware & Configuration DB Configuration Template DB Standalone Java web-start application Resource Service DB XML DAQ Configuration

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 24 DAQ Hardware & Configuration DB Equipment Set hosts & networks custom hardware all cabling Admin tool Updated when hardware / cabling changes on average every 2 weeks so far Super Fragment Builder Set Front-End Drivers to read out their grouping DAQ Partition Set Definition of Slices Event Manager Readout Units Builder/Filter Units Storage Managers Updated to balance Super-Fragment Builders Updated to scale system / exclude faulty hardware SFBuilder Set Java Filler DAQPartition Set Java Filler Relational Scheme equipment sets for different DAQ setups evolution over time

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 25 CMS DAQ Hardware and Configuration Database Organization A) Equipment Sets B) Super-Fragment Builder Sets C) DAQ Partition Sets Directories may be used for grouping at all levels

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 26 DAQ Configurator workflow Compute configuration of custom electronics using connectivity information from Equipment Set Readout Links: hosts, slots, masks, expected IDs Trigger Throttling: hosts, slots, masks, mappings Super-Fragment Builder: input & output addresses Compute configuration of custom electronics using connectivity information from Equipment Set Readout Links: hosts, slots, masks, expected IDs Trigger Throttling: hosts, slots, masks, mappings Super-Fragment Builder: input & output addresses Resource Service DB Hardware & Configuration DB Configuration Template DB DAQ Configurator (Java) Duplicator Compute configuration details Compute configuration details Router XML Generator XML Generator Parameterize Template Duplicate Template Functional Units for each unit in the high-level layout Use parameters from layout Connect Sources with Targets Globally / within slice SOAP / I2O single / multiple rails Different routing algorithms Connect Sources with Targets Globally / within slice SOAP / I2O single / multiple rails Different routing algorithms Generate an XML file for each XDAQ executive Slice Unit SubUnit Parameter Generic high-level layout XML user input

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 27 1) Select DAQPartitionSet from Hardware & Configuration DB 2) Select Site Settings 3) Select Account Settings 4) Select Configuration Template 5) Optionally Adjust Variables 6) Select Output Location in Resource Service Configuration 8) Optionally Visualize & Edit configuration 7) Go! timing report: typical time to create full 8-slice DAQ configuration (10000 applications) seconds depending on database load timing report: typical time to create full 8-slice DAQ configuration (10000 applications) seconds depending on database load Graphical editor for Software Templates Graphical editor for Software Templates

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 28 Configuration Display (mid-size configuration) Useful for debugging small configurations …

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 29 The Configurator in practice … The Configurator has been used successfully to create all central DAQ configurations  for the DAQ test setups (development & integration)  for CMS commissioning and cosmic data taking since 2006  for the first LHC beam data taking in September 2008 Debris from particles hitting the collimator blocks. First LHC beam in September Cosmic Muon crossing CMS. Magnet Test, August 2006.

CHEP ‘09, 23 March 2009H. Sakulin / CERN PH 30 Summary Configuration of the CMS DAQ cluster needs to be highly flexible The Run Control System configures the DAQ cluster dynamically at the start of a session  configuration data is loaded from the Resource Service database and distributed via SOAP / XML  online software is loaded dynamically depending on the configuration Developed a powerful tool to create configurations for the CMS DAQ System: CMS DAQ Configurator  high-level configuration layout + configuration template = configuration  algorithmic computation of configuration details using underlying database of connectivity information  fast turn-around time  successfully in use since 2006