Design Principles of the CMS Level-1 Trigger Control and Hardware Monitoring System Ildefons Magrans de Abril Institute for High Energy Physics, Vienna Project context Problem complexity Design principles IEEE NPSS 16th Real Time Conference 2009 Control and Monitoring Systems I 14th May 2009

? Project context Electromagnetic Calorimeter: Hadronic Calorimeter: Measure energy of particles interacting electromagnetically Particle physicist Project context Hadronic Calorimeter: Measure energies of particles interacting via the strong nuclear force ? This a transverse slice of the CMS detector In the left side we can see the collision point Particles produced out of the collision are detected though several layers of sensors … Silicon Tracker: Find charged particle tracks and measure momenta Muon detector: Find muon tracks and also measure momenta

What is there between CMS and the particle physicist? 40 million events/second ~55 million Channels ~1 Mbyte per event Solution based on two filter levels: Level-1 Trigger (HW) High Level Trigger (SW) What is there between CMS and the particle physicist? Control system: coordinates experiment operation We would like to record every event but… we do not have the capacity to do that. It is feasible to store O(100) events/second

…To control and monitor… What is this project about? SOFTWARE Experiment Control System HARDWARE L1 Decision Loop and detector front-ends. What is this project about?

The L1-Trigger Decision Loop Configuration: 64 crates O(103) boards Firmware ~ 15 MB/board O(102) regs/board The L1-Trigger Decision Loop Testing: O(103) links The L1 Trigger decision loop consists of the L1 and the TTC system The Level-1 Trigger (L1) is a custom pipelined hardware logic intended to analyze the bunch crossing data every 25 ns using special coarsely segmented trigger data from the muon systems and the calorimeters. The L1 reduces the rate of crossings to below 100 kHz While the L1 is taking its decision the full high-precision data of all detector channels are stored in analog or digital buffers, which are only read out if the event is accepted. The L1 decision loop takes 3.2 μs or 128 bunch crossings which is the size of the front-end buffers. The Level-1 Accept (L1A) decision is communicated to the sub-detectors through the Timing, Trigger and Control (TTC) system

Experiment Control System Run Control and Monitoring System (RCMS): Overall experiment control and monitoring RCMS framework implemented with java L1-Trigger Control and Hardware Monitoring System: Provides a machine and a human interface to operate, test and monitor the Level-1 decision loop hardware components. (8) Experiment Control System Cross-platform Data AcQuisition middleware (XDAQ): C++ component based distributed programming framework Used to implement the distributed event builder Detector Control System (DCS): Detector safety, gas and fluid control, cooling system, rack and crate control, high and low voltage control, and detector calibration. DCS is implemented with PVSSII

Why is this a complex work? Scale complexity ~11 subsystems O(103) boards Human complexity ~27 Institutes Peer collaboration vs strong hierarchical organization Interest and background (physics) ~20% students Time complexity ~20 years operational life Why is this a complex work? P4: Periodic upgrades Unforeseen operational needs P1: Large distributed HW system P3: Limited SW engineering background P2:Complex integration management

What did we propose? Management methodology P2: management complexity P4: unforeseen requirements Management methodology What did we propose? Distributed architecture = stable abstraction P1: Large distributed HW system P4: periodic upgrades P3: background and interests Subsystem integration facilities

Management methodology P2: management complexity P4: unforeseen requirements Management methodology Distributed architecture = stable abstraction P1: Large distributed HW system P4: periodic upgrades P3: background and interests Subsystem integration facilities

Xhannel infrastructure: Control panel plug-ins + e.g. GT panel e.g. DTTF panel HTTP/CGI: Automatically generated e.g. Cell FSM operation The Cell Other plug-ins: Command: RPC method. SOAP API extensions Monitoring items Synchronous and Asynchronous SOAP API Xhannel infrastructure: Designed to simplify access to web services (SOAP and HTTP/CGI) from operation transition methods Tstore (DB) Monitor collector Cells FSM Plug-ins

Management methodology P2: management complexity P4: unforeseen requirements Management methodology Distributed architecture = stable abstraction P1: Large distributed HW system P4: periodic upgrades P3: background and interests Subsystem integration facilities

1 central cell per sub-system ~ STABLE ABSTRACTION OF THE L1T Hierarchical control system 1 central cell per sub-system ~ STABLE ABSTRACTION OF THE L1T Multicrate subsystems ~ 2 levels of subsystem cells (1 subsystem central cell)

Management methodology P2: management complexity P4: unforeseen requirements Management methodology Distributed architecture = stable abstraction P1: Large distributed HW system P4: periodic upgrades P3: background and interests Subsystem integration facilities

Management methodology S4 New operational capabilities can be coordinated by particle physicist managers without SW expertise S1 S2 S3 Management methodology e43() Particle physicist manager Subsystem SW developer The prove that this is working is that (even if I continue being necessary, just because every resource is necessary) I am no longer essential. Subsystem developers are in place, TS framework maintenance is in place, is well documented, and the management understand the building blocks and the methodology to coordinate the implementation of new services. e12() e23() e12() e23() S1 S2 S3 S1 S2 S3 Entry cell Operation states Operation transitions Operation transition methods Service test

Trigger Supervisor design summary P2: management complexity P4: unforeseen requirements Time complexity Management methodology Trigger Supervisor design summary Distributed architecture = stable abstraction Scale complexity P1: Large distributed HW system p4: periodic upgrades p3: background and interests Human complexity Subsystem integration facilities