1. Design Principles of the CMS Level-1 Trigger Control and Hardware Monitoring System Ildefons Magrans de Abril Institute for High Energy Physics, Vienna
Project context
Problem complexity
Design principles
IEEE NPSS 16th Real Time Conference 2009 Control and Monitoring Systems I
14th May 2009

2. ? Project context Electromagnetic Calorimeter: Hadronic Calorimeter:
Measure energy of particles interacting electromagnetically
Particle physicist
Project context
Hadronic Calorimeter:
Measure energies of particles interacting via the strong nuclear force ?
This a transverse slice of the CMS detector
In the left side we can see the collision point
Particles produced out of the collision are detected though several layers of sensors …
Silicon Tracker:
Find charged particle tracks and measure momenta
Muon detector:
Find muon tracks and also measure momenta

3. What is there between CMS and the particle physicist?
40 million events/second
~55 million Channels
~1 Mbyte per event
Solution based on two filter levels:
Level-1 Trigger (HW)
High Level Trigger (SW)
What is there between CMS and the particle physicist?
Control system:
coordinates experiment operation
We would like to record every event but… we do not have the capacity to do that.
It is feasible to store O(100) events/second

4. …To control and monitor… What is this project about?
SOFTWARE
Experiment Control System
HARDWARE
L1 Decision Loop and detector front-ends.
What is this project about?

5. The L1-Trigger Decision Loop
Configuration:
64 crates
O(103) boards
Firmware ~ 15 MB/board
O(102) regs/board
The L1-Trigger Decision Loop
Testing:
O(103) links
The L1 Trigger decision loop consists of the L1 and the TTC system
The Level-1 Trigger (L1) is a custom pipelined hardware logic intended to analyze the bunch crossing data every 25 ns using special coarsely segmented trigger data from the muon systems and the calorimeters. The L1 reduces the rate of crossings to below 100 kHz
While the L1 is taking its decision the full high-precision data of all detector channels are stored in analog or digital buffers, which are only read out if the event is accepted. The L1 decision loop takes 3.2 μs or 128 bunch crossings which is the size of the front-end buffers. The Level-1 Accept (L1A) decision is communicated to the sub-detectors through the Timing, Trigger and Control (TTC) system

6. Experiment Control System
Run Control and Monitoring System (RCMS):
Overall experiment control and monitoring
RCMS framework implemented with java
L1-Trigger Control and Hardware Monitoring System:
Provides a machine and a human interface to operate, test and monitor the Level-1 decision loop hardware components.
(8)
Experiment Control System
Cross-platform Data AcQuisition middleware (XDAQ):
C++ component based distributed programming framework
Used to implement the distributed event builder
Detector Control System (DCS):
Detector safety, gas and fluid control, cooling system, rack and crate control, high and low voltage control, and detector calibration.
DCS is implemented with PVSSII

7. Why is this a complex work?
Scale complexity
~11 subsystems
O(103) boards
Human complexity
~27 Institutes
Peer collaboration vs strong hierarchical organization
Interest and background (physics)
~20% students
Time complexity
~20 years operational life
Why is this a complex work?
P4:
Periodic upgrades
Unforeseen operational needs
P1: Large distributed HW system
P3: Limited SW engineering background
P2:Complex integration management

8. What did we propose? Management methodology
P2: management complexity P4: unforeseen requirements
Management methodology
What did we propose?
Distributed architecture = stable abstraction
P1: Large distributed HW system
P4: periodic upgrades P3: background and interests
Subsystem integration facilities

9. Management methodology
P2: management complexity P4: unforeseen requirements
Management methodology
Distributed architecture = stable abstraction
P1: Large distributed HW system
P4: periodic upgrades P3: background and interests
Subsystem integration facilities

10. Xhannel infrastructure:
Control panel plug-ins +
e.g. GT panel
e.g. DTTF panel
HTTP/CGI: Automatically generated
e.g. Cell FSM operation
The Cell
Other plug-ins:
Command: RPC method. SOAP API extensions
Monitoring items
Synchronous and Asynchronous SOAP API
Xhannel infrastructure:
Designed to simplify access to web services (SOAP and HTTP/CGI) from operation transition methods
Tstore (DB)
Monitor collector
Cells
FSM Plug-ins

11. Management methodology
P2: management complexity P4: unforeseen requirements
Management methodology
Distributed architecture = stable abstraction
P1: Large distributed HW system
P4: periodic upgrades P3: background and interests
Subsystem integration facilities

12. 1 central cell per sub-system ~ STABLE ABSTRACTION OF THE L1T
Hierarchical control system
1 central cell per sub-system ~ STABLE ABSTRACTION OF THE L1T
Multicrate subsystems ~ 2 levels of subsystem cells (1 subsystem central cell)

13. Management methodology
P2: management complexity P4: unforeseen requirements
Management methodology
Distributed architecture = stable abstraction
P1: Large distributed HW system
P4: periodic upgrades P3: background and interests
Subsystem integration facilities

14. Management methodologyS4
New operational capabilities can be coordinated by particle physicist managers without SW expertise S1 S2 S3
Management methodology
e43()
Particle physicist manager
Subsystem SW developer
The prove that this is working is that (even if I continue being necessary, just because every resource is necessary) I am no longer essential. Subsystem developers are in place, TS framework maintenance is in place, is well documented, and the management understand the building blocks and the methodology to coordinate the implementation of new services.
e12()
e23()
e12()
e23() S1 S2 S3 S1 S2 S3
Entry cell
Operation states
Operation transitions
Operation transition methods
Service test

15. Trigger Supervisor design summary
P2: management complexity P4: unforeseen requirements
Time complexity
Management methodology
Trigger Supervisor design summary
Distributed architecture = stable abstraction
Scale complexity
P1: Large distributed HW system
p4: periodic upgrades
p3: background and interests
Human complexity
Subsystem integration facilities