Tracker Control System – TCS & Tracker Safety System – TSS of CMS Andromachi Tsirou Piero Giorgio Verdini Alexander Dierlamm Guido Dirkes Lorenzo Masetti Manuel Fahrer Robert Stringer Yousaf Shah Frank Hartmann DCSM Zeuthen 28.09.07

Detector Control at the LHC: PVSS PVSS is a SCADA toolkit Supervisory Control And Data Acquisition Developed by an Austrian company ETM Chosen globally by LHC experiments in 1999 Other CERN users have since adopted PVSS It runs on & Tracker runs on only! 07/11/2006

What does PVSS do? PVSS has a highly distributed architecture A PVSS application consists of many processes Event Manager Data Manager Device Drivers User Interfaces Control Managers API Managers Archive Manager Processes can be spread across several PCs Quite static structure and device oriented Built mainly on static Datapoints and Scripts Ctrl API EV D UIM DM DB 07/11/2006

Great Possibilities – Great Constraints PVSS is a commercial SCADA system Customized and extended at CERN by the Joint COntrol Project Framework (group within IT) Customized further by the CMS Central Team Tracker DCS did benefit from all these tools See this talk CMS Tracker DCS CMS Central Team Developments JCOP Framework (including FSM) PVSS Global Rack View Installation, Maintenance and Recovery E.g. CAEN driver Configuration database tools E.g. Archive Alarm handling Access Control DCSM Zeuthen 28.09.07

What's TCS & TSS doing? Hardware view BCM ? WIRES Cooling system SCADA Conf. DB Cond. DB RCMS XDAQ Hybrids (T,V,I) (~0.5 MB/m) CCUM (T) (~10KB/m) DCU values FEC TCS PVSS Thermal screen States PLC (S7) Dry air system PLC PLC TSS Conditioning Interlocks (T, H) S7 Data (Spy) Soap all DCU values data & commands LV & HV OPC high/low voltages environmental CAEN Power supplies Soap Commands (only in standalone) Keep in mind, each block considers several PCs

The Power system DCSM Zeuthen 28.09.07

Powering System Control Chain Rack One Power Group powered by 1 Power Supply Unit (2 LV and 2HV) Up to 9 Power Supply Modules (18 PSU) are lodged into one EASY 4000 crate. Up to 6 crates may be located in one 52U high rack Each rack is controlled by one Branch Controller Up to 16 branch controllers may be controlled by one SY1527. crate Final Aim: The shifter in the control room clicks ON (without knowing any details) Actually, the installation and commissioning is under German responsibility! 4 Mainframes 29 racks 120 crates 1944 PSU 110 Control Power supplies 7 PLC Systems Hostile Environment A1676 branch controller SY1527 Supervisor

A bit reality Back of a crate DCSM Zeuthen 28.09.07

Power system plus detector reflected in TCS 15232 modules Each one needs LV and HV powering 1944 Power Groups Groups of modules sharing LV and HV powering 356 Control Groups Groups of PG sharing one line of CTRL powering 180 Cooling Loops Groups of Control Groups sharing cooling lines

The Tracker safety system DCSM Zeuthen 28.09.07

Tracker Safety System and beyond Three levels of safety: CMS Central Safety System Acts on the general services in case of danger for persons (e.g. fire) Tracker Safety System (TSS) Switches off (parts of) the tracker in case of dangerous environmental conditions Implemented in PLCs Temperature & Humidity Cooling, Thermal Screen, powercut, etc. Software Safeties (TCS) Act in the direction of avoiding the intervention of the Tracker Safety System 1000 probes needed for the tracker: a typical airplane control system has 20-30 probes

TSS System Interlocks Tracker PSU Patch panel Cond. card PLC Conditioning Card HMX (RH) Thermistors (T) ~5000 T & RH TOB/TEC via LIC TIB via PLCC selected ~1000 T & RH TSS patch panel DB15 6 PLC racks TIB+/- TOB+/- TEC+/- Interlocks Interlock cards Petal/rod/string  LIC/PLCC  PSU backplane DB15  TSS patch panel  condition card  PLC input Interlock  front panel of each crate ALL HARDWIRED!!!!!!!!!!!!!!!!!!!!!!!!!!!

to one „Control Group“ with six „Power Groups“ Hardwired probes: TEC T on Si RH T in Air (on PCB) Two petals correspond to one „Control Group“ with six „Power Groups“ T_Si on each petal on R5: interlock Combination of T_Air and RH on 12°, 3°, 6°, 9° o‘clock (D 1, 5, 9): dewpoint monitoring 288 T_Si + 24 RH + 24 T_Air  336 probes / 2 large PLC racks

The Tracker Control System The Hierarchy of the TCS The Tracker Control System DCSM Zeuthen 28.09.07

How is our hierarchy structured? Shutdown on global Status commands Shutdown on PLC Shutdown on CCU DCU TOB: 44 CL 92 CG 688 PG TIB/TID 72 CL 120 CG 488 PG TEC (both TECs) 16 sectors 64 CL 144 CG 768 PG Shutdown on Frontend DCU

Hierarchy (tree) Design Reflecting of Tracker structure was the first motive Allow to have summary info Allow easy tree part exclusion As slim as possible Tracker was the guinea pig with respect to system size and scaling up experience Standard way of inputs from the outside Dedicated automatic shutdown actions to different hierarchy levels Over-temperature from PLC or DCU Thermal Screen failure Etc. Balancing on several PCs This had also an impact on the distribution of power supplies in the cavern DCSM Zeuthen 28.09.07

Partial Tracker hierarchy Root node: interface to CMS Control Unit Logical Unit Power Group Channel Cooling Loops: Run as Control Unit (separate processes) Access Control! Majority Voting! Summary! Unterstrukturen! Exclude/include! Each Control Group has 1 control channel and several power groups Each Power Group has 4 channels ( 2 LV and 2 HV )

Shifter „immediate“ action, when a channel makes trouble Do OFF & Exclude/Disable Click Here Expert comes later and follows the yellow sign All panels look similar Expert panels available to experts Small command set ON, ON_LV, ON_CTRL, OFF; STANDBY Small state set on TOP node: ON, OFF, STANDBY, ERROR

How do we BROWSE the Tracker? Navigate through the given hierarchy There are obvious drawbacks, but THIS is the standard LHC way! 2. Click in the 3D display Click on specific alarms to shortcut the navigation

TCS 3D During MTCC in the central DCS window the 3D display of CMS including the Tracker was tested The colors reflect the states of the module Clicking leads to the corresponding FSM panel We can go to the granularity of the module and link to the DCU information DCSM Zeuthen 28.09.07

How to build and check the TCS System consistently Linda’s Cable File Guido’s PLC Cable File + Some definitions Excel macro (1) Number checking Oracle Database Detector hierarchy description Detector to CAEN cabling Internal PLC connection Grouping in the PLC Relay To Crate PLC Limits and constants Smoothing and archive smoothing Initial settings DCU mapping Alarms Dew points (coupling T and RH) (2) First consistency checking by referential integrity Everything from one source PVSS Scripts (3) Used to build the system @ P5 Datapoints, configuration of Finite State Machine hierarchy and logical view, addresses for PLC, smoothing, archive parameters, etc. In short, the full system distributed on several PCs (4) Sophisticated checkout routines to compare with reality (5) Dynamic mapping of detector to power supply via DCU info

4 SY 1527 Power Supply Controllers Display Terminal (Control Room) TCS @ P5 4 SY 1527 Power Supply Controllers 7 PLC Systems PSX SERVICE PC PLC pc Power Supply PC 1 Power Supply PC 2 Power Supply PC 3 Power Supply PC 4 DCU PC Supervisor PC Cron Jobs Master PLC Experiment Private Network Display Terminal (Control Room) SSH tunneling Installed at P5 with software General Purpose CERN network Display Terminals (Tracker Control Room @ 186)

TCS & TSS – a summary in numbers TCS: Tracker Control System Control, Monitoring, Analysis, Trending and Archiving of Detector Power Supplies Environmental sensors Temperature; humidity Auxiliary systems: Cooling Plant, BCM, Magnet, Thermal Screen DCU information from XDAQ Based on LHCC JCOP framework, PVSS, proper LAN connection Obeys CMS DCS or TK_RCMS (local running) Includes dedicated access control TSS: Tracker Safety System Autarkic PLC system on UPS, interlocks power supplies on basis Temperatures from hardwired sensors Small number of RH probes on exhaust pipes CMS DSS System  104 param ~ 10 PCs ~ 2000 PSU ~ 300 Ctrl PS ~ 1000 hardwired probes ~ 16000 DCUs  10³ param  105 param Ah, Yes, it works Tracker has sophisticated scale up knowledge, now transfered to other sub detectors 6 large PLC racks 1 PLC Master System ~ 1000 hardwired probes We sucessfully controled 25% of the Tracker during commissiong in the Tracker Integration Faccility

Responsibilties of Karlsruhe Design and realization of the TCS system Final consistent GUI for Control room Full CMS & LHC compatibility Interfaces to TSS incl. TSS planning and mapping Planning and installation of power supply system and TSS to power system connections @P5 Commissioning of the Tracker with respect to control and safety (This would be a talk for itself) Also cable connection and consistency check with TCS DCSM Zeuthen 28.09.07

Backup DCSM Zeuthen 28.09.07

How is our hierarchy structured II? TOB: 688 PG 44 CL 92 CG TIB/TID 488 PG 72 CL 120 CG TEC (both TECs) 16 sectors 64 CL 144 CG 768 PG PG: 1 PSU (LV&HV) 1 FE DCU Total: CL: 180 PG: 1944 CG: 356

Reminder: Granularity of SWITCH OFF Shutdown due to DCU analysis Shutdown due to PLC interlock   TOB 1.1.4.2 1.3.3.2 1.2.4.2 1.4.3.1 1.3.1.1 1.5.3.2 1.4.3.2 1.6.4.1 13P-1C 15P-1C 1.3.1.2 1.4.3.3 1.5.4.2 1.6.1.1 17P-1C   TOB 1.1.4.2 1.3.3.2 1.2.4.2 1.4.3.1 1.3.1.1 1.5.3.2 1.4.3.2 1.6.4.1 13P-1C 15P-1C 1.3.1.2 1.4.3.3 1.5.4.2 1.6.1.1 17P-1C   TOB 1.1.4.2 1.3.3.2 1.2.4.2 1.4.3.1 1.3.1.1 1.5.3.2 1.4.3.2 1.6.4.1 13P-1C 15P-1C 1.3.1.2 1.4.3.3 1.5.4.2 1.6.1.1 17P-1C Shutdown due to PLC analysis

Solution: majority voting 95 % HV_mixed Mixed states: needed for security We need to know if any channel in that state Main issue: deal with inclusion and exclusion of subtrees Computation of the percentages allows computing of the state overriding the FSM logic ON_LV 95 % LV_mixed ON_CTRL 95 % CTRL mixed OFF ERROR 5 %