1. Tracker Control System – TCS & Tracker Safety System – TSS of CMS
Andromachi Tsirou
Piero Giorgio Verdini
Alexander Dierlamm
Guido Dirkes
Lorenzo Masetti
Manuel Fahrer
Robert Stringer
Yousaf Shah
Frank
Hartmann
Specify people affiliation
DCSM Zeuthen

2. What's TCS & TSS doing? Hardware view
BCM ?
WIRES
Cooling system
SCADA
Conf. DB
Cond. DB
RCMS
XDAQ
Hybrids (T,V,I) (~0.5 MB/m)
CCUM (T) (~10KB/m)
DCU values
FEC
TCS
PVSS
Thermal screen
States
PLC (S7)
Dry air
system
PLC
PLC
TSS
Conditioning
Interlocks
(T, H) S7
Data (Spy)
Soap
all DCU values
data & commands
LV & HV
OPC
high/low voltages
environmental
CAEN
Power supplies
Soap
Commands (only in standalone)
Keep in mind, each block considers several PCs, impossible task, show all information but switch on like a home television or light bulb

3. Detector Control at the LHC: PVSS
PVSS is a SCADA toolkit
Supervisory Control And Data Acquisition
Developed by an Austrian company
ETM
Chosen globally by LHC experiments in 1999
Other CERN users have since adopted PVSS
Controls Gothard tunnel; HH harbor, Austrian food industry, quite a lot of German Airtraffic
It runs on &
Tracker runs on only!
07/11/2006

4. What does PVSS do? PVSS has a highly distributed architecture
A PVSS application consists of many processes
Event Manager
Data Manager
Device Drivers
User Interfaces
Control Managers
API Managers
Archive Manager
Processes can be spread across several PCs
Quite static structure and device oriented
Built mainly on static Datapoints and Scripts
Ctrl
API EV D
UIM DM DB
LHC was right to pool all resources into one common solution, unlike CDF
07/11/2006

5. Great Possibilities – Great Constraints
PVSS is a commercial SCADA system
Customized and extended at CERN by the Joint COntrol Project Framework (group within IT)
Customized further by the CMS Central Team
Tracker DCS did benefit from all these tools
See this talk
CMS Tracker DCS
CMS Central Team Developments
JCOP Framework
(including FSM)
PVSS
Global Rack View
Installation, Maintenance and Recovery
E.g. CAEN device models
Configuration database tools
Finite State Machine
E.g. Archive
Alarm handling
Access Control
DCSM Zeuthen

6. The Power system
DCSM Zeuthen

7. Powering System Control Chain
Rack
One Power Group powered by 1 Power Supply Unit (2 LV and 2HV)
Up to 9 Power Supply Modules (18 PSU) are lodged into one EASY 4000 crate.
Up to 6 crates may be located in one 52U high rack
Each rack is controlled by one Branch Controller
Up to 16 branch controllers may be controlled by one SY1527.
crate
Final Aim:
The shifter in the control room clicks ON (without knowing any details)
Actually, the installation and commissioning
is under German responsibility!
4 Mainframes
29 racks
120 crates
1944 PSU
110 Control Power supplies
7 PLC Systems
Hostile Environment
A1676
branch controller
SY1527 Supervisor

8. A bit reality
Back of a crate
DCSM Zeuthen

9. Power system plus detector reflected in TCS
15232 modules
Each one needs LV and HV powering
1944 Power Groups
Groups of modules sharing LV and HV powering
356 Control Groups
Groups of PG sharing one line of CTRL powering
180 Cooling Loops
Groups of Control Groups sharing cooling lines

10. The Tracker safety system
DCSM Zeuthen

11. Tracker Safety System and beyond
Three levels of safety:
CMS Central Safety System
Acts on the general services in case of danger for persons (e.g. fire)
Tracker Safety System (TSS)
Switches off (parts of) the tracker in case of dangerous environmental conditions
Implemented in PLCs
Temperature & Humidity
Cooling, Thermal Screen, powercut, etc.
Software Safeties (TCS)
Act in the direction of avoiding the intervention of the Tracker Safety System
1000 probes needed for the tracker: a typical airplane control system has probes
Tested and improved during TIF running; global shutdown gives several seconds head warning to give ramp down possibility

12. to one „Control Group“ with six „Power Groups“
Hardwired probes: TEC
T on Si RH
T in Air (on PCB)
Two petals correspond
to one „Control Group“ with six „Power Groups“
Defined together with Aachen people
T_Si on each petal on R5:
interlock
Combination of T_Air and RH on 12°, 3°, 6°, 9° o‘clock (D 1, 5, 9):
dewpoint monitoring
288 T_Si + 24 RH + 24 T_Air
 336 probes / 2 large PLC racks

13. TSS System Interlocks Tracker PSU Patch panel Cond. card PLC
Conditioning Card
HMX (RH)
Thermistors (T)
~5000 T & RH
TOB/TEC via LIC
TIB via PLCC
selected
~1000 T & RH
TSS patch
panel
DB15
6 PLC
racks
TIB+/-
TOB+/-
TEC+/-
The way is quite complicated and fix/hardwired
Interlocks
Interlock cards
Petal/rod/string  LIC/PLCC  PSU backplane DB  TSS patch panel  condition card  PLC input
Interlock  front panel of each crate
ALL HARDWIRED!!!!!!!!!!!!!!!!!!!!!!!!!!!

14. The Tracker Control System
The Hierarchy of the TCS
The Tracker Control System
DCSM Zeuthen

15. How is our hierarchy structured?
Shutdown on global
Status
commands
Shutdown on PLC
Shutdown on CCU DCU
REFLECT TRACKER HARDWARE STRUCTURES; Higher blobs can be driven in individual PCs  Balancing, slim, red blobs in PVSS only and on different PCs (our standard way) Commands always travel to all children nodes; info is summarized (majority); standard code for each node-level
TOB:
44 CL
92 CG
688 PG
TIB/TID
72 CL
120 CG
488 PG
TEC (both TECs)
16 sectors
64 CL
144 CG
768 PG
Shutdown on Frontend DCU

16. Hierarchy (tree) Design
Reflecting of Tracker structure was the first motive
Allow to have summary info
Allow easy tree part exclusion
As slim as possible
Tracker was the guinea pig with respect to system size and scaling up experience
Standard way of inputs from the outside
Dedicated automatic shutdown actions to different hierarchy levels
Over-temperature from PLC or DCU
Thermal Screen failure
Etc.
Balancing on several PCs
This had also an impact on the distribution of power supplies in the cavern
DCSM Zeuthen

17. Partial Tracker hierarchy
Root node: interface to CMS
Control Unit
Logical Unit
Power Group
Channel
Cooling Loops:
Run as Control Unit
(separate processes)
Access Control! Majority Voting! Summary! Standard sub structures! Exclude/include! Quite a lot of hints from Aachen people
Small set of states & commands Take from CL upwards possible; all panels look similar
Each Control Group has 1 control channel and several power groups
Each Power Group has 4 channels ( 2 LV and 2 HV )

18. How do we BROWSE the Tracker?
Navigate through the given hierarchy
There are obvious drawbacks, but THIS is the standard LHC way!
2. Click in the 3D display
Click on specific alarms to shortcut the navigation

19. How to build and check the TCS System consistently
Linda’s Cable File
Guido’s PLC Cable File +
Some definitions
Excel macro
(1) Number checking
Oracle Database
Detector hierarchy description
Detector to CAEN cabling
Internal PLC connection
Grouping in the PLC
Relay To Crate
PLC Limits and constants
Smoothing and archive smoothing
Initial settings
DCU mapping
Alarms
Dew points (coupling T and RH)
(2) First consistency checking by referential integrity
Everything from one source
PVSS Scripts
(3) Used to build the P5
Datapoints, configuration of Finite State Machine hierarchy and logical view, addresses for PLC, smoothing, archive parameters, etc.
In short, the full system distributed on several PCs
(4) Sophisticated checkout routines to compare with reality!
PP1 checkout!
(5) Dynamic mapping of detector to power supply via DCU info

20. 4 SY 1527 Power Supply Controllers Display Terminal (Control Room)P5
4 SY 1527 Power Supply Controllers
7 PLC Systems
PSX SERVICE PC
PLC pc
Power Supply PC 1
Power Supply PC 2
Power Supply PC 3
Power Supply PC 4
DCU PC
Supervisor PC
Cron Jobs
Master
PLC
Experiment Private Network
Display Terminal (Control Room)
SSH tunneling
Tested thoroughly during commissioning in TIF
Installed
at P5
with software
General Purpose CERN network
Display Terminals (Tracker Control 186)

21. TCS & TSS – a summary in numbers
TCS: Tracker Control System
Control, Monitoring, Analysis, Trending and Archiving of
Detector
Power Supplies
Environmental sensors
Temperature; humidity
Auxiliary systems:
Cooling Plant, BCM, Magnet, Thermal Screen
DCU information from XDAQ
Based on
LHCC JCOP framework, PVSS, proper LAN connection
Obeys CMS DCS or TK_RCMS (local running)
Includes dedicated access control
TSS: Tracker Safety System
Autarkic PLC system on UPS, interlocks power supplies on basis
Temperatures from hardwired sensors
Small number of RH probes on exhaust pipes
CMS DSS System
 104 param
~ 10 PCs
~ 2000 PSU
~ 300 Ctrl PS
~ 1000 hardwired probes
~ DCUs
 10³ param
 105 param
Ah, Yes, it works
Tracker has sophisticated scale up knowledge, now transfered to other sub detectors
6 large PLC racks
1 PLC Master System
~ 1000 hardwired probes
We sucessfully controled 25% of the Tracker during commissiong in the Tracker Integration Faccility

22. Responsibilties of Karlsruhe
Design and realization of the TCS system
Final consistent GUI for Control room
Full CMS & LHC compatibility
Interfaces to TSS incl. TSS planning and mapping
Planning and installation of power supply system and TSS to power system
Commissioning of the Tracker with respect to control and safety (This would be a talk for itself)
Also cable connection and consistency check with TCS
Everybody is invited to have a thorough look and understanding of the system to become an expert for the operation period
DCSM Zeuthen

23. Backup
DCSM Zeuthen

24. How is our hierarchy structured II?
TOB:
688 PG
44 CL
92 CG
TIB/TID
488 PG
72 CL
120 CG
TEC (both TECs)
16 sectors
64 CL
144 CG
768 PG
PG:
1 PSU (LV&HV)
1 FE DCU
Total:
CL: 180
PG: 1944
CG: 356

25. Reminder: Granularity of SWITCH OFF
Shutdown due to DCU analysis
Shutdown due to PLC interlock
TOB
13P-1C
15P-1C
17P-1C
TOB
13P-1C
15P-1C
17P-1C
TOB
13P-1C
15P-1C
17P-1C
Shutdown due to PLC analysis

26. Solution: majority voting
95 %
HV_mixed
Mixed states: needed for security
We need to know if any channel in that state
Main issue: deal with inclusion and exclusion of subtrees
Computation of the percentages allows computing of the state overriding the FSM logic
ON_LV
95 %
LV_mixed
ON_CTRL
95 %
CTRL mixed
OFF
ERROR
5 %

27. TCS 3D
During MTCC in the central DCS window the 3D display of CMS including the Tracker was tested
The colors reflect the states of the module
Clicking leads to the corresponding FSM panel
We can go to the granularity of the module and link to the DCU information
DCSM Zeuthen

28. Shifter „immediate“ action, when a channel makes trouble
Do OFF & Exclude/Disable
Click Here
Expert comes later and follows the yellow sign
All panels look similar
Expert panels available to experts
Small command set
ON, ON_LV, ON_CTRL, OFF; STANDBY
Small state set on TOP node:
ON, OFF, STANDBY, ERROR