1. CMS Operational Experience
Oliver Holme
On behalf of those who built, maintain & support CMS DCS
JCOP Workshop
4th November 2015

2. CMS DCS architecture

3. Current CMS DCS overview
Distributed system of 34 WinCC OA systems running on 28 DELL Blades
Runs on Windows Server 2008 R2
WinCC OA redundancy
A CMS component customises WinCC OA to make it work for us
Several JCOP components were upgraded to make this possible
68 WinCC OA projects on 56 DELL blades
Resilience to computer failures versus increased system complexity
Access control
Access control server is used to synchronise data across all systems
Group membership is currently stored outside of WinCC OA
New solution with e-groups puts all configuration outside of WinCC OA

4. Production deployment
Sub-system developers/experts have no access to DCS computers
Everything must be packaged as a JCOP component
Components are read from SVN
Production configuration is versioned and can be rebuilt if needed
Installation Tool and Configuration DB System Information are critical
Hardware and logical views are stored in the ConfigDB
No standard solution to store FSM trees
CMS custom XML export/import for the FSM
CMS install library standardises and simplifies postInstall scripts:
Add/stop/restart drivers and managers
Target ConfigDB configurations to the system
Defined in a custom XML file – only one line of code needed

5. Automation and protection
LHC handshake and beam modes trigger automated DCS actions
CMS sub-detectors are powered ON via the FSM
A stronger action is preferred to power OFF
Not safety, but as close as possible in DCS → CMS protection mechanism
Channels are powered OFF, locked at the DP level and a confirmation is issued

6. Remote access
Sub-system experts can open UIs via a Windows Terminal Server
Every CMS member can monitor DCS via CMSONLINE
CMSONLINE – visualisation & expert tools:
FSM tree and states
Sub-detector screenshots
Plotting of historical data
Alert screen
Project console
Task Manager
Log files
Trigger new component installations
View/filter/control managers across many projects
Central logging used to collect logs
Does not always show what we need (debugs or missed messages)

7. CMS specific implementations
FSM tree cache
FSM tree is slow to navigate via standard FSM library functions
Get parent, get children etc.
A CMS component creates a cache of the tree which is fast
FSM majority (not really a majority)
Uses cache to determine number of devices which are in a state
Used to show shifter the % of HV channels ON in each sub-detector
FSM handling of ConfigDB recipes – not the JCOP version
Verification of successful recipe application
Wait and check hardware readback values
Recipe cache ↔ DP consistency check
Recipe cache ↔ ConfigDB consistency check

8. Shifter Monitoring Alarms FSM
Standard JCOP alarm screen with sub-detector grouping of alarms
Small extension to announce new alarms in the control room
Global and local (sub-system) alarm concept used
Web-based alarm help stored in DB
Delayed alerts – triggered after a condition is present for a certain time
Limited use of summaries and alarm reduction
FSM
Standard interface states/actions between central and sub-systems
Experience shows that writing valid & correct FSM logic is hard
Behaviour at logical object level is relatively complex and limited
CMS tool to check the logic for problems

9. System Monitoring Not using System Overview
Many solutions already existing in CMS
Implemented with:
Custom API manager – publishes process info, allows to stop processes
Cronjobs – checks such as HW & DIP communication, RAIMA corruption etc.
CMSONLINE tools – visualisation and control of managers across projects
CMS tool to link SMS/ s to alarms
Messages can be sent to individuals or groups

10. Experience with JCOP tools

11. CMS experience – JCOP FW
Experience of core functionality is good
Libraries of tools and functions are useful to speed up development
Some trouble with documentation
Hard to know what functionality is available
Lack of realistic sample code
Generic “one function for all situations” can be heavy/complex
e.g. configs
Sometimes it feels easier to write it yourself

12. CMS experience - Devices
CAEN, Wiener and ELMBs are used in CMS
Very quick to get something working
OPC servers
Device models
Integration into Device Editor & Navigator
A convenient place to display only DCS DPs (better than PARA)
Some limitations
Not straightforward to put new DP types into Device Editor & Navigator
Device definition contents is hard to configure
Could it be in a more open format like XML?
Currently its functionality is limited by its DP type
OPC server & hardware interface changes caused issues
Lots of support needed over the years

13. CMS experience - ConfigDB
To store hardware/logical view (static configuration)
To store recipes
Static configuration saving is heavy (lots of support needed), but worth it
Devices must be created in WinCC OA first of all
Methods to create the initial configuration vary (other DBs, files, scripts, by hand)
Static configurations are difficult to inspect (check validity/completeness)
CMS tools make it easy to re-save predefined lists of devices
Recipes are very widely used
Applying “less static” information during installation
Mode changes
Assuring correct hardware settings after interventions

14. CMS experience - Trending
Mixed opinions
Can provide a quick way to generate/manage plots
Can be complicated and limiting
Struggle with limitations versus effort of using WinCC OA widget
Trending Tool does not support data analysis in plots (fitting etc.)
ROOT features in WinCC OA CTRL or export of data to ROOT
CMSONLINE has its own plotting tool
Balance the functionality on the WinCC OA plots versus web plots

15. What is missing? Improved development tools
Improved editor (e.g. in the direction of the Eclipse work of JCOP)
Better checks of code at development time
More support for test creation and testing environments
Object-oriented script language and/or better interface to DPs
Graphical representation (& editing) of FSMs
State diagrams / Richer tree visualisation (node names + types)
Better tools for mass DP configuration
Both in panels and with CTRL functions
Avoid having multiple custom scripts with FOR loops
Standardised procedure for installation (postInstall) & release management
e.g. clean installation versus upgrades

16. Thanks A big thank you to everyone involved in JCOP for
Tools
Training
Support
A large contribution to success of CMS DCS
Many critical components are from JCOP
Reliable JCOP products – DSS & GCS
Continued collaboration & support will be essential
We all have a part to play in the future success of JCOP

17. Questions???