CMS Operational Experience Oliver Holme On behalf of those who built, maintain & support CMS DCS JCOP Workshop 4th November 2015

CMS DCS architecture

Current CMS DCS overview Distributed system of 34 WinCC OA systems running on 28 DELL Blades Runs on Windows Server 2008 R2 WinCC OA redundancy A CMS component customises WinCC OA to make it work for us Several JCOP components were upgraded to make this possible 68 WinCC OA projects on 56 DELL blades Resilience to computer failures versus increased system complexity Access control Access control server is used to synchronise data across all systems Group membership is currently stored outside of WinCC OA New solution with e-groups puts all configuration outside of WinCC OA

Production deployment Sub-system developers/experts have no access to DCS computers Everything must be packaged as a JCOP component Components are read from SVN Production configuration is versioned and can be rebuilt if needed Installation Tool and Configuration DB System Information are critical Hardware and logical views are stored in the ConfigDB No standard solution to store FSM trees CMS custom XML export/import for the FSM CMS install library standardises and simplifies postInstall scripts: Add/stop/restart drivers and managers Target ConfigDB configurations to the system Defined in a custom XML file – only one line of code needed

Automation and protection LHC handshake and beam modes trigger automated DCS actions CMS sub-detectors are powered ON via the FSM A stronger action is preferred to power OFF Not safety, but as close as possible in DCS → CMS protection mechanism Channels are powered OFF, locked at the DP level and a confirmation is issued

Remote access Sub-system experts can open UIs via a Windows Terminal Server Every CMS member can monitor DCS via CMSONLINE CMSONLINE – visualisation & expert tools: FSM tree and states Sub-detector screenshots Plotting of historical data Alert screen Project console Task Manager Log files Trigger new component installations View/filter/control managers across many projects Central logging used to collect logs Does not always show what we need (debugs or missed messages)

CMS specific implementations FSM tree cache FSM tree is slow to navigate via standard FSM library functions Get parent, get children etc. A CMS component creates a cache of the tree which is fast FSM majority (not really a majority) Uses cache to determine number of devices which are in a state Used to show shifter the % of HV channels ON in each sub-detector FSM handling of ConfigDB recipes – not the JCOP version Verification of successful recipe application Wait and check hardware readback values Recipe cache ↔ DP consistency check Recipe cache ↔ ConfigDB consistency check

Shifter Monitoring Alarms FSM Standard JCOP alarm screen with sub-detector grouping of alarms Small extension to announce new alarms in the control room Global and local (sub-system) alarm concept used Web-based alarm help stored in DB Delayed alerts – triggered after a condition is present for a certain time Limited use of summaries and alarm reduction FSM Standard interface states/actions between central and sub-systems Experience shows that writing valid & correct FSM logic is hard Behaviour at logical object level is relatively complex and limited CMS tool to check the logic for problems

System Monitoring Not using System Overview Many solutions already existing in CMS Implemented with: Custom API manager – publishes process info, allows to stop processes Cronjobs – checks such as HW & DIP communication, RAIMA corruption etc. CMSONLINE tools – visualisation and control of managers across projects CMS tool to link SMS/Emails to alarms Messages can be sent to individuals or groups

Experience with JCOP tools

CMS experience – JCOP FW Experience of core functionality is good Libraries of tools and functions are useful to speed up development Some trouble with documentation Hard to know what functionality is available Lack of realistic sample code Generic “one function for all situations” can be heavy/complex e.g. configs Sometimes it feels easier to write it yourself

CMS experience - Devices CAEN, Wiener and ELMBs are used in CMS Very quick to get something working OPC servers Device models Integration into Device Editor & Navigator A convenient place to display only DCS DPs (better than PARA) Some limitations Not straightforward to put new DP types into Device Editor & Navigator Device definition contents is hard to configure Could it be in a more open format like XML? Currently its functionality is limited by its DP type OPC server & hardware interface changes caused issues Lots of support needed over the years

CMS experience - ConfigDB To store hardware/logical view (static configuration) To store recipes Static configuration saving is heavy (lots of support needed), but worth it Devices must be created in WinCC OA first of all Methods to create the initial configuration vary (other DBs, files, scripts, by hand) Static configurations are difficult to inspect (check validity/completeness) CMS tools make it easy to re-save predefined lists of devices Recipes are very widely used Applying “less static” information during installation Mode changes Assuring correct hardware settings after interventions

CMS experience - Trending Mixed opinions Can provide a quick way to generate/manage plots Can be complicated and limiting Struggle with limitations versus effort of using WinCC OA widget Trending Tool does not support data analysis in plots (fitting etc.) ROOT features in WinCC OA CTRL or export of data to ROOT CMSONLINE has its own plotting tool Balance the functionality on the WinCC OA plots versus web plots

What is missing? Improved development tools Improved editor (e.g. in the direction of the Eclipse work of JCOP) Better checks of code at development time More support for test creation and testing environments Object-oriented script language and/or better interface to DPs Graphical representation (& editing) of FSMs State diagrams / Richer tree visualisation (node names + types) Better tools for mass DP configuration Both in panels and with CTRL functions Avoid having multiple custom scripts with FOR loops Standardised procedure for installation (postInstall) & release management e.g. clean installation versus upgrades

Thanks A big thank you to everyone involved in JCOP for Tools Training Support A large contribution to success of CMS DCS Many critical components are from JCOP Reliable JCOP products – DSS & GCS Continued collaboration & support will be essential We all have a part to play in the future success of JCOP

Questions???