Alarm Training Making the transition from alh to CSS Alarm Tools EPICS Collaboration Meeting Aix-en-Provence June 1st , 2010 Matthias Clausen - DESY

Alarm Training Record Processing Monitor Types Setting Alarms Alarm Handling

EPICS Record Processing EPICS (Experimental Physics and Industrial Control System) Client (Configuration- and Operator Interfaces) Channel Access IP based Communication Protocol I/O Controller is independent from clients I/O Controller (control loops, Sequence programs) I/O System (Field bus) P T

EPICS Information exchange Client initiating basic requstes: get/ set/ monitor Client Register for Changes Display-Deadband Archive-Deadband Alarm-Status Sending request via Channel Access Register Client for chanegs in specified record for Alarm changes Whenever Displa/ Archive/ Alarm-Deadband/ Status changes -> check whether clients have been registered. If true: -> send updated data P T

EPICS default: focus on ‘requesting’ data -> not ‘expecting’ data How to stay ‘in synch’ with alarm deadbands/ status in IOC? Client initiating basic requstes: get/ set/ monitor P T get/ set register for monitors New/ Changed Alarm Limits Alarm/Status

Introducing a Message based Protocol: JMS (keeping Channel Access for ‘basic’ monitors) Basic: get/ set/ monitor InterConnection Server P T register for monitors Accepting any alarms from IOC New/ Changed Alarm Limits Alarm/Status get/ set

JMS: For Alarms and all Log-based Messages Basic: get/ set/ monitor InterConnection Server P T register for monitors New/ Changed Alarm Limits Alarm/Status get/ set SysLog-Messages PutLog-Messages SNL-Log-Messages

Existing Log Server Implementations Sys-Log Message Server Put-Log Message Server Basic: get/ set/ monitor P T register for monitors get/ set SysLog-Messages PutLog-Messages

JMS does not add more levels of complexity (just new ones …) Basic: get/ set/ monitor InterConnection Server P T register for monitors New/ Changed Alarm Limits Alarm/Status get/ set SysLog-Messages PutLog-Messages SNL-Log-Messages

How to implement the new Logging Mechanism Using the existing Alarm Hook in Record Processing Using the existing Hooks in System Logging Using the existing Hooks in (ca)Put Logging Using the new Function in SNL Logging

Changes in iocCore for Alarm Logging very small hook in recGbl  recGblResetAlarms /* Hook Routines */ RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL; ....... loading and starting the alarm logging software redefines recGblAlarmHook if(sevr!=nsev || stat!=nsta) { /* HOOK for alarm logging */ if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr);

implenented as ring buffer Alarm Logging The special queue has sequential input .... add Hook Routine replace SCAN context and parallel access if space is rare and the record is stored in the overflow area. implenented as ring buffer Control task Message tasks Beacon task Port B Port A

EPICS Channel-Access-Put-Logging timeout index queue register Hook Routine for caPut operation asTrapWriteRegisterListener(caPutLog2Array) caPutLog2Array (...) (caller context) caPutLog2Q (...) (interrupt context) caPutLogTask () write caPut-Message into Array timeout copy into Queue index queue send Message to Network

SNL Log ‘Just’ a new supported function in the snl executive. Code example: when(TV001zu){ snlMsgLog( SEVR_INFO, "<Start> TV001 has been closed"); } state Auf50 Sent to the log table in CSS. Different severy types can be set -> used to color-code the messages

EPICS IOC-Logging Overview Alarms caPut SysLog SNL-Log ioc log control Command Reply Beacon Alarm caPutLog sysMsg snlMsg Alarm ICS ICS ICS

Alarm Table

Looking back into the Archive (24h)

Selecting the Time Span

Any Message in the System

Log Messages (System, Put, SNL)

Properties View (default Eclipse view)

Alarm System - Alarm Tree view Shows the current status of the alarm state from the IOC Delete and create records and subcomponents by context menu Configuration-Changes are stored in the LDAP server (or in XML file) Alarm status is propagated to root component Property view to display and edit tree items

Demo Alarm Views Changing / setting alarm Limits Severity Settings Different kind of Displays

How to get there without JMS? The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources: JMS DAL (Channel Access) (currently only available in the development environment foreseen for the next release)

How to display an Alarm Tree without LDAP? Use the new alternative to configure your LDAP Tree from an XML file Create XML file by Hand Create XML file from DCT (next release) Create XML file from ‘your’ database creation tool Use LDAP Tree to manually configure your tree and save your configuration from tree ‘save as’

How to integrate IOC’s without iocLogClient-Support? A new headless application will handle this: Reading an XML config file Use channel access to register for Alarm Changes Send alarm changes to JMS

The transition Plan: Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree Get your configuration file e.g. from DCT Get the hierarchy visualized in the Alarm Tree Get all configured alarms in the Alarm Table Install JMS Servers Immediately get your CSS log messages into you CSS Alarm views Use dal2jms to send Alarm State changes to JMS Install iocLogClient-Support on your IOCs Install interConnection-Server(s) Get ALL your messages (alarm and log) from your IOCs Install jms2Ora (rdb) to store your messages

Alarm System - Acknowledgement CSS Instance Acknowledge Alarm message Ack. Message (JMS) Update Ack Persistant Store (LDAP) JMS Server Ack Ack Ack Ack CSS Instance CSS Instance CSS Instance CSS Instance

Alarm System - Alarm Management System (AMS) Message (JMS) CSS Alarm Configu- rator Filter Manager Filter Write Configuration Read configuration Action DB JMS JMS JMS SMS Connector Voice Mail Connector Mail Connector Voice Mail SMS Mail

Alarm Management System Slides from Markus Möller

Filter-Konfiguration Filter-Konfiguration AMS - Overview Message Sources Derby Db Filter-Konfiguration ActiveMQ JMS-Server DepartmentDecision (FilterManager) ALARM T T_AMS_CON_SMS T SmsConnector T_AMS_CON_MAIL T EMailConnector T_AMS_MESSAGEMINDER T T_AMS_CON_JMS T JmsConnector ActiveMQ JMS-Server MessageMinder T_AMS_DISTRIBUTE T T_AMS_CON_VOICEMAIL T VoicemailConnector Derby Db Filter-Konfiguration Distributor

AMS - Applications • DepartmentDecision (FilterManager) Compares the incoming message with the filter configuration • MessageMinder Reduces message storms • Distributor Creates the alarm messages and sends them to the responsible Connector • Connector Sends the message to the user via SMS e-Mail telephone call JMS message producer to a user defined JMS topic

User – Configuration (GUI)

User – Configuration • Name • Status code • Rubrik (rubric, category) - The name of the user • Status code - To deactivate a user or a group • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Confirm code - To confirm a alarm chain • User is active - Only active users receive a alarm message • E-Mail - E-Mail address of the user • SMS number - Number of the user‘s cell phone • Voicemail number - Number that is used for the telephone call • Prefered alarming type

Group – Configuration (GUI)

Group – Configuration • Name • List of group members - The name of the group • List of group members - Every member of a group can be set active or not active - Only active member of a group receive messages • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Minimale Anzahl aktiver Mitglieder - Minimum number of active users in this group • Wartezeit bis Rückmeldung - Number of seconds to wait for an alarm chain answer • Alarmgruppe aktiv - Only the member of an active group receive messages

Condition – Configuration (GUI)

Condition – Configuration • Name - The name of the condition • Compare Key - Name of the message property that is compared by this condition • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Operator - Type of operator that is used to compare the value • Description - Description for this condition • Compare Value - The value that is used for comparison • Filtertype - Type of this filter (String, String Array, PV, Time based)

Filter – Configuration (GUI)

Filter – Configuration • Name - The name of the condition • List of receivers - Name of the receiver - Type of action (SMS, Mail, Voicemail, JMS) • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Operator - Type of operator that is used to compare the value • Templates - Message template (not used yet) • Compare Value - The value that is used for comparison • Default Message - Alarm message that will be send to the users • Filterconditions - The configuration / combination of the conditions

Questions?

Active ICS Connection Overview ICS: InterConnection Server (between IOC and JMS)

Disabling Messages (for Testing/ Reboot …)

Alarm system - Structure Alarm / Log message Sources EPICS IOC D3 PCM CSS Instance Updated from IC Archive DB Persistent Store (LDAP) JMS Server Alarm Management System CSS Alarm Tools (Views, Con- figuration, …) Message Table Message Archive SMS Mail AMS Configuration Alarm Tree

Message sources JMS Communication EPICS IOC and D3 PCM send alarm messages in special format Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format CSS uses log4j and sends log messages in JMS format Generic message system for alarm messages Easy to add other sources EPICS IOC D3 PCM Special Format Special Format JMS Communication Interconnection Server D3 Alarm Reader Other Sources JMS Server (Active MQ) CSS Instance

Alarm System - AMS Filter Checks if the filter matches Creates a new message with the relevant information of the alarm message Forwards the message to an action Filter condition: A Filter is a combination of filter conditions Filter conditions can be connected with AND and OR Available condition types are: Compare strings, Check current PV, Time based condition, …

Alarm System - AMS operators and groups Receive alarm messages via mail, sms, … Status active or inactive can be set PIN Code to acknowledge alarm messages Groups: Operators responsible for specific facilities Defines priority who should be informed first, second, … Maximum delay for acknowledgment of alarm messages

Logging : 2X Log Server Verbindung CSS mit direct Channel Access (DAL) connection As a transition to JMS Demo DAL Demo JMS ICS und JMS Folien AMS Folien Config in CSS Log Table  Topics

Application Development Integrating new I/O devices Configuration of I/O Hardware And Process Controls Programmer Hardware Engineer Process Engineer Prepare new IOC Load Image Get driver from EPICS site Configure Profibus Configure Database Profibus Configuration Tool Database Creation Tool Write text file to load directory Load IOC with -Operating System -EPICS core -Database