XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Matthias Clausen, DESY XFEL Refrigerator Controls – April Alarm Training Making the transition from alh to CSS Alarm Tools EPICS Collaboration Meeting Aix-en-Provence June 1 st, 2010 Matthias Clausen - DESY

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 2 Alarm Training Record Processing Monitor Types Setting Alarms Alarm Handling

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 3 EPICS Record Processing PT EPICS (Experimental Physics and Industrial Control System) I/O System (Field bus) I/O Controller (control loops, Sequence programs) Client (Configuration- and Operator Interfaces) Channel Access IP based Communication Protocol I/O Controller is independent from clients

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 4 EPICS Information exchange PT Client initiating basic requstes: get/ set/ monitor Whenever Displa/ Archive/ Alarm- Deadband/ Status changes -> check whether clients have been registered. If true: -> send updated data Client Register for Changes Display-Deadband Archive-Deadband Alarm-Status Sending request via Channel Access Register Client for chanegs in specified record for Alarm changes

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 5 EPICS default: focus on ‘requesting’ data -> not ‘expecting’ data PT Client initiating basic requstes: get/ set/ monitor get/ set register for monitors New/ Changed Alarm Limits Alarm/Status How to stay ‘in synch’ with alarm deadbands/ status in IOC?

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 6 Introducing a Message based Protocol: JMS (keeping Channel Access for ‘basic’ monitors) PT Basic: get/ set/ monitor get/ set register for monitors New/ Changed Alarm Limits Alarm/Status Accepting any alarms from IOC InterConnection Server JMS

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 7 JMS: For Alarms and all Log-based Messages PT Basic: get/ set/ monitor get/ set register for monitors New/ Changed Alarm Limits Alarm/Status InterConnection Server JMS SysLog-Messages PutLog-Messages SNL-Log-Messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 8 Existing Log Server Implementations PT Basic: get/ set/ monitor get/ set register for monitors PutLog-Messages SysLog-Messages Sys-Log Message Server Put-Log Message Server

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 9 JMS does not add more levels of complexity (just new ones …) PT Basic: get/ set/ monitor get/ set register for monitors New/ Changed Alarm Limits Alarm/Status InterConnection Server JMS SysLog-Messages PutLog-Messages SNL-Log-Messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 10 How to implement the new Logging Mechanism Using the existing Alarm Hook in Record Processing Using the existing Hooks in System Logging Using the existing Hooks in (ca)Put Logging Using the new Function in SNL Logging

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 11 Alarm Logging Changes in iocCore for Alarm Logging very small hook in recGbl  recGblResetAlarms /* Hook Routines */ RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL; loading and starting the alarm logging software redefines recGblAlarmHook if(sevr!=nsev || stat!=nsta) { /* HOOK for alarm logging */ if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr);

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 12 Alarm Logging Hook Routine Control task SCAN context Message tasks Port BPort A The special queue has sequential input.... and parallel access if space is rare and the record is stored in the overflow area. Beacon task add replace implenented as ring buffer

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 13 asTrapWriteRegisterListener(caPutLog2Array) caPutLog2Array (...) (caller context) caPutLog2Q (...) (interrupt context) caPutLogTask () EPICS copy into Queue register Hook Routine for caPut operation write caPut-Message into Array timeout Channel-Access-Put-Logging index queue send Message to Network

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 14 SNL Log ‘Just’ a new supported function in the snl executive. Code example: when(TV001zu){ snlMsgLog( SEVR_INFO, " TV001 has been closed"); } state Auf50 Sent to the log table in CSS. Different severy types can be set -> used to color-code the messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 15 EPICS IOC-Logging Overview Beacon AlarmscaPutSysLogSNL-Log Alarm caPutLogsysMsgsnlMsg Command Reply ioc log control ICS

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 16 Alarm Table

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 17 Looking back into the Archive (24h)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 18 Selecting the Time Span

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 19 Any Message in the System

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 20 Log Messages (System, Put, SNL)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 21 Properties View (default Eclipse view)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 22 Alarm System - Alarm Tree view Shows the current status of the alarm state from the IOC Delete and create records and subcomponents by context menu Configuration-Changes are stored in the LDAP server (or in XML file) Alarm status is propagated to root component Property view to display and edit tree items

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 23 Demo Alarm Views Changing / setting alarm Limits Severity Settings Different kind of Displays

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 24 How to get there without JMS? The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources: -JMS -DAL (Channel Access) (currently only available in the development environment foreseen for the next release)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 25 How to display an Alarm Tree without LDAP? Use the new alternative to configure your LDAP Tree from an XML file –Create XML file by Hand –Create XML file from DCT (next release) –Create XML file from ‘your’ database creation tool –Use LDAP Tree to manually configure your tree and save your configuration from tree ‘save as’

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 26 How to integrate IOC’s without iocLogClient-Support? A new headless application will handle this: Reading an XML config file Use channel access to register for Alarm Changes Send alarm changes to JMS

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 27 The transition Plan: Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree –Get your configuration file e.g. from DCT –Get the hierarchy visualized in the Alarm Tree –Get all configured alarms in the Alarm Table Install JMS Servers –Immediately get your CSS log messages into you CSS Alarm views –Use dal2jms to send Alarm State changes to JMS –Install iocLogClient-Support on your IOCs Install interConnection-Server(s) Get ALL your messages (alarm and log) from your IOCs –Install jms2Ora (rdb) to store your messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 28 CSS Instance Acknowledge Alarm message Alarm System - Acknowledgement Ack. Message (JMS) Update Persistant Store (LDAP) JMS Server Ack CSS Instance Ack

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 29 CSS Alarm Configu- rator Alarm System - Alarm Management System (AMS) DB Filter Manager Filter SMS JMS Read configuration Action Alarm Message (JMS) Write Configuration SMS ConnectorVoice Mail ConnectorMail Connector JMS Mail JMS Voice Mail

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 30 Alarm Management System Slides from Markus Möller

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 31 AMS - Overview ActiveMQ JMS-Server DepartmentDecision (FilterManager) MessageMinder Distributor Derby Db Filter-Konfiguration Voic Connector Connector SmsConnector JmsConnector ALARM T T_AMS_MESSAGEMINDER T T_AMS_DISTRIBUTE T T_AMS_CON_SMS T T_AMS_CON_MAIL T T_AMS_CON_JMS T T_AMS_CON_VOIC T Derby Db Filter-Konfiguration ActiveMQ JMS-Server Message Sources

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 32 AMS - Applications DepartmentDecision (FilterManager) Compares the incoming message with the filter configuration MessageMinder Reduces message storms Distributor Creates the alarm messages and sends them to the responsible Connector Connector Sends the message to the user via SMS telephone call JMS message producer to a user defined JMS topic

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 33 User – Configuration (GUI)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 34 User – Configuration Name - The name of the user Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) - address of the user SMS number - Number of the user‘s cell phone Voic number - Number that is used for the telephone call Prefered alarming type Status code - To deactivate a user or a group Confirm code - To confirm a alarm chain User is active - Only active users receive a alarm message

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 35 Group – Configuration (GUI)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 36 Group – Configuration Name - The name of the group Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) Minimale Anzahl aktiver Mitglieder - Minimum number of active users in this group Wartezeit bis Rückmeldung - Number of seconds to wait for an alarm chain answer Alarmgruppe aktiv - Only the member of an active group receive messages List of group members - Every member of a group can be set active or not active - Only active member of a group receive messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 37 Condition – Configuration (GUI)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 38 Condition – Configuration Name - The name of the condition Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) Description - Description for this condition Filtertype - Type of this filter (String, String Array, PV, Time based) Compare Key - Name of the message property that is compared by this condition Operator - Type of operator that is used to compare the value Compare Value - The value that is used for comparison

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 39 Filter – Configuration (GUI)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 40 Filter – Configuration Name - The name of the condition Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) Templates - Message template (not used yet) Default Message - Alarm message that will be send to the users List of receivers - Name of the receiver - Type of action (SMS, Mail, Voic , JMS) Operator - Type of operator that is used to compare the value Compare Value - The value that is used for comparison Filterconditions - The configuration / combination of the conditions

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 41 Questions?

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 42 Active ICS Connection Overview ICS: InterConnection Server (between IOC and JMS)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 43 Disabling Messages (for Testing/ Reboot …)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 44

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 45 Alarm / Log message Sources Alarm system - Structure EPICS IOCD3 PCMCSS Instance JMS Server Persistent Store (LDAP) Archive DB CSS Alarm Tools (Views, Con- figuration, …) Message Table Message Archive Alarm Management System AMS Configuration Alarm Tree SMS Mail Updated from IC

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 46 JMS Communication Message sources Special Format Other Sources EPICS IOC and D3 PCM send alarm messages in special format Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format CSS uses log4j and sends log messages in JMS format Generic message system for alarm messages Easy to add other sources EPICS IOC D3 PCM CSS Instance Interconnection Server D3 Alarm Reader Special Format JMS Server (Active MQ)

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 47 Alarm System - AMS Filter Filter: Checks if the filter matches Creates a new message with the relevant information of the alarm message Forwards the message to an action Filter condition: A Filter is a combination of filter conditions Filter conditions can be connected with AND and OR Available condition types are: Compare strings, Check current PV, Time based condition, …

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 48 Alarm System - AMS operators and groups Operators: Receive alarm messages via mail, sms, … Status active or inactive can be set PIN Code to acknowledge alarm messages Groups: Operators responsible for specific facilities Defines priority who should be informed first, second, … Maximum delay for acknowledgment of alarm messages

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 49 Logging : 2X Log Server Verbindung CSS mit direct Channel Access (DAL) connection As a transition to JMS Demo DAL Demo JMS ICS und JMS Folien AMS Folien Config in CSS Log Table  Topics

Matthias Clausen, DESY XFEL Refrigerator Controls – April 27, 2010 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 50 Application Development Integrating new I/O devices Configuration of I/O Hardware And Process Controls Programmer Process Engineer Get driver from EPICS site Prepare new IOC Load Image Configure Database Creation Tool Write text file to load directory Load IOC with -Operating System -EPICS core -Database Hardware Engineer Profibus Configuration Tool Configure Profibus