1. Alarm Training Making the transition from alh to CSS Alarm Tools
EPICS Collaboration Meeting Aix-en-Provence
June 1st , 2010
Matthias Clausen - DESY

2. Alarm Training
Record Processing
Monitor Types
Setting Alarms
Alarm Handling

3. EPICS Record Processing
EPICS (Experimental Physics and Industrial Control System)
Client
(Configuration- and Operator Interfaces)
Channel Access
IP based Communication Protocol
I/O Controller is independent from clients
I/O Controller
(control loops, Sequence programs)
I/O System
(Field bus) P T

4. EPICS Information exchange
Client initiating basic requstes: get/ set/ monitor
Client
Register for Changes
Display-Deadband
Archive-Deadband
Alarm-Status
Sending request via Channel Access
Register Client for chanegs in specified record for Alarm changes
Whenever Displa/ Archive/ Alarm-Deadband/ Status changes -> check whether clients have been registered.
If true: -> send updated data P T

5. EPICS default: focus on ‘requesting’ data -> not ‘expecting’ data
How to stay ‘in synch’ with alarm deadbands/ status in IOC?
Client initiating basic requstes: get/ set/ monitor P T
get/
set
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status

6. Introducing a Message based Protocol: JMS (keeping Channel Access for ‘basic’ monitors)
Basic: get/ set/ monitor
InterConnection
Server P T
register
for
monitors
Accepting any alarms from IOC
New/ Changed
Alarm Limits
Alarm/Status
get/
set

7. JMS: For Alarms and all Log-based Messages
Basic: get/ set/ monitor
InterConnection
Server P T
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
get/
set
SysLog-Messages
PutLog-Messages
SNL-Log-Messages

8. Existing Log Server Implementations
Sys-Log
Message
Server
Put-Log
Message
Server
Basic: get/ set/ monitor P T
register
for
monitors
get/
set
SysLog-Messages
PutLog-Messages

9. JMS does not add more levels of complexity (just new ones …)
Basic: get/ set/ monitor
InterConnection
Server P T
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
get/
set
SysLog-Messages
PutLog-Messages
SNL-Log-Messages

10. How to implement the new Logging Mechanism
Using the existing Alarm Hook in Record Processing
Using the existing Hooks in System Logging
Using the existing Hooks in (ca)Put Logging
Using the new Function in SNL Logging

11. Changes in iocCore for Alarm Logging
very small hook in recGbl  recGblResetAlarms
/* Hook Routines */
RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL;
loading and starting the alarm logging software redefines recGblAlarmHook
if(sevr!=nsev || stat!=nsta) {
/* HOOK for alarm logging */
if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr);

12. implenented as ring buffer
Alarm Logging
The special queue has sequential input ....
add
Hook Routine
replace
SCAN context
and parallel access if space is rare and the record is stored in the overflow area.
implenented as ring buffer
Control task
Message tasks
Beacon task
Port B
Port A

13. EPICS Channel-Access-Put-Logging timeout index queue
register Hook Routine for caPut operation
asTrapWriteRegisterListener(caPutLog2Array)
caPutLog2Array (...) (caller context)
caPutLog2Q (...) (interrupt context)
caPutLogTask ()
write caPut-Message into Array
timeout
copy into Queue
index queue
send Message to Network

14. SNL Log ‘Just’ a new supported function in the snl executive.
Code example:
when(TV001zu){
snlMsgLog( SEVR_INFO, "<Start> TV001 has been closed");
} state Auf50
Sent to the log table in CSS.
Different severy types can be set -> used to color-code the messages

15. EPICS IOC-Logging Overview Alarms caPut SysLog SNL-Log ioc log control
Command
Reply
Beacon
Alarm
caPutLog
sysMsg
snlMsg
Alarm
ICS
ICS
ICS

16. Alarm Table

17. Looking back into the Archive (24h)

18. Selecting the Time Span

19. Any Message in the System

20. Log Messages (System, Put, SNL)

21. Properties View (default Eclipse view)

22. Alarm System - Alarm Tree view
Shows the current status of the alarm state from the IOC
Delete and create records and subcomponents by context menu
Configuration-Changes are stored in the LDAP server (or in XML file)
Alarm status is propagated to root component
Property view to display and edit tree items

23. Demo Alarm Views
Changing / setting alarm Limits
Severity Settings
Different kind of Displays

24. How to get there without JMS?
The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources:
JMS
DAL (Channel Access)
(currently only available in the development environment foreseen for the next release)

25. How to display an Alarm Tree without LDAP?
Use the new alternative to configure your LDAP Tree from an XML file
Create XML file by Hand
Create XML file from DCT (next release)
Create XML file from ‘your’ database creation tool
Use LDAP Tree to manually configure your tree and save your configuration from tree ‘save as’

26. How to integrate IOC’s without iocLogClient-Support?
A new headless application will handle this:
Reading an XML config file
Use channel access to register for Alarm Changes
Send alarm changes to JMS

27. The transition Plan:
Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree
Get your configuration file e.g. from DCT
Get the hierarchy visualized in the Alarm Tree
Get all configured alarms in the Alarm Table
Install JMS Servers
Immediately get your CSS log messages into you CSS Alarm views
Use dal2jms to send Alarm State changes to JMS
Install iocLogClient-Support on your IOCs
Install interConnection-Server(s)
Get ALL your messages (alarm and log) from your IOCs
Install jms2Ora (rdb) to store your messages

28. Alarm System - Acknowledgement
CSS Instance
Acknowledge
Alarm message
Ack.
Message
(JMS)
Update
Ack
Persistant Store (LDAP)
JMS Server
Ack
Ack
Ack
Ack
CSS Instance
CSS Instance
CSS Instance
CSS Instance

29. Alarm System - Alarm Management System (AMS)
Message
(JMS)
CSS Alarm
Configu-
rator
Filter
Manager
Filter
Write
Configuration
Read
configuration
Action DB
JMS
JMS
JMS
SMS Connector
Voice Mail Connector
Mail Connector
Voice
Mail
SMS
Mail

30. Alarm Management System
Slides from Markus Möller

31. Filter-Konfiguration Filter-Konfiguration
AMS - Overview
Message
Sources
Derby Db
Filter-Konfiguration
ActiveMQ
JMS-Server
DepartmentDecision
(FilterManager)
ALARM T
T_AMS_CON_SMS T
SmsConnector
T_AMS_CON_MAIL T
Connector
T_AMS_MESSAGEMINDER T
T_AMS_CON_JMS T
JmsConnector
ActiveMQ
JMS-Server
MessageMinder
T_AMS_DISTRIBUTE T
T_AMS_CON_VOIC T
Voic Connector
Derby Db
Filter-Konfiguration
Distributor

32. AMS - Applications • DepartmentDecision (FilterManager)
Compares the incoming message with the filter configuration
• MessageMinder
Reduces message storms
• Distributor
Creates the alarm messages and sends them to the responsible Connector
• Connector
Sends the message to the user via
SMS
telephone call
JMS message producer to a user defined JMS topic

33. User – Configuration (GUI)

34. User – Configuration • Name • Status code • Rubrik (rubric, category)
- The name of the user
• Status code
- To deactivate a user or a group
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Confirm code
- To confirm a alarm chain
• User is active
- Only active users receive a alarm message •
- address of the user
• SMS number
- Number of the user‘s cell phone
• Voic number
- Number that is used for the telephone call
• Prefered alarming type

35. Group – Configuration (GUI)

36. Group – Configuration • Name • List of group members
- The name of the group
• List of group members
- Every member of a group can be set active
or not active
- Only active member of a group receive
messages
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Minimale Anzahl aktiver Mitglieder
- Minimum number of active users in this group
• Wartezeit bis Rückmeldung
- Number of seconds to wait for an alarm chain
answer
• Alarmgruppe aktiv
- Only the member of an active group receive
messages

37. Condition – Configuration (GUI)

38. Condition – Configuration
• Name
- The name of the condition
• Compare Key
- Name of the message property that is
compared by this condition
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Operator
- Type of operator that is used to compare the
value
• Description
- Description for this condition
• Compare Value
- The value that is used for comparison
• Filtertype
- Type of this filter
(String, String Array, PV, Time based)

39. Filter – Configuration (GUI)

40. Filter – Configuration
• Name
- The name of the condition
• List of receivers
- Name of the receiver
- Type of action (SMS, Mail, Voic , JMS)
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Operator
- Type of operator that is used to compare the
value
• Templates
- Message template (not used yet)
• Compare Value
- The value that is used for comparison
• Default Message
- Alarm message that will be send to the users
• Filterconditions
- The configuration / combination of the conditions

41. Questions?

42. Active ICS Connection Overview ICS: InterConnection Server (between IOC and JMS)

43. Disabling Messages (for Testing/ Reboot …)

45. Alarm system - Structure
Alarm / Log message
Sources
EPICS IOC
D3 PCM
CSS Instance
Updated from IC
Archive DB
Persistent Store (LDAP)
JMS Server
Alarm Management
System
CSS Alarm
Tools
(Views, Con-
figuration, …)
Message
Table
Message
Archive
SMS
Mail
AMS
Configuration
Alarm Tree

46. Message sources JMS Communication
EPICS IOC and D3 PCM send alarm messages in special format
Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format
CSS uses log4j and sends log messages in JMS format
Generic message system for alarm messages
Easy to add other sources
EPICS IOC
D3 PCM
Special Format
Special Format
JMS Communication
Interconnection
Server
D3 Alarm
Reader
Other Sources
JMS Server
(Active MQ)
CSS Instance

47. Alarm System - AMS Filter
Checks if the filter matches
Creates a new message with the relevant information of the alarm message
Forwards the message to an action
Filter condition:
A Filter is a combination of filter conditions
Filter conditions can be connected with AND and OR
Available condition types are: Compare strings, Check current PV, Time based condition, …

48. Alarm System - AMS operators and groups
Receive alarm messages via mail, sms, …
Status active or inactive can be set
PIN Code to acknowledge alarm messages
Groups:
Operators responsible for specific facilities
Defines priority who should be informed first, second, …
Maximum delay for acknowledgment of alarm messages

49. Logging : 2X Log Server Verbindung
CSS mit direct Channel Access (DAL) connection
As a transition to JMS
Demo DAL
Demo JMS
ICS und JMS Folien
AMS Folien
Config in CSS Log Table  Topics

50. Application Development
Integrating new I/O devices
Configuration of I/O Hardware
And Process Controls
Programmer
Hardware Engineer
Process Engineer
Prepare new
IOC
Load Image
Get driver
from
EPICS site
Configure
Profibus
Configure
Database
Profibus
Configuration
Tool
Database
Creation
Tool
Write text file
to load
directory
Load IOC with
-Operating System
-EPICS core
-Database