Controls and Monitoring Implementation Plan J. Leaver 03/06/2009

13/06/2015Imperial College 2 Implementation Issues Organisation & responsibilities General EPICS infrastructure –EPICS server / client organisation –Unification of control systems Remote access –Monitoring –Controls Configuration database Schedule

13/06/2015Imperial College 3 Organisation of Control Systems Original plan was for Daresbury Lab (DL) to provide all controls for the experiment –DL responsible for many existing C&M systems (excellent quality) Unfortunately, recent funding issues have limited collaboration’s ability to pay DL for new work –DL to continue with current projects (where possible) –MICE community to take responsibility for additional C&M systems

13/06/2015Imperial College 4 Organisation of Control Systems MICE Online Group (MOG) created in January –Aim: Organise data acquisition, C&M & online reconstruction Controls & Monitoring Leader (JL) –Identify control requirements for each section of MICE –Decide on most appropriate solution –Coordinate the effort of those involved in implementing agreed solution

13/06/2015Imperial College 5 Organisation of Control Systems MOG directly responsible for C&M infrastructure –Network/hardware organisation –Integration of control systems (with each other & the rest of MICE) –User experience (i.e. how operators interact with ‘global’ C&M system) For individual projects, each group within MICE should be responsible for own system(s) –Contributing either EPICS development effort or funds for a 3 rd party (e.g. DL) to complete required work Where necessary, MOG contributes developer effort –However, very limited resources available (~1.5 man years per year) –Currently seeking additional support within the community

13/06/2015Imperial College 6 EPICS Client / Server Overview

13/06/2015Imperial College 7 EPICS Server / Client Organisation Wide variety of EPICS server applications permitted –Typically connect to physical hardware Impossible to enforce common interface/processor/OS specifications –Each server maintained by ‘owner’ of respective control system Strict central administration unnecessary – ‘end user’ only concerned with availability of PVs on network EPICS clients also varied, but must be uniformly accessible –Users should not have difficulty finding/launching clients –Applications should be consistently organised/updated –MOG responsibility

13/06/2015Imperial College 8 EPICS Client Organisation All client-side applications run on miceecserv –Central installation repository greatly simplifies configuration/maintenance/backup –MOG collates individual applications, applies updates when available from control system ‘owners’ miceecservmiceopi1miceopi2 EPICS IOC Portable CA Server EPICS IOC Controls Network EPICS server applications EPICS client applications

13/06/2015Imperial College 9 EPICS Client Organisation Client control/monitoring GUIs viewed directly on miceecserv, or one of 2 ‘Operator Interface’ PCs –OPI PCs act as ‘dumb terminals’, running displays from miceecserv via SSH miceecservmiceopi1miceopi2 EPICS IOC Portable CA Server EPICS IOC Controls Network EPICS server applications EPICS client applications

13/06/2015Imperial College 10 Unification of Control Systems At user level: Simple ‘wrapper’ GUI provides menu for launching individual client applications At system level: Employ 2 standard EPICS tools (running as background services on miceecserv) –Alarm Handler Monitors all servers & warns operators of abnormal/dangerous conditions –Channel Archiver Automatically records PV parameters to disk & provides several visualisation options See P. Hanlet’s talk

13/06/2015Imperial College 11 User Interface

13/06/2015Imperial College 12 User Interface Large wall-mounted display Alarm Handler Message log Any important parameters for current run

13/06/2015Imperial College 13 User Interface Client application launcher Standard desktop monitor Client GUI

13/06/2015Imperial College 14 User Interface Connected to miceecserv

13/06/2015Imperial College 15 User Interface Connected to miceopi1 Connected to miceopi2

13/06/2015Imperial College 16 Remote Monitoring: General Principles Remote users should have simple, easily accessible interface for routine monitoring ‘Expert’ remote users should have access to monitoring displays which match those in MLCR No machine on Controls Network should be directly accessible over the internet System load generated by remote monitoring should have minimal impact on control & monitoring services

13/06/2015Imperial College 17 Remote Monitoring: Web Server miceecserv EPICS IOC Portable CA Server EPICS IOC RAL Gateway Channel Archiver Web Server PV Archive Data Server CGI Export Controls Network Java Archive Viewer Web browser NFS Mount PPD Network Internet

13/06/2015Imperial College 18 Remote Monitoring: Direct PV Access Could recreate normal client displays using web interface, but would involve impractical development overheads –Provide direct read only access to PVs so actual client GUIs may be run remotely miceecserv EPICS IOC Portable CA Server EPICS IOC RAL Gateway Controls Network Standard client GUI running on remote PC (read only) CA Gateway (read only)

13/06/2015Imperial College 19 Remote Monitoring: Direct PV Access CA Gateway makes PVs available across subnets (with full access control), while minimising load on underlying servers To simplify end-user support, virtual machine disk image containing EPICS + all client applications will be made available miceecserv EPICS IOC Portable CA Server EPICS IOC RAL Gateway Controls Network Standard client GUI running on remote PC (read only) CA Gateway (read only)

13/06/2015Imperial College 20 Remote Control Where possible, operations affecting the state of any MICE system should only be performed within MLCR –Remote users accessing controls can lead to unknown/unexpected running conditions – should be discouraged If necessary, off-site experts will be permitted to run control client applications on miceecserv, via SSH through RAL Gateway –Each expert will have an account on miceecserv which only contains client applications for their designated system

13/06/2015Imperial College 21 Configuration Database Necessary to integrate control systems with central MICE Configuration Database 1)Read set point values from database 2)Upload PV values to EPICS servers 3)Modify PVs with client GUIs 4)Download PV values from EPICS servers 5)Write new set point values to database For (2) & (4), could use standard EPICS Backup & Restore Tool (BURT) –Backup/restore PV values to/from ‘snapshot’ files However, interfacing snapshot files with database introduces significant overheads –Propose creation of custom backup/restore client

13/06/2015Imperial College 22 Configuration Database Simple client application –Read/write PV values via MICE C++ wrapper for CA C-bindings XML configuration file specifies PV names, correct sequence for write operations –Import/export sets of PV values from/to XML string –Read/write XML string from/to database via Configuration Database API Manual backup/restore –State tagged with time, user-generated identification string, etc. Monitoring of DATE DAQ state –Automatic backup at start of each run

13/06/2015Imperial College 23 Configuration Database Additional requirements –Throughout each DAQ run, all set point values should be held in state defined by the last Configuration Database ‘snapshot’ If values change, system in unknown state  Cannot perform automated analysis of run data –While DAQ in run state, client monitors all set point values –If any parameters are modified Set PV to indicate invalid run state (read into DAQ stream) Set warning on Alarm Handler display

13/06/2015Imperial College 24 Configuration Database Configuration Database interface still in early design stages – work not commenced –J. Leaver/P. Hanlet to develop EPICS client –D. Forrest to implement database API functions for parsing/formatting EPICS set point XML strings Details of run state PV monitoring to be confirmed

13/06/2015Imperial College 25 Infrastructure Schedule