1. The DØ Control System J. Frederick Bartlett For The DØ Controls Group

2. Outline  Fermilab and the D0 Experiment  The DØ Control/Monitoring System  IOC extensions  Process Variable Naming Conventions  Database Management  Significant Event System  Detector Configuration Management  Operator Display Support Library

3. The Fermilab Collider Complex DØ CDF

4. DØ Detector Scintillating Fibers Liquid Argon/Uranium Calorimeter Central Muon Layer A Silicon Tracker Superconducting Solenoid Central Muon Layer B Central Muon Layer C Forward Muon Layer C Muon Toroid

5. DØ Run II Control System  EPICS-based controls  Field buses VME MIL/STD1553B CANBUS (Run IIB)  Size  ~15 host-level processors  ~100 IOCs (Input/Output Controllers) MVME2301, MVME2304, MVME5500  ~6600 high-level devices  ~138000 process variables (records)

6. DØ Run II Control System  15 major detector sub-systems  Host-Level processes written in Python  Source management - CVS  Operating systems  Host processors - Linux  IOC processors – vxWorks  Software versions  Now - EPICS 3.14.6  Next - EPICS 3.14.7 with patches, using SNS linux build tools  Controls staff  Core system - 3 FTEs (4 people)  Detector-specific components - ~2 FTEs Primarily from other institutions

7. Devices and Records DetectorDevicesRecords Argon/Uranium Calorimeter8449219 Scintillating Fiber Tracker44535900 Muon Spectrometer157220027 Silicon Microstrip Tracker223545706 Controls4403775 Trigger Systems69421644 Other Systems4682693 Total6653138964

8. EPICS IOC Extensions Channel Access Database Access Record Support Device Support Driver VME + MIL/STD1553 Scanners DB Load Ethernet LAN IOC Database Customizing Elements MIL/STD1553 Bus and CANBUS VME Bus Access With Retries HV Generator State Machine Off On OfflineDisabled Average Holding Tripped Paused Locked Start Begin Ramp End Ramp Mid Ramp

9. Gateways to Other Systems  Accelerator  Gateway link to ACNET system  Bidirectional  Data access only (no control)  Cryogenics and Gas  Gateway link to DMACS system  Read-only  Data access only (no control)

10. ACNET Gateway xmlrpc Server ToACNET FromACNET ToFromITC ToEPICS FromEPICS CACHECACHE REMOTEREMOTE EXPORTEXPORT xmlrpc ITC CLIENTS CA EPICS RECORDS CLIENTS xmlrpc CLIENTS D0BD

11. DØ Naming Convention  Name elements  Detector CAL  Sub-det N  Device VBD  Locator 01  Attribute STATUS  I/O W  Field SCAN  Template [ ]_ _ / [: ] [. ] CALN_VBD_01/STATUS:W.SCAN Frequently used in filters Many utility tools use this field for device-specific actions

12. Database Management  Basic EPICS uses flat text files  Template definition file Multiple record definitions with substitution parameters  Template generator file Create a group of records (instance of a template) with values for parameters  Record instance file Instances of records read by IOC to create record database

13. Relational Database System  EPICS database creation from text files  Record structure defined by DBD files  Template structure defined by template definition files  Template instances defined by template generator files or from Browser GUI  IOC record instances (record instance file) are extracted from database  Database support programs written in Python

14. Relational Database System Oracle Hardware Database Template Definitions DB Creation Record Field Definitions Record Extract Template Generators List of Records For IOC IOC Id Template Extract Template Generators For IOC Instance Creation

15. Significant Event System  The significant event philosophy  Alarms are a only a sub-set of the significant events  EPICS does not generate all of the significant events of interest  Alarm utilities enhance reliability Detect impending failures and fix them before they fail Minimize the time to correct failures  Archive all event transitions The archive is a history of the state transitions of the experiment Tools available to search the event archive What was your system doing last month?

16. Alarm/Event Distribution  SES is a server-based, event (alarm) system:  IOCs and user processes connect to and send alarm transitions to the server Pushed by sources not pulled by the server or client  Server holds the current experiment (alarm) state  Server has one or more filters for each receiving client Makes use of name structure  Rapid display startup of receiving clients  User processes may also declare events via API (C, C++, Python)  Written in Python

17. Significant Event System Significant Event Server EPICS IOC Process Watcher Periodic Heartbeat Alarm Display F Alarm Watcher F Run Suspend Run Control (COOR) F F Filter SE Message Filtered Message seBrowser seLogger F EPICS IOC Process

18. SES Alarm Display Select Guidance button Events in line selected by filter Select minor alarm button

19. Detector Configuration Management  EPICS Save-and-Restore is adequate for accelerators  Accelerators are tuned for an operation mode and the configuration is saved for future restoration  Some complex devices like HEP detectors do not work this way  There are too many distinct modes to be saved  Tuning may not be required Settings are often predictable from trigger selection Many detector component settings are taken from separate calibration measurements

20. Comics Server  Manages the configuration of the detector  Receives load requests from:  Run control  Expert GUIs  Program APIs  Load map is a directed graph (tree)  Tree node (intermediate) Establishes a layered hierarchy Establishes an execution order  Action node (leaf) Issues EPICS CA requests

21. Comics Server  Constructed on the server model with multiple clients sending commands  Server may be activated independently for configuring detector component  Coded in the Python language

22. Configuration Management Tree Intermediate Tree Node CFTSMTCALMUO DEV2DEV3 S0 S2 S1 DEV0DEV1 T0 Root Node Configuration Data Action Node Uses EPICS Channel Access Links determine the order of execution DØ

23. Comics Expert GUI

24. Operator Display Library  Standard, process flow (synoptic) displays do not adapt well to the monitoring of HEP detectors  Like accelerators, there are many similar devices that make up the typical detector component (cells in a calorimeter)  The components are not related in a serial fashion like a cryogenic plant  Tabular (spread-sheet designs) are more natural  Similar properties for different devices are easily compared  Deviations are apparent

25. Operator Display Library  DØ has developed a series of Python display classes for building tabular displays that collect information from EPICS process variables

26. Resource Display Device Window Basic Frame with Menu Bar, Status Window, and Button Bar Tabbed Pages with CA Connect and Disconnect Device Analog Property Device Status (Bit-Encoded) Property Right-Click on Name Field