1. EPICS Experience at Fermilab Geoff Savage August 2005 Controls and Monitoring Group

2. 31 Aug 2005MICE DAQ and Controls Workshop2 Outline  Fermilab and DØ  What is EPICS?  EPICS at DØ - RunIIa  EPICS at DØ – RunIIb  EPICS and the CFT  EPICS at Fermilab  EPICS lessons

3. 31 Aug 2005MICE DAQ and Controls Workshop3 Main Injector & Recycler Tevatron Chicago   p source Booster pp p p pp 1.96 TeV CDF DØ Fermilab

4. 31 Aug 2005MICE DAQ and Controls Workshop4 DØ Detector Scintillating Fibers Liquid Argon/Uranium Calorimeter Central Muon Layer A Silicon Tracker Superconducting Solenoid Central Muon Layer B Central Muon Layer C Forward Muon Layer C Muon Toroid

5. 31 Aug 2005MICE DAQ and Controls Workshop5 What is EPICS?  Experimental Physics and Industrial Control System Collaboration Control System Architecture Software Toolkit  Integrated set of software building blocks for implementing a distributed control system  www.aps.anl.gov/epics

6. 31 Aug 2005MICE DAQ and Controls Workshop6 Why EPICS?  Availability of device interfaces that match or are similar to our hardware  Ease with which the system can be extended to include our experiment-specific devices  Existence of a large and enthusiastic user community that understands our problems and are willing to offer advice and guidance.

7. 31 Aug 2005MICE DAQ and Controls Workshop7 EPICS Architecture Input Output Controller (IOC) Operator Interface (OPI) Local Area Network (LAN) OPI – a workstation running EPICS tools. Operating Systems: Linux and Windows XP. IOC – platform supporting EPICS run-time database. Example: VME based PPC processor running vxWorks, Linux workstation. LAN – communication path for Channel Access (CA), the EPICS communication protocol.

8. 31 Aug 2005MICE DAQ and Controls Workshop8 IOC Architecture

9. 31 Aug 2005MICE DAQ and Controls Workshop9 IOC Database  Collection of records  Each record represents a system parameter (process variable, PV) Unique name Set of attributes Attributes and value can be modified  Records must process to do something An input record can read a value every 10 seconds A CA write to an output record causes the record to process Either input or output, not both

10. 31 Aug 2005MICE DAQ and Controls Workshop10 EPICS Records  Input Analog In (AI) Binary In (BI) String In (SI)  Algorithm/control Calculation (CALC) Subroutine (SUB)  Output Analog Out (AO) Binary Out (BO)  Custom – only needed when existing record types or a collection of existing record types are inadequate

11. 31 Aug 2005MICE DAQ and Controls Workshop11 Channel Access IOC (CA Server) LAN EPICS Tool (CA Client) One client can connect to many servers. A channel connects to the value or an attribute of a PV. Each channel has a unique name = record name + attribute. CA Services: Search – find a channel by name Get – retrieve a channel value Put – modify a channel value Add monitor – notification of state change

12. 31 Aug 2005MICE DAQ and Controls Workshop12 EPICS at DØ - RunIIa  Support for new devices and additional field busses  Centralized database with supporting tools for defining the control system configuration  Report EPICS alarms to our global event reporting system  Interface to the Python programming language used to develop host tools for downloading and monitoring  Detector configuration management - COMICS  Archiving  Gateways

13. 31 Aug 2005MICE DAQ and Controls Workshop13 MIL-STD-1553B Bus  Restricted detector access while running  Provides a robust and highly reliable connection to electronics in the remote collision hall  Developed a queuing driver, device support, and a generic record  12 IOCs with ~70 1553 busses from the counting house to the detector and ~10 busses within the counting house

14. 31 Aug 2005MICE DAQ and Controls Workshop14 DØ Specific Records  SVX sequencer - configures and controls the ADCs in the tracking detectors  High voltage Implemented using a state machine model Linear ramping with retries Trip condition recovery Limits control  Rack monitor - Generic I/O module 64 A/D inputs, 4 D/A output, 4 16-bit I/O words  MIL-1553 – debugging/testing on a 1553 bus

15. 31 Aug 2005MICE DAQ and Controls Workshop15 DØ Specific Device Support  Access hardware registers across the MIL-1553 bus and VME backplane using standard EPICS records – long input, long output, …  Support for DZERO specific records Bira 4877 power supplies SVX sequencers Rack monitor  Complex MIL-1553 bus transactions Dual port memory on the AFE boards Write the address to access followed by a read or write of the data

16. 31 Aug 2005MICE DAQ and Controls Workshop16 Describe a Device with Records Template File Generator File EPICS Database EPICS tool Multiple record definitions with substitution parameters This defines a device. Define instances of a template. Assign values to the substitution parameters in the template file. Database Load ASCII file Instances of records read by IOC to create record database.

17. 31 Aug 2005MICE DAQ and Controls Workshop17 Centralized Database System Oracle Hardware Database Template File DB Creation EPICS Database Record Extract Generator File EPICS Database IOC Id Template Extract Generator File Instance Creation Python Scripts Web Access

18. 31 Aug 2005MICE DAQ and Controls Workshop18 PV Naming Convention  [subdet]_ _ / [:io][. field] - Required detector system  CTL, SMT, CFT, CPS, CAL, FPS, ICD, MUO [subdet] - Optional subdetector  North, South, East, West, Forward, Test - Required Device Type  ADC, BLS, HV, PLS, CETEC,... - Required Location indicator  For example a crate number: M201, PC03, F313 - Required device attribute  PEDS, MODE, SFM, CTEC, HEAD, VER,... [:io] - Optional Input/Output designation Direction is from the viewpoint of the front end crate [.field] - Optional Epics field name, default is.VAL  Example: CTL_LV_M201/P5V

19. 31 Aug 2005MICE DAQ and Controls Workshop19 Global Event Reporting  Collect and distribute significant events from detector and DAQ Run control Alarms - data corruption  Added a connection to EPICS Allows for notification of all EPICS alarms Otherwise a user must register an event  Significant Event System (SES)

20. 31 Aug 2005MICE DAQ and Controls Workshop20 SES Server Front Ends Host Apps SE Logger Alarm Watcher Alarm Display COOR L1 L2 L3 Collector /Router Data Logger Disk SE Log File Guidance Files Message Key: SE = Significant Event FIL = Filter configuration RC = Run Control FM = Filtered SE Message = Filter FIL FM RC SE

21. 31 Aug 2005MICE DAQ and Controls Workshop21 Alarm Display

22. 31 Aug 2005MICE DAQ and Controls Workshop22 Python Interface  “Python is an interpreted, interactive, object- oriented programming language.”  A Python interface to Channel Access (C library) has been developed and extended to be almost object oriented  Tkinter and Python Mega Widgets used to develop graphical user interfaces (GUI)  Developed a GUI framework to speed development time and provide a consistent look and feel

23. 31 Aug 2005MICE DAQ and Controls Workshop23 Left click each tab to move between displays. Look for status information. IOC Resource Monitor Display Processor name. Each box is a PV

24. 31 Aug 2005MICE DAQ and Controls Workshop24

25. 31 Aug 2005MICE DAQ and Controls Workshop25 Detector Configuration Management  The COMICS server manages the configuration of the detector  Receives load requests from: Run control Expert GUIs Program APIs  Load map is a directed graph (tree) Tree node (intermediate)  Establishes a layered hierarchy  Establishes an execution order Action node (leaf)  Issues EPICS CA requests

26. 31 Aug 2005MICE DAQ and Controls Workshop26 Configuration Management Tree Intermediate Tree Node CFTSMTCALMUO DEV2DEV3 S0 S2 S1 DEV0DEV1 T0 Root Node Configuration Data Action Node Uses EPICS Channel Access Links determine the order of execution DØ

27. 31 Aug 2005MICE DAQ and Controls Workshop27 Comics Expert GUI

28. 31 Aug 2005MICE DAQ and Controls Workshop28 Archiving Data  Maintain a history of PV values  Channel Archiver – EPICS tool Fast sampling rates Immediate analysis Infrequent access to old historical data Large files Stored to tape once a week  EPICS/Oracle Archiver - DØ Long-term studies Slow sampling rates Easy access to data at any moment Minimal maintenance Data stored directly in an Oracle database

29. 31 Aug 2005MICE DAQ and Controls Workshop29 Gateways  Accelerator Gateway link to ACNET system Bidirectional Data access only (no control)  Cryogenics and Gas systems Gateway link to DMACS system Portable Channel Access Server – EPICS tool Read-only Data access only (no control)

30. 31 Aug 2005MICE DAQ and Controls Workshop30 ACNET Gateway xmlrpc Server ToACNET FromACNET ToFromITC ToEPICS FromEPICS CACHECACHE REMOTEREMOTE EXPORTEXPORT xmlrpc ITC CLIENTS CA EPICS RECORDS CLIENTS xmlrpc CLIENTS D0BD

31. 31 Aug 2005MICE DAQ and Controls Workshop31 EPICS at DØ - RunIIb  CanBus driver for EPICS Weiner power supplies  Ethernet driver for EPICS Gigabit Ethernet communication Raw Ethernet protocol  Both use support from asynDriver – EPICS tool Asynchronous Driver Support “asynDriver is a general purpose facility for interfacing device specific code to low level communication drivers.” “A primary target for asynDriver is EPICS IOC device support but, other than using libCom, it is independent of EPICS.”  Framework reduces development time.

32. 31 Aug 2005MICE DAQ and Controls Workshop32 Support for CFT  Device support for the AFE board using the MIL-1553 driver  AFE and AFE power supply templates  Record and device support for the SVX sequencers  Expert GUI for downloading Does not use COMICS  Calibration software Runs on processor in VRB crate Communicates with AFE

33. 31 Aug 2005MICE DAQ and Controls Workshop33 Analog Front End Fiber Wave guide Cryostat VLPC Analog Front End Stereo Board Cassette MIL-1553 Communication

34. 31 Aug 2005MICE DAQ and Controls Workshop34 SVX Sequencer AFE VBDVBD VRBVRB VRBCVRBC ProcessorProcessor SequencerSequencer SequencerSequencer SequencerSequencer SequencerSequencer Level 3 VME MIL-1553 Communication Calibration Software

35. 31 Aug 2005MICE DAQ and Controls Workshop35 EPICS at Fermilab  SMTF Superconducting module test facility Cavity test in October DOOCS for LLRF – speaks CA EPICS for everything else No ACNET – beams division controls system  ILC Control system to be decided Management structure now forming

36. 31 Aug 2005MICE DAQ and Controls Workshop36 EPICS Lessons  Three Layers Tools  EPICS tools  OPI development - CA library Applications  Build base  Build application - combine existing pieces  Develop device templates Development  Records, device, and driver support  EPICS is not simple to use Need expertise in each layer Support is an email away  IOC operating system selection

37. 31 Aug 2005MICE DAQ and Controls Workshop37 The End