1. Fermilab Control System (“ACNET”) J. Patrick February 17, 2005

2. SMTF Review 2/17/05 J. Patrick2 Fermilab Control System  Overview  Applications  Central Services  Front-ends/Instrumentation  Timing  Security and Remote Access  Summary

3. SMTF Review 2/17/05 J. Patrick3 Fermilab Control System  aka “ACNET”  Unified control system for the entire complex  400 MeV Linac  8 GeV Booster Synchrotron  120 GeV Main Injector Synchrotron  1 TeV Tevatron Synchrotron  antiproton source – target/debuncher/accumulator  antiproton “Recycler” storage ring  fixed target lines  Simultaneous operation of  Tevatron proton-antiproton collider (storage ring)  Antiproton production and storage (~0.5 Hz)  120 GeV fixed target to Meson lab (~0.1 Hz)  120 GeV fixed target to NUMI/MINOS (~0.5 Hz)  8 GeV fixed target to MiniBoone (~5 Hz)

4. SMTF Review 2/17/05 J. Patrick4 Fermilab Accelerator Complex

5. SMTF Review 2/17/05 J. Patrick5 Control System Overview Console Applications Central Services IRM Front-Ends Java Applications Web Applications MOOC Front-Ends Labview Front-Ends Open Access Clients Field Hardware ethernet Database CAMAC, VME, PMC, IP, Multibus, CIA, GPIB, … field bus: VME, SLD, Arcnet, ethernet, … ServletsConsolidators Application Central Front-End

6. SMTF Review 2/17/05 J. Patrick6 History  Originally developed for Tevatron in early 80’s  Substantial evolution over the years  CAMAC field hardware -> very broad diversity  PDP-11/Mac-16 front-ends -> x86 Multibus -> VME based processors + Labview  DEC PCL network -> IEEE 802.5 token ring -> ethernet  VAX database -> Sybase  VAX central services -> Java  PDP-11 applications -> VAXstations -> Java application infrastructure -> Port of VAX code to Linux in progress

7. SMTF Review 2/17/05 J. Patrick7 System Scale  200,000 devices/350,000 properties  475 front-ends  90 Sun Netras + 12 Linux PCs that run Java based data loggers, servlets, open access clients  + VAXes to run VAX central services  70 VAXes that run high console level applications  + 70 Linux PCs to to parallel VAXes until VAX retirement.  X-displayed out to desktop PCs

8. SMTF Review 2/17/05 J. Patrick8 Device Model  <= 8 character devices names  e.g. T:BEAM  Informal conventions provide some rationality but no enforcement  More verbose hierarchy layer created, but not utilized/maintained  Each device has one or more properties:  Reading, setting, analog & digital alarms, digital status & control  Device descriptions stored in Sybase database  Scaling transformations defined in device database  Add new devices with “DABBEL” command language on VAX  Templates available for various device types  View/modify device definitions with GUI  Monotype data, arrays are transparently supported  Structured data is often done, though support is not transparent  Application (library) layer must properly interpret the data from the front-end and perform required data translations

9. SMTF Review 2/17/05 J. Patrick9 Device Database GUI

10. SMTF Review 2/17/05 J. Patrick10 Communication  Applications/Central Services/Front-ends communicate via “ACNET” protocol  Custom protocol now exclusively based on UDP  Extensive network/switches etc…  Asynchronous request/reply; request multiple reply  Timeout/retry mitigates UDP unreliability.  “RETDAT/SETDAT” protocol for readings/settings  Can have multiple devices/packet  Fast Time Plot (FTP) returns blocks of readings taken at up to 1440 Hz every 0.5 seconds  Snapshot protocol returns blocks of 2048 points taken at an arbitrarily high rate

11. SMTF Review 2/17/05 J. Patrick11 Communication  Application communication with front-ends passes through central layer  Datapool Manager on VMS via shared memory  Data Acquisition/Server Engine (DAE) in Java (via RMI)  Consolidation of requests to avoid overloading front-ends  Centralizes security, monitoring, error logging  Issues include:  Limited packet size and internal device address space.  Timestamps not part of protocol packet, applied on data receipt  GPS sync clock event sent to all front-ends, can be included in structured devices  New get/set32 protocol addresses the above, only on centrals now  Lack of transparent support for structured data.  Bridging to network devices that don’t directly support ACNET is via front-ends or Open Access Clients

12. SMTF Review 2/17/05 J. Patrick12 Database  Central Sybase database  Three parts:  Device and node database  Application database  Used by applications as they wish  Save/Restore/Shot Data database  MySQL used for data loggers, and for some other specialized purposes. Not for general application use

13. SMTF Review 2/17/05 J. Patrick13 Application Environment  Two major high level application environments  Console environment  VAX based  C language (+ legacy FORTRAN)  Port to Linux in progress  Expect some operational utilization in the fall  Java environment  Platform independent  Windows/Sun/Linux/Mac all supported and utilized  Full access to entire control system  Both environments have a standard application framework that provides a common look and feel  All applications launched from “Index Pages”  Currently separate pages for Console, Java environments  Also some browser applications of varying styles

14. SMTF Review 2/17/05 J. Patrick14 Index Page (Console)

15. SMTF Review 2/17/05 J. Patrick15 Application Index (Java) Web Browser Standalone Application

16. SMTF Review 2/17/05 J. Patrick16 Application Overview  Utility applications  Parameter Page  Fast Time Plot  Data Logger viewer  Database Viewer  Synoptic displays  Diagnostics  Save/Restore  Alarms  …  Machine Specific Applications  ~600 Primary + 125 Secondary applications in console environment  >100 applications in Java environment  Mostly utility, shot data analysis

17. SMTF Review 2/17/05 J. Patrick17 Parameter Page (Console)

18. SMTF Review 2/17/05 J. Patrick18 Parameter Page (Java)

19. SMTF Review 2/17/05 J. Patrick19 Fast Time Plot (Java)

20. SMTF Review 2/17/05 J. Patrick20 Synoptic Display (Console)  Graphical Builder “Lex Draw”

21. SMTF Review 2/17/05 J. Patrick21 Synoptic Builder (Java)

22. SMTF Review 2/17/05 J. Patrick22 Synoptic Display (Web)

23. SMTF Review 2/17/05 J. Patrick23 Sequencer (Console)  Automatically Sequence Collider Shots, other activities  Relies on ACL scripting language

24. SMTF Review 2/17/05 J. Patrick24 Finite State Machine (Java)  Graphical Builder  Primary current usage is Pelletron control  Keeps track of HV sparks and beam related trips

25. SMTF Review 2/17/05 J. Patrick25 Sequenced Data Acquisition  Automated acquisition of data for collider shots  Configure readings for various stages of collider shot  Snapshots/FTP as well as single scalar readings  SDAEdit tool makes it easy to add new things  Data acquired automatically and stored in Sybase database  Summary html reports generated when store goes in  Step efficiencies, error reports, luminosity  “Supertable” captures ~200 most popular items  Html, Excel, Java Analysis Studio formats automatically generated  Suite of tools to view detailed data  Data can be directly read by Excel, Java Analysis Studio, potentially other external tools  Simple Java API for user written code

26. SMTF Review 2/17/05 J. Patrick26 Alarms Display

27. SMTF Review 2/17/05 J. Patrick27 Scripting  Accelerator Control Language (ACL)  Console environment  Access to much of the functionality in the control system  VMS DCL style syntax  Extensively used in Sequencer  Jython  Java API in principle accessible to Jython  Some but limited testing/use to date

28. SMTF Review 2/17/05 J. Patrick28 External Tools  Excel  Excel based parameter page  Easy cut/paste of Java tables into Excel  Excel data can be imported into a variety of tools  MathCAD, OptimA used locally  MATLAB  MATLAB supposedly has a standard Java interface, in theory it should be straightforward to integrate with the control system  However we haven’t tried it  An attempt to do this with Mathematica > 1 year ago was not successful, it does not use a standard Java distribution  Java Analysis Studio  Currently data logger and shot data plug-ins

29. SMTF Review 2/17/05 J. Patrick29 Excel Parameter Page

30. SMTF Review 2/17/05 J. Patrick30 Java Analysis Studio

31. SMTF Review 2/17/05 J. Patrick31 Save/Restore  Devices may be assigned to save/restore lists  Console applications to save/restore to/from database on user request  Scheduled saves of the entire control system 4 times/day

32. SMTF Review 2/17/05 J. Patrick32 Electronic Log  Based on Oak Ridge product  Web browser client  Perl/cgi back-end  Shift log + machine / subsystem logs  Used by A0  API to add entries programmatically, including images  Both Console and Java environments  Java application framework includes standard menu item to make posts.

33. SMTF Review 2/17/05 J. Patrick33 Central Layer  Open Access Clients (“virtual front-ends”)  Java Servlets  Bridge requests from outside firewall  Request Consolidation  Alarms server  Front-end download  Other things …

34. SMTF Review 2/17/05 J. Patrick34 Open Access Clients  Processes with no user interface, always running  Obey same communication protocol as front-ends  Compare to EPICS “Soft IOC”  Several classes:  Utility – Data loggers, scheduled data acquisition, virtual devices  Calculations – Both database driven and custom  Process control – Autotune of fixed target lines  Bridges –to ethernet connected scopes, instrumentation, control systems…  Easy access to devices on multiple front-ends  Friendlier programming environment than VxWorks front-ends  Framework transparently handles ACNET communication  Access to database, other high level operating system features  Do not provide hard real-time response  Clock events via ethernet multicast  > 100; Almost all run in Java framework; VAX framework deprecated

35. SMTF Review 2/17/05 J. Patrick35 Data Loggers  ~70 instances running in parallel on central nodes  ~80 GB compressed storage per node in MySQL database  Circular buffer; wraparound time depends on device count/rates  Data at maximum 1 Hz rate extracted to “Backup” logger every 24 hours  Backup logger is available online => data accessible forever  Generally allocated to various departments to configure as they wish  No central management of archiving lists, so inevitable duplication  Tasks follow the Open Access Client architecture  Simple GUI to specify logging lists  Single readings or snapshots; on clock events or at fixed rates to 15 Hz  Programmatic logging interface available for custom loggers  Variety of retrieval tools available  Standard console applications, Console and Java environments  Web applications  Java Analysis Studio  programmatic APIs

36. SMTF Review 2/17/05 J. Patrick36 Front-Ends  Interface field hardware to the high level control system  Several architectures  MOOC – Minimally Object-Oriented Communication  Very wide diversity of functionality  Processors all VME (VXI) based  Very wide diversity of field hardware supported  Links to many other form factors such as CAMAC, etc.  MVME 162 (68k) and 2400 (Power PC)  VxWorks Operating System  ~275 in the system

37. SMTF Review 2/17/05 J. Patrick37 Front-Ends - IRM  IRM – Internet Rack Monitor  MVME 162; PSOS operating system  Chassis includes 64 channel x 1 KHz 16 bit continuous digitizer  8 x 10-20 MHz option available  8 bytes external digital I/O  8 x 12 bit DAC outputs  Expandable via IndustryPak I/O  ~125 in the system  Eight used at A0 photo-injector Variety of functions  A few at TTF for modulators

38. SMTF Review 2/17/05 J. Patrick38 Front-Ends  Linac front-ends  VME based Power PC processors  Non-MOOC infrastructure; tied to 15 Hz Linac cycle rate  ~50 in the system  HRM – Hotlink Rack Monitor  Evolution of IRM architecture  I/O chassis separate from processor chassis  64 x 10 KHz ADC; up to 2 per chassis  10-20 MHz snapshot board  Timestamps included with ADC data  8 byte digital I/O per chassis  Multiple chassis’ per PPC processor  VxWorks (not MOOC)  Initial build of 15 units in progress

39. SMTF Review 2/17/05 J. Patrick39 Labview  Front-Ends  General control/acquistion from high level control system  Interact via ACNET protocol written in Labview VI language  No FTP/Snap  Non-local console access via PCAnywhere, Timbuktu etc.  ~25 in system  Instrumentation:  Flying Wires  Sample Bunch Display  …  Both PCs & Macs

40. SMTF Review 2/17/05 J. Patrick40 Field Hardware  Wide variety of hardware supported  CAMAC, VME, VXI, Multibus, CIA, …  Too extensive to describe here…  Support for wide variety of commercial instrumentation  Scopes, spectrum analyzers, signal generators, DVMs, …  Ethernet->GPIB; direct ethernet  Through front-ends; through OACs; direct in some cases  Mostly in console environment, ports to Java as needed  Siemens-Moore APACS PLC supervisor  Used for cryogenic control  Interface through Open Access Client using library from company  Other PLCs also supported through front-ends

41. SMTF Review 2/17/05 J. Patrick41 Bridge to External Systems  Accelerator NETwork control system (AccelNET) is a software package designed for control of electrostatic particle accelerator systems.  Runs on PC hardware under the Linux operating system  Modular Linux programs that communicate to each other through a central database server using TCP/IP.  CAMAC- GROUP3 I/O  ACNET  AccelNET Bridge

42. SMTF Review 2/17/05 J. Patrick42 Timing and Control Networks  TCLK (Clock Events)  10 MHz Manchester encoded signal with 8 + 2 bits/event  256 possible events defined for the complex  Decoding hardware/signal generators for variety of buses  Standalone version of this system used at A0 photo-injector  MDAT (Machine Data)  10 Mbit/sec serial link; 8 bit type + 16 bits data per frame  Originally developed to rapidly distribute ramp currents around the main ring and Tevatron; usage has broadened  Beam Sync – revolution markers derived from RF  States  State devices defined  Any node may issue transition request, gets multicast everywhere  Can be distributed via MDAT  Beam permit/abort  CAMAC interfaces….

43. SMTF Review 2/17/05 J. Patrick43 Security  Control system network inside a firewall  Restricted access from/to outside the accelerator division network  Privilege classes assigned based on account, service, (node)  Control from where applications may be run  Control which programs/users/nodes may do settings  Emphasis on accountability as well as security  Setting log  Application log

44. SMTF Review 2/17/05 J. Patrick44 Remote Access  Console environment via remote X display  Administrative controls on usage, particularly settings etc.  Java applications can be launched from anywhere from a browser via Web Start  Access to inner parts of control system only via VPN  Applications to be run remotely w/o VPN must be designed for that  Applications that work via servlets;  “Secure Controls Framework” – Subset of data access API implemented in this way, better performance than servlets/xml  Access from outside the standard application environments  xml-rpc is preferred method  Readings of any device; setting of “virtual” devices  No setting of hardware associated devices  Used by all current experiments

45. SMTF Review 2/17/05 J. Patrick45 ACNET Summary  Should meet requirements for SMTF  Has been/is being continually modernized  Extensive experience at FNAL.  Less time to get a system running  A broad range of people could contribute quickly  Limited experience outside of Fermilab  Remotely accessible but full system not portable  Documentation…