1. Fermilab Control System (“ACNET”)
J. Patrick
February 17, 2005

2. Fermilab Control System
Overview
Applications
Central Services
Front-ends/Instrumentation
Timing
Security and Remote Access
Summary
SMTF Review 2/17/05
J. Patrick

3. Fermilab Control System
aka “ACNET”
Unified control system for the entire complex
400 MeV Linac
8 GeV Booster Synchrotron
120 GeV Main Injector Synchrotron
1 TeV Tevatron Synchrotron
antiproton source – target/debuncher/accumulator
antiproton “Recycler” storage ring
fixed target lines
Simultaneous operation of
Tevatron proton-antiproton collider (storage ring)
Antiproton production and storage (~0.5 Hz)
120 GeV fixed target to Meson lab (~0.1 Hz)
120 GeV fixed target to NUMI/MINOS (~0.5 Hz)
8 GeV fixed target to MiniBoone (~5 Hz)
SMTF Review 2/17/05
J. Patrick

4. Fermilab Accelerator Complex
SMTF Review 2/17/05
J. Patrick

5. Control System Overview
Console
Applications
Java
Applications
Web
Applications
Database
Application
Central Services
Servlets
Open Access
Clients
Consolidators
Central
ethernet
MOOC
Front-Ends
Labview
Front-Ends
IRM
Front-Ends
Front-End
field bus: VME, SLD, Arcnet, ethernet, …
Field Hardware
CAMAC, VME, PMC, IP, Multibus, CIA, GPIB, …
SMTF Review 2/17/05
J. Patrick

6. History Originally developed for Tevatron in early 80’s
Substantial evolution over the years
CAMAC field hardware -> very broad diversity
PDP-11/Mac-16 front-ends -> x86 Multibus -> VME based processors + Labview
DEC PCL network -> IEEE token ring -> ethernet
VAX database -> Sybase
VAX central services -> Java
PDP-11 applications -> VAXstations -> Java application infrastructure -> Port of VAX code to Linux in progress
SMTF Review 2/17/05
J. Patrick

7. System Scale 200,000 devices/350,000 properties 475 front-ends
90 Sun Netras + 12 Linux PCs that run Java based data loggers, servlets, open access clients
+ VAXes to run VAX central services
70 VAXes that run high console level applications
+ 70 Linux PCs to to parallel VAXes until VAX retirement.
X-displayed out to desktop PCs
SMTF Review 2/17/05
J. Patrick

8. Device Model <= 8 character devices names
e.g. T:BEAM
Informal conventions provide some rationality but no enforcement
More verbose hierarchy layer created, but not utilized/maintained
Each device has one or more properties:
Reading, setting, analog & digital alarms, digital status & control
Device descriptions stored in Sybase database
Scaling transformations defined in device database
Add new devices with “DABBEL” command language on VAX
Templates available for various device types
View/modify device definitions with GUI
Monotype data, arrays are transparently supported
Structured data is often done, though support is not transparent
Application (library) layer must properly interpret the data from the front-end and perform required data translations
SMTF Review 2/17/05
J. Patrick

9. Device Database GUI
SMTF Review 2/17/05
J. Patrick

10. Communication
Applications/Central Services/Front-ends communicate via “ACNET” protocol
Custom protocol now exclusively based on UDP
Extensive network/switches etc…
Asynchronous request/reply; request multiple reply
Timeout/retry mitigates UDP unreliability.
“RETDAT/SETDAT” protocol for readings/settings
Can have multiple devices/packet
Fast Time Plot (FTP) returns blocks of readings taken at up to 1440 Hz every 0.5 seconds
Snapshot protocol returns blocks of 2048 points taken at an arbitrarily high rate
SMTF Review 2/17/05
J. Patrick

11. Communication
Application communication with front-ends passes through central layer
Datapool Manager on VMS via shared memory
Data Acquisition/Server Engine (DAE) in Java (via RMI)
Consolidation of requests to avoid overloading front-ends
Centralizes security, monitoring, error logging
Issues include:
Limited packet size and internal device address space.
Timestamps not part of protocol packet, applied on data receipt
GPS sync clock event sent to all front-ends, can be included in structured devices
New get/set32 protocol addresses the above, only on centrals now
Lack of transparent support for structured data.
Bridging to network devices that don’t directly support ACNET is via front-ends or Open Access Clients
SMTF Review 2/17/05
J. Patrick

12. Database Central Sybase database Three parts:
Device and node database
Application database
Used by applications as they wish
Save/Restore/Shot Data database
MySQL used for data loggers, and for some other specialized purposes. Not for general application use
SMTF Review 2/17/05
J. Patrick

13. Application Environment
Two major high level application environments
Console environment
VAX based
C language (+ legacy FORTRAN)
Port to Linux in progress
Expect some operational utilization in the fall
Java environment
Platform independent
Windows/Sun/Linux/Mac all supported and utilized
Full access to entire control system
Both environments have a standard application framework that provides a common look and feel
All applications launched from “Index Pages”
Currently separate pages for Console, Java environments
Also some browser applications of varying styles
SMTF Review 2/17/05
J. Patrick

14. Index Page (Console)
SMTF Review 2/17/05
J. Patrick

15. Application Index (Java)
Web Browser
Standalone Application
SMTF Review 2/17/05
J. Patrick

16. Application Overview Utility applications
Parameter Page
Fast Time Plot
Data Logger viewer
Database Viewer
Synoptic displays
Diagnostics
Save/Restore
Alarms …
Machine Specific Applications
~600 Primary Secondary applications in console environment
>100 applications in Java environment
Mostly utility, shot data analysis
SMTF Review 2/17/05
J. Patrick

17. Parameter Page (Console)
SMTF Review 2/17/05
J. Patrick

18. Parameter Page (Java)
SMTF Review 2/17/05
J. Patrick

19. Fast Time Plot (Java)
SMTF Review 2/17/05
J. Patrick

20. Synoptic Display (Console)
Graphical Builder “Lex Draw”
SMTF Review 2/17/05
J. Patrick

21. Synoptic Builder (Java)
SMTF Review 2/17/05
J. Patrick

22. Synoptic Display (Web)
SMTF Review 2/17/05
J. Patrick

23. Sequencer (Console)
Automatically Sequence Collider Shots, other activities
Relies on ACL scripting language
SMTF Review 2/17/05
J. Patrick

24. Finite State Machine (Java)
Graphical Builder
Primary current usage is Pelletron control
Keeps track of HV sparks and beam related trips
SMTF Review 2/17/05
J. Patrick

25. Sequenced Data Acquisition
Automated acquisition of data for collider shots
Configure readings for various stages of collider shot
Snapshots/FTP as well as single scalar readings
SDAEdit tool makes it easy to add new things
Data acquired automatically and stored in Sybase database
Summary html reports generated when store goes in
Step efficiencies, error reports, luminosity
“Supertable” captures ~200 most popular items
Html, Excel, Java Analysis Studio formats automatically generated
Suite of tools to view detailed data
Data can be directly read by Excel, Java Analysis Studio, potentially other external tools
Simple Java API for user written code
SMTF Review 2/17/05
J. Patrick

26. Alarms Display
SMTF Review 2/17/05
J. Patrick

27. Scripting Accelerator Control Language (ACL) Jython
Console environment
Access to much of the functionality in the control system
VMS DCL style syntax
Extensively used in Sequencer
Jython
Java API in principle accessible to Jython
Some but limited testing/use to date
SMTF Review 2/17/05
J. Patrick

28. External Tools Excel MATLAB Java Analysis Studio
Excel based parameter page
Easy cut/paste of Java tables into Excel
Excel data can be imported into a variety of tools
MathCAD, OptimA used locally
MATLAB
MATLAB supposedly has a standard Java interface, in theory it should be straightforward to integrate with the control system
However we haven’t tried it
An attempt to do this with Mathematica > 1 year ago was not successful, it does not use a standard Java distribution
Java Analysis Studio
Currently data logger and shot data plug-ins
SMTF Review 2/17/05
J. Patrick

29. Excel Parameter Page
SMTF Review 2/17/05
J. Patrick

30. Java Analysis Studio
SMTF Review 2/17/05
J. Patrick

31. Save/Restore Devices may be assigned to save/restore lists
Console applications to save/restore to/from database on user request
Scheduled saves of the entire control system 4 times/day
SMTF Review 2/17/05
J. Patrick

32. Electronic Log Based on Oak Ridge product Web browser client
Perl/cgi back-end
Shift log + machine / subsystem logs
Used by A0
API to add entries programmatically, including images
Both Console and Java environments
Java application framework includes standard menu item to make posts.
SMTF Review 2/17/05
J. Patrick

33. Central Layer Open Access Clients (“virtual front-ends”) Java Servlets
Bridge requests from outside firewall
Request Consolidation
Alarms server
Front-end download
Other things …
SMTF Review 2/17/05
J. Patrick

34. Open Access Clients Processes with no user interface, always running
Obey same communication protocol as front-ends
Compare to EPICS “Soft IOC”
Several classes:
Utility – Data loggers, scheduled data acquisition, virtual devices
Calculations – Both database driven and custom
Process control – Autotune of fixed target lines
Bridges –to ethernet connected scopes, instrumentation, control systems…
Easy access to devices on multiple front-ends
Friendlier programming environment than VxWorks front-ends
Framework transparently handles ACNET communication
Access to database, other high level operating system features
Do not provide hard real-time response
Clock events via ethernet multicast
> 100; Almost all run in Java framework; VAX framework deprecated
SMTF Review 2/17/05
J. Patrick

35. Data Loggers ~70 instances running in parallel on central nodes
~80 GB compressed storage per node in MySQL database
Circular buffer; wraparound time depends on device count/rates
Data at maximum 1 Hz rate extracted to “Backup” logger every 24 hours
Backup logger is available online => data accessible forever
Generally allocated to various departments to configure as they wish
No central management of archiving lists, so inevitable duplication
Tasks follow the Open Access Client architecture
Simple GUI to specify logging lists
Single readings or snapshots; on clock events or at fixed rates to 15 Hz
Programmatic logging interface available for custom loggers
Variety of retrieval tools available
Standard console applications, Console and Java environments
Web applications
Java Analysis Studio
programmatic APIs
SMTF Review 2/17/05
J. Patrick

36. Front-Ends Interface field hardware to the high level control system
Several architectures
MOOC – Minimally Object-Oriented Communication
Very wide diversity of functionality
Processors all VME (VXI) based
Very wide diversity of field hardware supported
Links to many other form factors such as CAMAC, etc.
MVME 162 (68k) and 2400 (Power PC)
VxWorks Operating System
~275 in the system
SMTF Review 2/17/05
J. Patrick

37. Front-Ends - IRM IRM – Internet Rack Monitor
MVME 162; PSOS operating system
Chassis includes 64 channel x 1 KHz 16 bit continuous digitizer
8 x MHz option available
8 bytes external digital I/O
8 x 12 bit DAC outputs
Expandable via IndustryPak I/O
~125 in the system
Eight used at A0 photo-injector Variety of functions
A few at TTF for modulators
SMTF Review 2/17/05
J. Patrick

38. Front-Ends Linac front-ends HRM – Hotlink Rack Monitor
VME based Power PC processors
Non-MOOC infrastructure; tied to 15 Hz Linac cycle rate
~50 in the system
HRM – Hotlink Rack Monitor
Evolution of IRM architecture
I/O chassis separate from processor chassis
64 x 10 KHz ADC; up to 2 per chassis
10-20 MHz snapshot board
Timestamps included with ADC data
8 byte digital I/O per chassis
Multiple chassis’ per PPC processor
VxWorks (not MOOC)
Initial build of 15 units in progress
SMTF Review 2/17/05
J. Patrick

39. Labview
Front-Ends
General control/acquistion from high level control system
Interact via ACNET protocol written in Labview VI language
No FTP/Snap
Non-local console access via PCAnywhere, Timbuktu etc.
~25 in system
Instrumentation:
Flying Wires
Sample Bunch Display …
Both PCs & Macs
SMTF Review 2/17/05
J. Patrick

40. Field Hardware Wide variety of hardware supported
CAMAC, VME, VXI, Multibus, CIA, …
Too extensive to describe here…
Support for wide variety of commercial instrumentation
Scopes, spectrum analyzers, signal generators, DVMs, …
Ethernet->GPIB; direct ethernet
Through front-ends; through OACs; direct in some cases
Mostly in console environment, ports to Java as needed
Siemens-Moore APACS PLC supervisor
Used for cryogenic control
Interface through Open Access Client using library from company
Other PLCs also supported through front-ends
SMTF Review 2/17/05
J. Patrick

41. Bridge to External Systems
Accelerator NETwork control system (AccelNET) is a software package designed for control of electrostatic particle accelerator systems.
Runs on PC hardware under the Linux operating system
Modular Linux programs that communicate to each other through a central database server using TCP/IP.
CAMAC- GROUP3 I/O
ACNETAccelNET Bridge
SMTF Review 2/17/05
J. Patrick

42. Timing and Control Networks
TCLK (Clock Events)
10 MHz Manchester encoded signal with bits/event
256 possible events defined for the complex
Decoding hardware/signal generators for variety of buses
Standalone version of this system used at A0 photo-injector
MDAT (Machine Data)
10 Mbit/sec serial link; 8 bit type + 16 bits data per frame
Originally developed to rapidly distribute ramp currents around the main ring and Tevatron; usage has broadened
Beam Sync – revolution markers derived from RF
States
State devices defined
Any node may issue transition request, gets multicast everywhere
Can be distributed via MDAT
Beam permit/abort
CAMAC interfaces….
SMTF Review 2/17/05
J. Patrick

43. Security Control system network inside a firewall
Restricted access from/to outside the accelerator division network
Privilege classes assigned based on account, service, (node)
Control from where applications may be run
Control which programs/users/nodes may do settings
Emphasis on accountability as well as security
Setting log
Application log
SMTF Review 2/17/05
J. Patrick

44. Remote Access Console environment via remote X display
Administrative controls on usage, particularly settings etc.
Java applications can be launched from anywhere from a browser via Web Start
Access to inner parts of control system only via VPN
Applications to be run remotely w/o VPN must be designed for that
Applications that work via servlets;
“Secure Controls Framework” – Subset of data access API implemented in this way, better performance than servlets/xml
Access from outside the standard application environments
xml-rpc is preferred method
Readings of any device; setting of “virtual” devices
No setting of hardware associated devices
Used by all current experiments
SMTF Review 2/17/05
J. Patrick

45. ACNET Summary Should meet requirements for SMTF
Has been/is being continually modernized
Extensive experience at FNAL.
Less time to get a system running
A broad range of people could contribute quickly
Limited experience outside of Fermilab
Remotely accessible but full system not portable
Documentation…
SMTF Review 2/17/05
J. Patrick