The Online Detector Control at the BaBar experiment at SLAC Pierluigi Paolucci I.N.F.N. of Napoli Introduction; DCS architecture; EPICS; Component Proxy; Database; Conclusions. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar detector BaBar consists of 5 sub-detectors: Silicon Vertex Tracker; Drift Chamber; Detector of Internally Reflected Cherenkov ; ElectroMagnetic Calorimeter; Instrumented Flux Return. m detection and neutral hadron identification 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

BaBar DCS architecture Online  EPICS  Hardware  16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli DCS requirements Responsibilities of the detector control system: Safe and reliable operation of BaBar detector; Safe and reliable operation of PEP II accelerator; Monitor BaBar and PEP II operations; Generate Alarms and Log files. Report status of detector to the online system and PEP II; Configure the detector for calibration/physics data taking; Archive ambient data. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Architecture and standard hardware We have adopted some hardware standards to focus operational expertise, to limit device-dependent code development, and to ease maintenance. The standard DCS crate is a 6U VME chassis, it is equipped with a Motorola MVME177 single board running VxWorks as operating system. Analog data are digitized by VME boards or directly on the detector. In this latter case, CANbus is the standard bus solution. A Greensprings carrier board receive and transmit the CAN messages. A general monitor board has been developed to interface the CANbus to the electronic placed on the detector. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Slave Crate Fast-OR signals Detector HV system Analog signals CaenNET CAN-bus Electronic house 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Experimental Physics and Industrial Control System EPICS is used to control and monitoring the detector subsystems, environmental conditions, and accelerator parameters. EPICS is a freely distributed software package that includes a real-time kernel running under the VxWorks operating system, drivers for commonly used hardware components (HV, I/O, TDC, ADC, CANbus…), and a suite of user applications for control, monitoring, and data analysis. EPICS source code is written in C language and is available for customization by the user. EPICS architecture is distributed, with the Input/Output Controller (IOC) as the fundamental component. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli DCS Architecture The BaBar DCS is a distributed architecture based on a TCP/IP network of 15 IOCs containing about 105 hardware and software channels describing the status of each subsystem and central system (gas and cooling system, electronic house, interaction hall….) Each detector has one or more IOCs (VME + CPU) running is own EPCIS software code and is able to communicate with all the others IOCs. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli EPICS, Channel Access EPICS toll-kit contains a lot of tools connected via the Channel Access client/server libraries. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Channel Access performance 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli EPICS framework EDD/DM Database EPICS Control layer Alarm Handler Sequencer 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli EPICS software tools I The EPICS internal database is used to define and store the records (process block). 2500 records use around 1 Mbytes. For each records it is possible to define a lot of fields and the most useful are: scan period (max 60Hz), alarm limit and severity Hardware units, precision, conversion, link Software Alarm, state machine and monitor EPICS tools Muon system has 2600 hardware records and 1400 software records used to: Monitor HV, LV, Gas and Cooling systems, environmental parameters, noise and front-end boards. summarize the results and the status of the IFR and of each partition (sector). 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

EPICS software tools II The State Notation Language (similar to the C language) is used to build state machines. The I/O variables of the state machines have to be defined as EPICS records and they can be load from different IOCs. The state machines run in the VxWorks environment. For the visualization has been used the EDD/DM tool that provides an interactive display editor and high performance display manager; more different tools are available from JAVA to tcl/tk, from LabVIEW to Visual Basic. DM is able to update about 2000 channels per second. ALH is the alarm handler providing alarm viewing, logging and annunciation. It can handle up to 500 alarm condition changes per second. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar Alarm Handler Alarm limit and severity are defined at database level; The three structure is defined using a configuration file. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar Alarm Structure Each EPICS channel has 2 alarm limits: The first one is just a warning (color code is yellow); The second one is a real alarm (color code is red). The alarms are sent from the EPICS database to the EPICS Screen, Alarm Handler, Java Browser and Database. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

BaBar Data tacking philosophy Sub-detectors are divided in logical components, called partitions, based on the hardware structure of the system. Muon is divided in Barrel, Endcap forward and backward; DIRC and EMC are divided in 12 sectors; Each partition has its own EPICS records, State Machine and Alarm threes; The partition concept has been introduced in order to have the possibility to take data with sub-parts of the detectors. A Main State Machine has to coordinate the partitions and define-calculate-monitor a RUNNABLE , INJECTABLE or CALIBRATION flag as STATUS of that detector. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Interface with PEP-II Close communication between BaBar and PEP-II is essential to maintaining high operational efficiency and low background conditions. Before injection can begin BaBar must be in safe state and it remains safe until injection is finished. This injection sequence is coordinate automatically by two state machines, one running on the BaBar IOC verifies the detector status and communicate it to a second process running on the PEP-II IOC. Special hardware button is used to prevent/stop injection from the BaBar console. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar State Machine The RUNNABLE flag corresponds to a state in which the detector is ready to take data  HV at nominal voltage. The INJACTABLE flag means that detector is safe for the PEP II injection  Tracker, Drift and Muon at safe voltage. The CALIBRATION flag is used if one or more detectors are ready to take special run dedicated to the calibration. The BaBar State Machine get all the flags from the detector state machines and analyze them in real time generating a BaBar flag (RUNNABLE , INJECTABLE or CALIBRATION), monitored in the main panel and sent to the PEP II panel and state machine 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar main panel 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Muon system main panel

Wiener crate panels

Pigi Paolucci - I.N.F.N. of Napoli Muon HV Console 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

BaBar DCS architecture (overview) Detector EPICS Control layer PEP II Database Run Control Component Proxy 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

BaBar DCS architecture (details) 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Component Proxy A dedicated object-oriented interface, called Component Proxy, have been developed to bridge the gap between EPICS, written in C, and the BaBar Online system, implemented in C++. Component Proxy design: Abstraction of BaBar hardware in terms of C++ objects; Communicates with Run Control, configure detector and archives data; Generate the RUNNABLE flag of each subsystem. BaBar runs 27 CP on two Sun Ultra 10 workstations with 0.74GB of memory. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Interface with Run Control Each logical component of BaBar (tracker, DIRC...) is represented by one or more instances of the Component Proxy. Each proxy reports the status of its associated hardware components to the Run Control in form of a RUNNABLE flag, defined at EPICS level (state machine). Run Control is not allowed to begin a data run until all components are RUNNABLE. If a system loses its RUNNABLE status the Run Control pause the run. The response feedback mechanism is typically < 1 sec. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Ambient Database BaBar has adopted an object oriented approach for its offline and online software. Objectivity/DB is the underlying technology Provides the storage of ambient data from the DCS Design similar to Conditions Database Operational information: 27 archiving processes write about 100 objects/hour; Total size of objects about 3 MB; Storing time about 1min; Data written every 60 min to reduce DB activity; Total rate of about 2 MB/hr (17.5 GB/yr). 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Ambient Database Java Browser Java Browser Ambient Database Ambient Server (C++/CORBA) 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Configuration Database (I) Provides storage for the configuring parameters of the online system; Interface to Dataflow ROMs (DAQ) via UNIX server; Configuration objects indexed on numeric keys; Hierarchical tree-like structure; Tree provides navigation from top, indexed by configuration key, to the subsystem configuration data on the leaves of the tree; Simplifies system partitioning. Allows authorization control on subsystem level; 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Configuration Database (II) Before data-taking begins, Run Control initiates a CONFIGURE transition by sending a configuration key (defined by the run type) to the DAQ, trigger and DCS systems. The key is used by the systems to read data from the Configuration database. In the case of DCS the key is received by the Component Proxy then read data (typically HV set-points) and write it into EPICS. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Configuration Database (III) The system allows for easy switching between run types (physics, cosmic, calibration...) and create an archive of detector configurations in the database, used to determine the running conditions at any time. Processes accessing the database with each configure transition: 32 Event Processing nodes 17 Detector Control processes Dataflow server Configuration time is about 15 sec 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Conclusion (I) The BaBar Detector Control is working since the 1999. The EPICS core infrastructure has proven to be very robust, serving about 105 data channel over 15 IOCs in a very heavily used network The communication between BaBar and PEP-II has also been very reliable, and has contributed to a high overall operating efficiency. EPICS proved to be a very powerful, reliable and friendly system to control and monitor both BaBar and PEP II. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli BaBar Efficiency 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli Conclusion (II) Implementation of the Component Proxy and ODC interface into the Online system has not been easy. We had some problems with operational efficiency and scalability of the Objectivity database. The locking mechanism that regulates reading and writing create conflicts between different online processes. A big step has been produced from the DCS of the LEP experiments and the BaBar DCS system. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli

Pigi Paolucci - I.N.F.N. of Napoli What we can improve ! Design a DCS upper layer for control and monitor detectors, accelerator and the services (gas, safety...); A more friendly alarm handler with more and more information about the way to fix the problems; A way to correlate data with the Ambient database and Configuration database information; An easy way to debug the detector problem starting from one or more alarms/warning. Automatic way to fix the more frequent error/problem. 16 November 2018 Pigi Paolucci - I.N.F.N. of Napoli