DAQ software for SND detector Budker Institute of Nuclear Physics, Novosibirsk M. Achasov, A. Bogdanchikov, A. Kim, A. Korol

VEPP-2000 collider In the past: SND @ VEPP-2M Today: SND @ VEPP-2000 Years: 1995 - 2000 Energy (2E): 0.36 – 1.4 GeV Today: SND @ VEPP-2000 Project started: 2000 Energy: 1.4 - 2 GeV Luminosity: 1032 (2 GeV) Perimeter: 24.4 m Collision time: 82 nsec Beam current: 0.2 A Bunch length: 3.3 cm Energy spread: 0.7 MeV ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

VEPP-2000 collider ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

SND DAQ requirements Loose electronic selection conditions (FLT), strong software (TLT) filtration (ratio to 10%). Readout process should work on the stand-alone equipment, separately from processes which consume a lot of computational power (TLT). Event selection using fully reconstructed events – L3 farm. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Readout and events building Main data flow 100 Hz 1 KB 1 kHz 4 KB 1 kHz 4 KB 1 kHz 1 KB Readout and events building Events packing Events filtering Storage Expected rates: Events fragments: 4 МB/s are read from IO processors over Ethernet; Event building: 4 MB/s; Event packing: 1 МB/s; Events filtering (90% screening): 100 KB/sec. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Raw events transfer (alternatives) Readout L3 Sum Storage L3 L3 Readout L3 Storage ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Raw events transfer (software layers) Abstract layer Container capture Read events from container Write events to container System layer Global File System (GFS) – tested Shared raw disks Network File System (NFS) Hardware layer Shared SCSI disk - tested Network disk ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

DAQ architecture Detecto r Front-end electronics Off-line Filtered events Backup X 16 X 12 KLUKVA CAMAC TLT computers Visualization Readout & Event Building Buffer Database Calibration process System support ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Main online processes Events treatment : Control : Readout & event building Scalers (statistics from electronic subsystems) L3 (packing, reconstruction, filtration, statistics) Calibration on events Control : Operator interface Distributed processes management Context recovery after process crash Informer (message analyzer) Electronics sub-system control ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Readout process state machine Auto-start Idle Monitoring Readout Pause Configuration Initialization Start Reject Full stop Stop Success Note: error branches are not shown, all errors cause Idle state. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Process starter service Database TLT1 Process A: 2 Process B: 1 Process C: 3 TLT2 Process A: 1 Process D: 3 Online Computer TLT1 Online Computer TLT2 Starter service Starter service Process A Process D Process A child Process B Process C Process A Process D Process A Process C OK, one session ID needs to stop Process C Process D needs to start ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Processes states and recovery After start processes should receive domain name from “domain distributor server“ (DD). Each domain contains fixed name variable to store a path to recover program. When process crashes (tears connection), DD starts recover program with domain name as parameter. Current process state data are stored in the OnlState database and may be accessed by other processes. State variable type: int, float, string, arrays; access index: domain_name/variable_name Atomic domain access. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Process context recovery Domain states: Free Context recovery Busy Confirmation Domain request Process finished Process crashed Distributor server crashed and restarted Start recovery program Confirmation timeout Re-register domain ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

} Online Databases Online state DB Conditions DB Configurations DB Bookkeeping DB } Closely tiered. To simplify access integrated interface is supported. Implementation: RDBMS (MySQL) ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Configurations & Bookkeeping API Table description (SQL format) table.sql parser C++ description and implementation table.h table.cc Application ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Operator interface Run control Web-log Database access Process control Communication Personal settings Status Selective views Run conditions Occupied resources Users contacts User info Start Stop Pause Message search Configurations Processes start list Chat UI details Message add UI Calibrations UI colors Silence Extended messages E-mail notifications Statistics Scalers selection ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Operator interface components HTTP server Configurations PHP API Calibrations PHP API Online States PHP API PHP scripts Recovery PHP API Web-log PHP API HTML javascript RDBMS Configurations API Calibrations API Online States API Online Process Recovery API Web-log API ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Web-log messages The web-log toolkit consists of several interfaces written in languages PHP and C ++, which allow to choose, add and expect the message. Each message has attributes: Alarm – is on when required to draw attention of the operator immediately. Date and time of the message. Topic of the message (what field it belongs to). Severity of the message (fatal mistake, error, warning, info, debug, trace). Sender: user, host, process name. Subject – the summary of the message. This is a basic field. Message attachments of any MIME-type. Full description of the problem, pictures, diagrams can be presented in the attachment. Attachment has attributes: name, size and inline. The attribute inline is a recommendation for browsers, whether to include attachment in the message directly or display just a link. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Statistics view (example) ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Design High level robustness: raw events deep buffering; High quality of filtering: full events reconstruction before filtering; Reuse of off-line framework software; Relational databases to store experiment configurations, states, calibrations; Operator interface: web technologies . ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software

Status DAQ system software status: Event transfer mechanism were tested on the maximum possible rates; The most important functionality were realized; Advanced features were added; 2003-2004: system were used to receive events from the drift chamber prototype. ACAT05, Zeuthen January 11, 2019 Session 1, SND DAQ software