The Software Framework available at the ATLAS ROD Crate DISCLAIMER: Focus on common software Might be biased by Level-1 Central Trigger THANKS TO: M. Joos, G. Lehmann R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Outline Introduction The Architecture Model The Software Framework Ethernet – Single-Board Computer – VME The Software Framework A layered framework Outlook R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Trigger/DAQ System VME PC Level-1 Trigger Level-2 Trigger Event Filter Electronics + Firmware Readout Link (SLink) Level-2 Trigger + Event Filter: Computers + Networks + Software Level-1 Trigger VME RODs + Trigger Processors Level-2 Trigger PC Except ROIBuilder (VME) and ROBIN (SLink-PCI ) Event Filter R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

The Architecture Model VME User (ssh) Ethernet SBC config ROD ROD_BUSY TRIG PROC Sub-detector specific Diagnostics status & monitoring ATLAS Run Control (TCP/IP) main data flow Readout Driver Crate: Can also be a trigger processor or other front-end crate; can also contain modules other than RODs, e.g. LTP, TTCvi, ROD_BUSY, etc. Provides interactive use, detector-specific diagnostic tools and ATLAS run control through a single-board computer (SBC) VME is used for configuration, status, and monitoring; the event or trigger data use other links (e.g. SLink or dedicated backplanes). Transfer protocol (parallel memory bus): single register READ/WRITE memory/FIFO block transfer interrupts R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

The Software: A Layered Framework ROD Crate DAQ Low-level Software* Drivers and Libraries Operating System Hardware * Mainly sub-detector specific R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: Hardware Provides common hardware infrastructure VME: Choice of crates, power supplies, and fans Common guidelines for the use of VME64x Common purchase & repair service (PH/ESE) Common “slow control”: ATLAS DCS over CANbus Single-board computers: A family of three generations of single-board computers: Intel-based central processing unit and PCI/X-VME bridge Additional Modules: ROD_Busy Module TTC Modules: TTCvi, TTCex, etc. LTP, LTPI Other modules available from the Electronics Pool which are used in a ROD Crate framework, e.g. at test beam and in lab tests, and for which common low-level software exists R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: Operating System Provides common operating system and environment Common Service: Provided by CERN/IT & ATLAS TDAQ System Administrators Operating System  Scientific Linux CERN Boot configuration (in particular loading DAQ drivers) Security (at Point 1) System/network monitoring (at Point 1) R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: Drivers&Libraries - 1 Provides common drivers and software infrastructure Common VME Driver (pure CERN product): Dynamically loaded driver with static mappings (vmetab file) Single READ and WRITE FAST: uses mapped I/O (ignore VME bus errors, requires SBC with h/w for byte swapping) SAFE: uses programmed I/O under driver control (catch VME bus errors) Slave mappings (SBC memory as slave on VME) To my knowledge not used Block transfer Allows several pipelined chains of block transfers. Interrupt handling Wait synchronously on semaphore or register POSIX signal with driver To my knowledge not often used Other features: CR/CSR access, SYSRST generation, SYSFAIL and bus error signalling, and debugging features Provides C library and C++ wrapper R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: Drivers&Libraries - 2 Provides common drivers and software infrastructure Other drivers and libraries: PCI: Read/write memory I/O Link/unlink PCI devices Could be reused in an xTCA environment. Contiguous memory Kernel pages (or previously BIG PHYS area) Other libraries: “Helpers”: Errors codes, time stamps, bit strings, JTAG chains, simple menu programs, XML module templates, etc. And their tools for testing and debugging R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: Low-level Software Provides sub-detector specific software Sub-detector specific diagnostic tools for early development or debugging Sub-detector specific calibration tasks detailed knowledge of the calibration and front-end system Libraries (C++) which “encapsulate” the specificities so that the ROD (or other modules) can be used from the next higher level This is probably the greater part of the software base. Common low-level software Common libraries for common modules, e.g. LTP, TTCvi, ROD_BUSY, etc. R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Framework: ROD Crate DAQ Provides Integration into ATLAS DAQ: Based on a skeleton program* (C++) which needs to be filled in with sub-detector specific calls + a set of common modules ready to be used. * A ReadoutApplication originally developed for the ROS DAQ Run Control: Distributed system of finite-state machines (controllers) Configuration Databases: Run parameters, calibration data, trigger configuration, etc. Monitoring: Status/error messages, status values, histograms, event monitoring, data quality, persistent information, etc. Ready-to-use Modules: Controller, configuration, and monitoring are readily available for common modules (e.g. ROD_BUSY) and other DAQ tasks Is based on the same software as the rest of the ATLAS TDAQ (ROS, L2, and EF). Have to see how that one evolves! R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

History From my recollection: Detector Interface Group (DIG): Define interface between sub-detectors and DAQ Define DAQ functionality required by sub-detectors TDAQ Interfaces with Front-end Systems Requirements Document + Readout Link (SLink) and Raw Event Format ROD Working Group (sub-group of DIG): Exchange of information on ROD hardware Define common solutions, e.g. VME hardware/driver & common modules ROD Crate DAQ Task Force (sub-group of DIG): Define common DAQ for the ROD crate → RCD Common solutions have proven to be useful in terms of effort for development and maintenance (in particular for smaller groups). R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

Outlook What will replace the existing services/infrastructure? If we intend to replace VME: What will replace the existing services/infrastructure? Common solutions in terms of common hardware/software? Common service for purchase & repair? Common DCS? What architecture/protocol to use? Architecture: Based on SBC which distributes commands to modules? Common driver and library? Protocol: Single or grouped READ/WRITE? Block transfers? Interrupts? Performance: What latency for control and throughput for monitoring are required? Common ROD Crate DAQ (same for all sub-detectors? same as ROS DAQ)? Common modules? How to arrive at common solution? Do we need another ROD Working Group? What is the manpower required to develop and maintain common framework? And who can provide that manpower? Must not forget the VME legacy. It will be around until phase 2 and probably beyond. R. Spiwoks Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012