1. The Software Framework available at the ATLAS ROD Crate
DISCLAIMER:
Focus on common software
Might be biased by Level-1 Central Trigger
THANKS TO:
M. Joos, G. Lehmann
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

2. Outline Introduction The Architecture Model The Software Framework
Ethernet – Single-Board Computer – VME
The Software Framework
A layered framework
Outlook
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

3. Trigger/DAQ System VME PC Level-1 Trigger Level-2 Trigger Event Filter
Electronics +
Firmware
Readout Link
(SLink)
Level-2 Trigger
+ Event Filter:
Computers +
Networks +
Software
Level-1
Trigger
VME
RODs + Trigger Processors
Level-2
Trigger PC
Except ROIBuilder (VME) and ROBIN (SLink-PCI )
Event Filter
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

4. The Architecture Model
VME
User (ssh)
Ethernet
SBC
config
ROD
ROD_BUSY
TRIG PROC
Sub-detector specific Diagnostics
status & monitoring
ATLAS Run Control (TCP/IP)
main data flow
Readout Driver Crate:
Can also be a trigger processor or other front-end crate; can also contain modules other than RODs, e.g. LTP, TTCvi, ROD_BUSY, etc.
Provides interactive use, detector-specific diagnostic tools and ATLAS run control through a single-board computer (SBC)
VME is used for configuration, status, and monitoring; the event or trigger data use other links (e.g. SLink or dedicated backplanes).
Transfer protocol (parallel memory bus):
single register READ/WRITE
memory/FIFO block transfer
interrupts
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

5. The Software: A Layered Framework
ROD Crate DAQ
Low-level Software*
Drivers and Libraries
Operating System
Hardware
* Mainly sub-detector specific
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

6. Framework: Hardware Provides common hardware infrastructure VME:
Choice of crates, power supplies, and fans
Common guidelines for the use of VME64x
Common purchase & repair service (PH/ESE)
Common “slow control”: ATLAS DCS over CANbus
Single-board computers:
A family of three generations of single-board computers: Intel-based central processing unit and PCI/X-VME bridge
Additional Modules:
ROD_Busy Module
TTC Modules: TTCvi, TTCex, etc.
LTP, LTPI
Other modules available from the Electronics Pool which are used in a ROD Crate framework, e.g. at test beam and in lab tests, and for which common low-level software exists
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

7. Framework: Operating System
Provides common operating system and environment
Common Service:
Provided by CERN/IT & ATLAS TDAQ System Administrators
Operating System  Scientific Linux CERN
Boot configuration (in particular loading DAQ drivers)
Security (at Point 1)
System/network monitoring (at Point 1)
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

8. Framework: Drivers&Libraries - 1
Provides common drivers and software infrastructure
Common VME Driver (pure CERN product):
Dynamically loaded driver with static mappings (vmetab file)
Single READ and WRITE
FAST: uses mapped I/O (ignore VME bus errors, requires SBC with h/w for byte swapping)
SAFE: uses programmed I/O under driver control (catch VME bus errors)
Slave mappings (SBC memory as slave on VME)
To my knowledge not used
Block transfer
Allows several pipelined chains of block transfers.
Interrupt handling
Wait synchronously on semaphore or register POSIX signal with driver
To my knowledge not often used
Other features:
CR/CSR access, SYSRST generation, SYSFAIL and bus error signalling, and debugging features
Provides C library and C++ wrapper
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

9. Framework: Drivers&Libraries - 2
Provides common drivers and software infrastructure
Other drivers and libraries:
PCI:
Read/write memory I/O
Link/unlink PCI devices
Could be reused in an xTCA environment.
Contiguous memory
Kernel pages (or previously BIG PHYS area)
Other libraries:
“Helpers”:
Errors codes, time stamps, bit strings, JTAG chains, simple menu programs, XML module templates, etc.
And their tools for testing and debugging
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

10. Framework: Low-level Software
Provides sub-detector specific software
Sub-detector specific diagnostic tools
for early development or debugging
Sub-detector specific calibration tasks
detailed knowledge of the calibration and front-end system
Libraries (C++) which “encapsulate” the specificities so that the ROD (or other modules) can be used from the next higher level
This is probably the greater part of the software base.
Common low-level software
Common libraries for common modules, e.g. LTP, TTCvi, ROD_BUSY, etc.
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

11. Framework: ROD Crate DAQ
Provides Integration into ATLAS DAQ:
Based on a skeleton program* (C++) which needs to be filled in with sub-detector specific calls + a set of common modules ready to be used. * A ReadoutApplication originally developed for the ROS DAQ
Run Control:
Distributed system of finite-state machines (controllers)
Configuration Databases:
Run parameters, calibration data, trigger configuration, etc.
Monitoring:
Status/error messages, status values, histograms, event monitoring, data quality, persistent information, etc.
Ready-to-use Modules:
Controller, configuration, and monitoring are readily available for common modules (e.g. ROD_BUSY) and other DAQ tasks
Is based on the same software as the rest of the ATLAS TDAQ (ROS, L2, and EF). Have to see how that one evolves!
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

12. History From my recollection: Detector Interface Group (DIG):
Define interface between sub-detectors and DAQ
Define DAQ functionality required by sub-detectors
TDAQ Interfaces with Front-end Systems Requirements Document + Readout Link (SLink) and Raw Event Format
ROD Working Group (sub-group of DIG):
Exchange of information on ROD hardware
Define common solutions, e.g. VME hardware/driver & common modules
ROD Crate DAQ Task Force (sub-group of DIG):
Define common DAQ for the ROD crate → RCD
Common solutions have proven to be useful in terms of effort for development and maintenance (in particular for smaller groups).
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012

13. Outlook What will replace the existing services/infrastructure?
If we intend to replace VME:
What will replace the existing services/infrastructure?
Common solutions in terms of common hardware/software?
Common service for purchase & repair?
Common DCS?
What architecture/protocol to use?
Architecture: Based on SBC which distributes commands to modules? Common driver and library?
Protocol: Single or grouped READ/WRITE? Block transfers? Interrupts?
Performance: What latency for control and throughput for monitoring are required?
Common ROD Crate DAQ (same for all sub-detectors? same as ROS DAQ)?
Common modules?
How to arrive at common solution?
Do we need another ROD Working Group?
What is the manpower required to develop and maintain common framework? And who can provide that manpower?
Must not forget the VME legacy. It will be around until phase and probably beyond.
R. Spiwoks
Possible Replacement of VME for the Upgrades of ATLAS, CERN, 12-JUL-2012