1. TDAQ commissioning and status Stephen Hillier, on behalf of TDAQ
System Overview
Level-1 Trigger
HLT infrastructure
DAQ infrastructure
DAQ software
Commissioning Runs
ATLAS Control Room during a TDAQ technical run
Everything under control!
February 12th, 2008
TDAQ commissioning and status

2. Trigger/DAQ architecture
Detector Front
End Electronics
(Detector responsibility)
Detector RODs
Level-1 Trigger
Custom Pipelined
Hardware
Region of Interest Builder
Custom Hardware
Readout System
Custom built buffers in ROS PC farm
High-Level Trigger
Large PC farm
High data bandwidth
Dedicated ‘Data’ Network
Event Building
More PC farms
on ‘data’ network
DAQ software – control, configuration, monitoring (control network)
February 12th, 2008
TDAQ commissioning and status

3. TDAQ commissioning and status
Level-1 Trigger System
Three major systems
Calorimeter Trigger
Muon Trigger
Central Trigger Processor (CTP)
Other triggers and signals also integrated by CTP
Minimum bias
Luminosity triggers
Beam Pick-up
CTP distributes all timing information
February 12th, 2008
TDAQ commissioning and status

4. Central Trigger Processor (CTP)
Core CTP available since 2006
Installation now complete
New trigger inputs added as they come online
Essential for detector commissioning
Continuous heavy demand, hence introduction of…
Local Trigger Processors Interfaces (LTPi)
Allow flexible subdetector local triggering across multiple LTPs
30 modules produced, 5 tested
Installation architecture planned for ‘Calorimeter Loops’
Disscussion of ID and muon usage required
Documentation and software available
February 12th, 2008
TDAQ commissioning and status

5. Level-1 Calorimeter Trigger
Installation in USA15 completed in December 2007
Almost 300 complex custom modules
Plus many simpler ones
Spread over 10 racks
Over 3000 cables laid
~800 thick analogue electrical
~2000 thin digital electrical
~400 optical fibres
Commissioning with Calorimeters well underway
In M3, M4, M5
Dedicated ‘Calorimeter’ weeks
Jet triggers already used, electron/gamma/tau now ready
February 12th, 2008
TDAQ commissioning and status

6. TDAQ commissioning and status
Level-1 Muon Trigger
On-Detector Electronics
Essentially complete
But Power Supply delivery is slow
Off-Detector Electronics
RODs, sector logic etc
Complete by end March
CTP interface components available
Cables, Fibres almost finished
February 12th, 2008
TDAQ commissioning and status

7. Level-1 Trigger Commissioning
Calorimeter trigger signals need thorough testing before access disappears
Probably end of April
About half tested so far
Muon trigger commissioning currently done sector by sector
Availability of gas and power supplies
Timing needs to be addressed
Require all triggers to have same timing w.r.t. (non-existent) bunch-crossing
The complete system – calo, muon - is rarely (or rather never) available
Timing workshop, 11th March
HAD
Correlation b/w Trigger Tower
(Level-1Calo) and Tile Energy
Correlation b/w hits in RPC and MDT
February 12th, 2008
TDAQ commissioning and status

8. Region of Interest Builder
Installed and working for a long time
Takes 8 inputs
1 CTP
1 Muon Trigger
6 Calo Trigger
Region of Interest outputs tested in commissioning runs
February 12th, 2008
TDAQ commissioning and status

9. High Level Trigger Infrastructure
HLT nodes consist of:
Level-2 Supervisors
Level-2 Processing Nodes
Event Filter Nodes
Assignment of roles flexible
eg in M5 64*Level-2, 96*EF
Granularity of rack
Currently 5 racks of PCs
>150 nodes 1U rack mounted dual quad core
Regularly used in commissioning runs
More to come in next three months
Expansion to 830 nodes
35% of foreseen system
Rapid installation (3 racks per week)
February 12th, 2008
TDAQ commissioning and status

10. High Level Trigger Algorithms
HLT algorithm integration tested in ‘technical runs’
Week long and 24 hour
Also Mx weeks
Performance and timing studies
See talk by Frank Winklmeier (this afternoon)
February 12th, 2008
TDAQ commissioning and status

11. Network Infrastructure
Two vital, fast networks
Data network
Dedicated data transport
Control network
Responsive configuration and monitoring
Network expansion in January 2008
Added second control and data core
Data core not yet connected to ROSes
February 12th, 2008
TDAQ commissioning and status

12. TDAQ commissioning and status
DAQ infrastructure
Dataflow view of TDAQ infrastructure:
Underground
Surface
Detectors
Permanent Storage
Readout Drivers
Sub Farm Output
Dedicated Optical Links
Data Flow
Manager
Readout
System
Node
Readout Buffers
Sub Farm Input
Data Network
February 12th, 2008
TDAQ commissioning and status

13. TDAQ commissioning and status
Readout System (ROS)
Detector Readout Drivers feed Readout Buffers (ROBINs) via 1592 Readout Links
All originally foreseen links installed
A few more needed for forward detectors
ROBINs located in 149 ROS PCs
All installed and tested in detector commissioning and technical runs
Plus 4 ‘hot’ spares
Fast network interfaces to data network
February 12th, 2008
TDAQ commissioning and status

14. Sub-Farm Input and Output
Sub Farm Inputs perform Event Building
Only on Level-2 selected events
Prepare events for Event Filter
32 PCs available and used
Event building at ~3kHz (event size 1.5 Mbyte)
Sub Farm Outputs write events to disk
On Event Filter selected events
Write to separate streams
6 available (final number)
300 Mbyte/s to computer centre
February 12th, 2008
TDAQ commissioning and status

15. Plus other necessary pieces
Online services
ATLAS control room PCs
Monitoring farm
32 PCs
File Servers
Gateways
Web servers
Infrastructure all tested during many runs
February 12th, 2008
TDAQ commissioning and status

16. TDAQ commissioning and status
TDAQ software
Provides common tools for
Control
Configuration
Configuration and Conditions Databases
Monitoring
Current version tdaq
Validation via technical run
4th – 11th February
Synchronized with HLT
February 12th, 2008
TDAQ commissioning and status

17. Monitoring and Data Quality
Extensive framework already exists
Many older and more mature tools
Status displays, histogram producers, gatherers, displays, data quality
Newer tools are also available and being improved/extended
Monitoring Data Archive (MDA)
Histograms, information automatically stored in CASTOR
Operational Monitoring Display (OMD)
Configurable display of any published information
Time evolution, statistical analysis
Trigger Presenter (TriP)
Signal point of display of Trigger quantities at all levels
Remote online monitoring becoming a reality:
Web Monitoring Interface has been commissioned
Run Status:
Data Quality:
Online remote access to P1 during data taking is being assessed
See
February 12th, 2008
TDAQ commissioning and status

18. TDAQ commissioning and status
Trigger Presenter
February 12th, 2008
TDAQ commissioning and status

19. Technical Run 4th-11th Feb
Concentration on stress tests
Stress data flow at all levels
Heavy use of new Run Control implementation
Achievements
Stable running (several hours without intervention)
Event sizes from 1 MB to 10 MB
Event Building and Data Writing ran at Gbit link limits
Controlled ~1500 applications over 350 nodes
Successful, though some tweaks needed in error handling
Playback of M4, M5 and Monte Carlo data to test all algorithm slices
Feedback of problems useful for M6 preparation
Generated 1031 data set used to test HLT menu for initial running
February 12th, 2008
TDAQ commissioning and status

20. TDAQ commissioning and status
Technical Run Display
1.5 hour
running time
stable
trigger rate
memory
leak spotted
February 12th, 2008
TDAQ commissioning and status

21. TDAQ commissioning and status
Summary
Vast majority of hardware now installed
And tested in many detector and technical runs
Most of requirements for initial luminosity met
Large program of HLT node installation planned
Commissioning weeks progressively more successful
Mx weeks: More detectors, more stability
Technical runs more ambitious
Work to do
Ongoing software development, improvement and testing
Timing of Level-1 trigger system
More experience with realistic triggers, rates, at all trigger levels
Learn how to operate in 24/7 mode
February 12th, 2008
TDAQ commissioning and status