1. The Performance and Scalability of the back-end DAQ sub-system
Igor SOLOVIEV
CERN ATLAS DAQ/EF-1
4/17/2019

2. Contents Introduction Test Results Summary and Future
ATLAS DAQ/EF P-1 project
back-end software overview & architecture
Test Results
component test results
integrated back-end sub-system test results
Summary and Future
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

3. ATLAS DAQ/EF P-1 Project
Goal:
to produce a prototype system representing a “full slice” of a DAQ suitable for evaluating candidate technologies and architectures for the final ATLAS DAQ system
Sub-systems:
Detector Interface
Data-Flow
Event Filter
Back-end
Status
Base-line system developed & working in lab. environment
Exploitation phase up to TDR (2001)
To be used on test-beam (summer 2000)
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

4. Back-end Sub-system
Is used to configure, control and monitor the DAQ system
It excludes management, processing and transportation of physical data
It talks to all the other online systems
(“glue” of the experiment)
More information:
WWW pages:
“Impact of Software Review and Inspection”: talk F331, today 17:50, Doris Burckhart
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

5. Back-end Architecture
Components
split back-end software into groups with similar functionality (Core + TDAQ & detector integration components)
Operational environment:
heterogeneous collection of UNIX workstations, PCs and embedded systems (e.g. PPC on VME under real-time Lynx OS) connected via a local network
developed in C++ and ported to several compilers on
Solaris, Linux, Lynx, HP-UX & Window NT
Design:
use freeware and commercial software:
Tools.h++, OODB, CORBA, CHSM, CLIPS, Motif/Java
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

6. Back-end core components
Configuration Databases
describes all aspects of the configuration
Information Service (IS)
general purpose information exchange facility
Message Reporting System (MRS)
allows software components to report messages in distributed environment
Process Manager (PMG)
performs distributed job control of components
Run Control (RC)
controls configuration and data taking operations
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

7. Component Unit Tests Results
Configuration Databases
used by many components during system start-up
tests done for different OKS configurations (single read-out crate, typical P-1 conf., expected ATLAS DAQ conf.)
on average workstation time to load P-1 conf. , make complete traverse and close is about 1.5 sec. and on PPC VME board the same test requires about 3 sec.
Information Systems (IS & MRS)
used by many components during all phases of system operation (publish/subscribe facilities)
scalable (multiple servers to split the load)
benchmarks done on single workstation and on several computers for different conf. (size, up to clients)
the response time is a few milliseconds
better results for distributed systems
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

8. Component Unit Tests Results
Process Manager
used during the system start-up and shutdown
results obtained on single Solaris workstation
time to start a process is a few 100s milliseconds and slowly increases with the number of managed processes
Run Control
required to change the state of the system
scalable by changing the structure of RC tree
tests on all available workstations (up to 250 controllers)
to change the system’s state with several 10s of nodes varies from several 100s milliseconds up to few seconds depending on the state of the system
the time to change running/configured states is <1 sec.
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

9. Component Unit Tests Conclusions
Unit tests made for back-end core components show that they are in accordance with DAQ P-1 requirements
Similar tests will be done for back-end integration components
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

10. Back-end Sub-system Tests
What:
bring together all the core and several TDAQ/detector integration components
Why:
to simulate the control & configuration of data taking sessions
Where:
back-end servers are running on UNIX workstation
others (PMG agent, LDAQ emulator & RC Ctrl.) on PC running Linux or VME based Power PC CPU board running Lynx OS
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

11. Test Configurations Network PMG Agent LDAQ RC Ctrl PMG Agent LDAQ
G IPC
P IPC
DF IS
PMG IS
RC IS
RDB RM
MRS
MRS L
DAQ Supervisor
IGUI
RC Root Ctrl
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

12. Test Description Done by shell script: start communication services
launch configuration processes via DAQ supervisor
marshal the hierarchy of RC controllers through different states: I - L - C - R - C - R - C - L - I
stop DAQ supervisor
processes
stop servers R C L B I
cold start
setup
warm stop
shutdown
cold stop
warm start
B - booted
I - initialized
L - loaded
C - configured
R - running
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

13. Start-up & warm start/stop
Time (seconds)
Number of processors/crates
PowerPC 100/200 MHz 32/64 MB
Lynx OS
Number of processors/crates
Pentium III 450 MHz 128 MB
Linux
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

14. Start-up & close Time (seconds) Number of processors/crates
PowerPC 100/200 MHz 32/64 MB
Lynx OS
Number of processors/crates
Pentium III 450 MHz 128 MB
Linux
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

15. Back-end system test summary
Results
time to start/stop processes depends on OS, computer architecture and configuration
once all processes started, the time to change system state remains constant (good distributed control)
the use of IS, MRS and conf. db has a negligible effect on the performance
the results even for the largest configurations is in acceptable range (< 1 minute to start-up on Linux)
Known problems
pmg agents started via RSH with long delays (20 sec)
the computers were not dedicated to tests
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

16. Summary & Future Individual back-end component test
done for core components and show that they are in accordance with the DAQ/EF P-1 requirements
similar tests have to be done for integration components
Integrated back-end system tests
performed employing the majority of the components
verified correct component inter-operation, ability to work in a distributed multi-platform environment
gathered performance measurements
Future
more statistics for larger configurations (more hosts)
script improvement and better start-up/shutdown synchronization
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

17. Appendix: Configuration Databases
Importance
are used by many components during initialization
performance is important for system start-up
Results (with OKS)
Time (s)
1 single read-out crate
10 prototype -1
200 expected ATLAS DAQ
Number of crates
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

18. Appendix: Information Service
Importance
used by many components
performance is important during all phases of system operation
Results
scalable (multiple
servers to split the
load)
update medium size
info. results presented
(on single host)
similar to publish
and remove
Update time (ms)
Number of sources
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

19. Appendix: Message Reporting System
Importance
used by many components
performance is important during all phases of system operation
Results
presented tests
obtained on
single host
better results
obtained in
distributed
environment
Report time per message (ms)
Number of senders
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

20. Appendix: Process Manager
Importance
performance is important for system start-up and shutdown
Results
obtained on
single Solaris
workstation
Time per process (ms)
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev

21. Appendix: Run Control Importance Results
required to change the state of the system
Results
scalable by
changing
the structure
of the RC tree
tests done on
all available
workstations
Time, (s)
Number of controllers
4/17/2019
The Performance and Scalability of the back-end DAQ sub-system - CHEP Igor Soloviev