1. IEEE NPSS Real Time Conference 2009
Commissioning and First Experience of the ALICE Data Acquisition System
Vasco Barroso
ALICE DAQ

2. Outline CERN, LHC & ALICE ALICE DAQ Commissioning Highlights from 2008
Monitoring
Calibration
Storage
Commissioning
Highlights from 2008
Plans for 2009
Conclusion

3. CERN European Organization for Nuclear Research Particle Physics
20 member states
Franco-Swiss border
8000 users, 85 nationalities
Founded in 1954

4. LHC 27 Km ring, 100 m underground
Superconducting temperatures (-271°C)
Particle accelerator
100 m in average
2 beams of particles travel close to speed of light in 2 separate beam pipes kept at ultrahigh vacuum

5. ALICE A Large Ion Collider Experiment
Study of strongly interacting matter and the quark-gluon plasma in nucleus-nucleus collisions at the LHC
Complex experiment for online systems
18 sub-detectors
10 sub-detectors providing input to the trigger
2 types of beams: pp and Pb-Pb
Multiple partitions and multiple detector clusters
> 1 PB of data every year
Not surprisingly: complex commissioning!
Focused primarily on the study of quark-gluon plasma, a state of matter that existed just after the Big Bang but cannot be found today in nature.
pp => very frequent collisions, small events
pb-pb => rare collisions, large events (tens of thousands of particles produced)

6. ALICE
26 x 16 x 16 meters 10,000 tonnes
Overview => ZDC, T0 (number, time), V0 (number), FMD
Tracking detectors => cylindrical (ITS => Inner tracking system, TPC, TRD => Transition Radiator Detector)
Identify detectors => TOF, HMPID TRD
Muon arm
Photons => PHOS, PMD, EMCal

7. ALICE DAQ Event Building Network Storage Network PDS GDC TDSM CTP LTU
TTC
FERO
LDC
BUSY
Rare/All
Event
Fragment
Sub-event
File
Storage Network
PDS
L0, L1a, L2
360 DDLs
D-RORC
EDM
Load Bal.
HLT Farm
FEP
DDL
H-RORC
10 DDLs
10 D-RORC
10 HLT LDC
120 DDLs
DA DQM
DSS
Event Building Network
430 D-RORC
90 Detector LDC
30 GDC
10 TDSM
18 DSS
20 DA/DQM
25 TDS
Archiving on Tape in the Computing Centre (Meyrin)
Trigger system providing signal to the FEE of the detectors. It filters what to readout because we cannot readout everything (too much data, 100 GB/s)
HLT receives data from the LDCs and gives back decision
TDS => Transient Data Storage

8. ALICE DAQ: Control Model
ECS: Experiment Control System DCS: Detector Control System CTP: Central Trigger Processor DAQ: Data Acquisition HLT: High Level Trigger
ECS
DCS
CTP
DAQ
HLT

9. ALICE DAQ: Standalone … ECS DCA DCS TPC LTU TPC Control
DCA: Detector Control Agent LTU: Local Trigger Unit
ECS DCA
DCS TPC
LTU
TPC Control
DAQ Run Control
TPC
HLT
TPC HV
Gas LV
TPC
LTU
LDC 1
LDC 2
LDC
216
Farm …
The Alice Detector is made up of 18 different sub-detectors.
All these sub-detectors need calibration, either because you want to change the configuration, because the components that make them vary with time or because the conditions change.
A characteristic of calibration is that it normally involves collecting large amounts of data or big statistics sets.
Another point is that it should be done online whenever possible.
Why? Because sometimes the detectors need to be configure their electronics and you don’t want to send all that data to the offline facilities, analyse it and send back the results for configuration. This would take too much time.
Another reason is that you normally need the calibration results in order to do a proper reconstruction. By doing it online, on the fly, you avoid an extra pass over the data to extract these calibration results.
Large data volumes => constraint for the architecture
Done online => Don’t use questions.

10. ALICE DAQ: Global PCA SPD DCA TPC DCA SPD DCA Muon DCS TPC LTU TPC
PCA: Partition Control Agent
PCA
SPD
DCA
TPC
DCA
SPD
DCA
Muon
DCS TPC
LTU
TPC
DAQ TPC
HLT
TPC
DCS Muon
LTU
Muon
DAQ Muon
HLT
Muon
DCS SPD
LTU
SPD
DAQ SPD
HLT
SPD
The Alice Detector is made up of 18 different sub-detectors.
All these sub-detectors need calibration, either because you want to change the configuration, because the components that make them vary with time or because the conditions change.
A characteristic of calibration is that it normally involves collecting large amounts of data or big statistics sets.
Another point is that it should be done online whenever possible.
Why? Because sometimes the detectors need to be configure their electronics and you don’t want to send all that data to the offline facilities, analyse it and send back the results for configuration. This would take too much time.
Another reason is that you normally need the calibration results in order to do a proper reconstruction. By doing it online, on the fly, you avoid an extra pass over the data to extract these calibration results.
Large data volumes => constraint for the architecture
Done online => Don’t use questions.

11. Monitoring Data Quality Monitoring System monitoring Operational logs
AMORE framework
System monitoring
LEMON
Operational logs
InfoLogger
Upcoming dedicated tool
Run statistics
ALICE Electronic Logbook

12. “The ALICE Data Quality Monitoring“
Barthélémy von Haller
Today CMS2 17:30-18:50
Monitoring: DQM
Framework in production.

13. Monitoring: ALICE Logbook

14. Calibration: Detector Algorithms
DA = Calibration task running online provided by the sub-detectors teams
Online
Offline
detector
data
reconstruction DA
configure

15. “THE ALICE STORAGE SYSTEM: an Analysis of the Impact on the Performance of the Configuration Parameters and of the Load of Concurrent Streams“
Adriana Telesca
Tomorrow TDA6 17:30-19:00
Storage
Movers => GDCs => out of the box
same thing (simpler)
Mesh, full crossbar within the switch
Disk arrays => control access disks, control RAID 6, optimize transfer at disk array level
Filesystem => StorNext to see/access everything (read/write), afinity => directory privilige to RAID volume
1 full LHC spill

16. Commissioning
1st Circulating
beam
10 Sep
Injection tests
8 Aug, 24 Aug
Helium leak
incident
19 Sep
First particles from
machine 15 Jun
2007
2008
2008
2009
Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May
1st global
period
10-21 Dec
2nd global
period
4 Feb-9 Mar
3rd global period
5 May – 20 Oct
Installation & Commissioning
24/7 operation
Installation & Upgrade

17. Commissioning: ACR
Day…
and night

18. Commissioning: Standalone
Standalone commissioning
Services: cabling, cooling, power supplies, etc
Detector hardware, firmware, software
Interfaces to online systems
Online commissioning process
Control procedure up to the state “Ready for Data Taking”
Exposed to variety of triggers (cosmic, pulser up to 40 MHz, random)
Data taking stability tests
Calibration procedure
Integration to global partition
Make online commissioning process shorter

19. Commissioning: Standalone
Explain plot axis
Be careful with progress

20. Commissioning: Global
Exercise stability
Sub-detectors, services, online systems
23 systems  < 0.5 !
Measure performance
Sub-detectors readout time and event size
Online systems
Take data
Cosmic data for sub-detector alignment
Calibration (e.g. TPC with krypton or laser)
Organize operations
Take data => not secondary!

21. Commissioning: Global

22. Commissioning: Global

23. Commissioning: Global runs
All detectors participated, not all all the same time because of different priorities

24. Commissioning: System stability
Stability of global runs has improved substantially during the cosmic runs
Explain axis

25. Commissioning: Totals
Global runs
Cumulated amount of data readout from sub-detectors
515 days data-taking
3 PB read-out
~350 TB recorded to tape in total

26. Commissioning: Period I
TOF
Comparison of hit time distribution in ACORDE and random triggers
SPD
Comparison of number of clusters per event in the 2 SPD layers

27. Commissioning: Period II
TPC
Cluster distribution
Resolution over drift length
MUON_TRK
Online data monitoring vs offline analysis

28. Commissioning: Period III
after
alignment
before
SPD
Top vs bottom alignment
SDD
Drift speed calibration vs position
SSD
p-n charge correlation
Explain better SPD alignment

29. Commissioning: 1st LHC particles
On 15 June 2008, the ALICE SPD in self-triggering mode (L0) sees one of the first “sign of life” of LHC during the beam injection test in Tl2
Run Time 18:10
First particles, not colision, impact points

30. Highlights from 2008 Services Detectors Online systems
Work in progress for firmware and software => readout time and zero suppression (event size)
Noise and grounding
Online systems
Control scalability
Data “reconstruction ready” formatting => CPU intensive
Sub-detector control sequences not yet fully automated
Successful commissioning of ALICE and ready for startup in September ‘08

31. Goals for and after
GDC
TDSM
CTP
LTU
TTC
FERO
LDC
BUSY
Rare/All
Event
Fragment
Sub-event
File
Storage Network
PDS
L0, L1a, L2
360 DDLs
D-RORC
EDM
Load Bal.
HLT Farm
FEP
DDL
H-RORC
10 DDLs
10 D-RORC
10 HLT LDC
120 DDLs
DA DQM
DSS
Event Building Network
430 D-RORC
125 Detector LDC
75 GDC
30 TDSM
18 DSS
60 DA/DQM
75 TDS
Archiving on Tape in the Computing Centre (Meyrin)
GDC
TDSM
CTP
LTU
TTC
FERO
LDC
BUSY
Rare/All
Event
Fragment
Sub-event
File
Storage Network
PDS
L0, L1a, L2
360 DDLs
D-RORC
EDM
Load Bal.
HLT Farm
FEP
DDL
H-RORC
10 DDLs
10 D-RORC
10 HLT LDC
120 DDLs
DA DQM
DSS
Event Building Network
430 D-RORC
90 Detector LDC
30 GDC
10 TDSM
18 DSS
20 DA/DQM
25 TDS
Archiving on Tape in the Computing Centre (Meyrin)
Ready for a nominal data taking year: 10 months pp + 1 month Pb-Pb
Full deployment of the DAQ system
Increase of HLT CPU power
Improve feedback to shift crew (DQM)
Reduce size of shift crews
Group of sub-detectors
Automation of atomic operations (configuration, calibration)
Automation of global planning
Central system configuration tool

32. Run Configuration Tool
Run Coordinator
Data-taking
Configuration
Trigger
Configuration
Time Period
Sub-detectors
Trigger
10:00 – 12:00
All
SPD
12: :00
ITS, TPC, V0, TRD
TOF => TRD
22:00 – 08:00
Muon Detectors
MUON_TRG …
ALICE
(sub-detectors,
online systems)

33. Installation & Upgrade
Planning for
LHC Startup
LHC Shutdown
2008
2009
2009
2010
Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar ...
4th global period
Installation & Upgrade
Standalone tests
24/7 operation

34. Conclusion
ALICE became a reality after almost 15 years of design and installation
Commissioning of sub-detectors and online systems lasted from Dec ‘07 to Sep ‘08. Work intensive!
Online systems contributed to the sub-detector commissioning, alignment and calibration
Experiment was ready to start with beam in Sep ‘08

35. Ready again this year!

36. Commissioning: Session I
December 2007
Global runs in 2 weeks
14 detectors participating
10-60 hours of data taking – 1-20 x 106 triggers

37. Commissioning: Session II
February/March 2008
Global runs in 5 weeks
13 detectors participating
hours of data taking – 2-50 x 106 triggers

38. Commissioning: Session III
May/October 2008
Global runs in 23 weeks
16 detectors participating
hours of data taking – x 106 triggers