1. The ATLAS Computing Model
BTeV Talk
5/3/2019
The ATLAS Computing Model
Roger Jones
Lancaster University
CHEP06
Mumbai 13 Feb. 2006
For January Collab Meeting

2. Brief summary ATLAS Facilities and their roles
BTeV Talk
5/3/2019
Overview
Brief summary ATLAS Facilities and their roles
Growth of resources
CPU, Disk, Mass Storage
Network requirements
CERN↔ Tier 1 ↔ Tier 2
Operational Issues and Hot Topics
RWL Jones Feb Mumbai
For January Collab Meeting

3. BTeV Talk
5/3/2019
Computing Resources
Computing Model fairly well evolved, documented in C-TDR
Externally reviewed
There are (and will remain for some time) many unknowns
Calibration and alignment strategy is still evolving
Physics data access patterns MAY be exercised from June
Unlikely to know the real patterns until 2007/2008!
Still uncertainties on the event sizes , reconstruction time
Lesson from the previous round of experiments at CERN (LEP, )
Reviews in 1988 underestimated the computing requirements by an order of magnitude!
RWL Jones Feb Mumbai
For January Collab Meeting

4. ATLAS Facilities Event Filter Farm at CERN Tier 0 Center at CERN
BTeV Talk
5/3/2019
ATLAS Facilities
Event Filter Farm at CERN
Located near the Experiment, assembles data into a stream to the Tier 0 Center
Tier 0 Center at CERN
Raw data  Mass storage at CERN and to Tier 1 centers
Swift production of Event Summary Data (ESD) and Analysis Object Data (AOD)
Ship ESD, AOD to Tier 1 centers  Mass storage at CERN
Tier 1 Centers distributed worldwide (10 centers)
Re-reconstruction of raw data, producing new ESD, AOD
Scheduled, group access to full ESD and AOD
Tier 2 Centers distributed worldwide (approximately 30 centers)
Monte Carlo Simulation, producing ESD, AOD, ESD, AOD  Tier 1 centers
On demand user physics analysis
CERN Analysis Facility
Analysis
Heightened access to ESD and RAW/calibration data on demand
Tier 3 Centers distributed worldwide
Physics analysis
RWL Jones Feb Mumbai
For January Collab Meeting

5. Processing Tier-0: Tier-1:
BTeV Talk
5/3/2019
Processing
Tier-0:
Prompt first pass processing on express/calibration physics stream
24-48 hours later, process full physics data stream with reasonable calibrations
Implies large data movement from T0 →T1s
Tier-1:
Reprocess 1-2 months after arrival with better calibrations
Reprocess all resident RAW at year end with improved calibration and software
Implies large data movement from T1↔T1 and T1 → T2
RWL Jones Feb Mumbai
For January Collab Meeting

6. Analysis model Analysis model broken into two components
BTeV Talk
5/3/2019
Analysis model
Analysis model broken into two components
Scheduled central production of augmented AOD, tuples & TAG collections from ESD
Derived files moved to other T1s and to T2s
Chaotic user analysis of augmented AOD streams, tuples, new selections etc and individual user simulation and CPU-bound tasks matching the official MC production
Modest job traffic between T2s
RWL Jones Feb Mumbai
For January Collab Meeting

7. Inputs to the ATLAS Computing Model (1)
BTeV Talk
5/3/2019
Inputs to the ATLAS Computing Model (1)
RWL Jones Feb Mumbai
For January Collab Meeting

8. Inputs to the ATLAS Computing Model (2)
BTeV Talk
5/3/2019
Inputs to the ATLAS Computing Model (2)
RWL Jones Feb Mumbai
For January Collab Meeting

9. Data Flow Tier 0 view Tier 2 view EF farm  T0
BTeV Talk
5/3/2019
Data Flow
EF farm  T0
320 MB/s continuous
T0 Raw data  Mass Storage at CERN
T0 Raw data  Tier 1 centers
T0 ESD, AOD, TAG  Tier 1 centers
2 copies of ESD distributed worldwide
T1  T2
Some RAW/ESD, All AOD, All TAG
Some group derived datasets
T2  T1
Simulated RAW, ESD, AOD, TAG
T0  T2 Calibration processing?
Tier 0 view
Tier 2 view
RWL Jones Feb Mumbai
For January Collab Meeting

10. ATLAS partial &“average” T1 Data Flow (2008)
RAW
ESD (2x)
AODm (10x)
1 Hz
85K f/day
720 MB/s
Tape
RAW
ESD2
AODm2
0.044 Hz
3.74K f/day
44 MB/s
3.66 TB/day
RAW
1.6 GB/file
0.02 Hz
1.7K f/day
32 MB/s
2.7 TB/day
Tier-0
disk buffer
AODm1
500 MB/file
0.04 Hz
3.4K f/day
20 MB/s
1.6 TB/day
AODm2
500 MB/file
0.04 Hz
3.4K f/day
20 MB/s
1.6 TB/day
ESD1
0.5 GB/file
0.02 Hz
1.7K f/day
10 MB/s
0.8 TB/day
AODm1
500 MB/file
0.04 Hz
3.4K f/day
20 MB/s
1.6 TB/day
RAW
1.6 GB/file
0.02 Hz
1.7K f/day
32 MB/s
2.7 TB/day
AOD2
10 MB/file
0.2 Hz
17K f/day
2 MB/s
0.16 TB/day
ESD2
0.5 GB/file
0.02 Hz
1.7K f/day
10 MB/s
0.8 TB/day
AOD2
10 MB/file
0.2 Hz
17K f/day
2 MB/s
0.16 TB/day
AODm2
500 MB/file
0.004 Hz
0.34K f/day
2 MB/s
0.16 TB/day T1
Each Tier-2
Plus simulation and analysis data flow
CPU farm
ESD2
0.5 GB/file
0.02 Hz
1.7K f/day
10 MB/s
0.8 TB/day
AODm2
500 MB/file
0.036 Hz
3.1K f/day
18 MB/s
1.44 TB/day
AODm2
500 MB/file
0.004 Hz
0.34K f/day
2 MB/s
0.16 TB/day
ESD2
0.5 GB/file
0.02 Hz
1.7K f/day
10 MB/s
0.8 TB/day
ESD2
0.5 GB/file
0.02 Hz
1.7K f/day
10 MB/s
0.8 TB/day
AODm2
500 MB/file
0.036 Hz
3.1K f/day
18 MB/s
1.44 TB/day T1
Other Tier-1s
disk storage T1
Other Tier-1s
RWL Jones Feb Mumbai

11. Total ATLAS Requirements in for 2008
BTeV Talk
5/3/2019
Total ATLAS Requirements in for 2008
RWL Jones Feb Mumbai
For January Collab Meeting

12. Important points: Discussion on disk vs tape storage at Tier-1’s
BTeV Talk
5/3/2019
Important points:
Discussion on disk vs tape storage at Tier-1’s
Tape in this discussion means low-access secure storage
No ‘disk buffers’ included except input to Tier 0
Storage of Simulation data from Tier 2’s
Assumed to be at T1s
Need partnerships to plan networking
Must have fail-over to other sites
Commissioning
Requirement of flexibility in the early stages
Simulation is a tunable parameter in T2 numbers!
Heavy Ion running still under discussion.
RWL Jones Feb Mumbai
For January Collab Meeting

13. ATLAS T0 Resources RWL Jones 13 Feb. 2006 Mumbai BTeV Talk 5/3/2019
For January Collab Meeting

14. ATLAS T1 Resources RWL Jones 13 Feb. 2006 Mumbai BTeV Talk 5/3/2019
For January Collab Meeting

15. ATLAS T2 Resources RWL Jones 13 Feb. 2006 Mumbai BTeV Talk 5/3/2019
For January Collab Meeting

16. Required Network Bandwidth
BTeV Talk
5/3/2019
Required Network Bandwidth
Caveats
No safety factors
No headroom
Just sustained average numbers
Assumes no years/datasets are ‘junked’
Physics analysis pattern still under study…
RWL Jones Feb Mumbai
For January Collab Meeting

17. Mainly outward data movement
BTeV Talk
5/3/2019
T1 ↔ CERN Bandwidth I+O
The projected time profile of the nominal bandwidth required between CERN and the Tier-1 cloud.
Mainly outward data movement
RWL Jones Feb Mumbai
For January Collab Meeting

18. About half is scheduled analysis
BTeV Talk
5/3/2019
T1 ↔ T1 Bandwidth I+O
About half is scheduled analysis
The projected time profile of the nominal bandwidth required T1 and T1 cloud.
RWL Jones Feb Mumbai
For January Collab Meeting

19. T1↔ T2 Bandwidth I+O Dominated by AOD
BTeV Talk
5/3/2019
T1↔ T2 Bandwidth I+O
Dominated by AOD
The projected time profile of the nominal aggregate bandwidth expected for an average ATLAS Tier- 1 and its three associated Tier-2s.
RWL Jones Feb Mumbai
For January Collab Meeting

20. Issues 1: T1 Reprocessing
Reprocessing at Tier 1s is understood in principle
In practice, requires efficient recall of data from archive and processing
Pinning, pre-staging, DAGs all required?
Requires the different storage roles to be well understood
RWL Jones Feb Mumbai

21. Issues 2: Streaming This is *not* a theological issue
All discussions are about optimisation of data access
TDR has 4 streams from event filter
primary physics, calibration, express, problem events
Calibration stream has split at least once since!
At AOD, envisage ~10 streams
ESD streaming?
Straw man streaming schemes (trigger based) being agreed
Will explore the access improvements in large-scale exercises
Will also look at overlaps, bookkeeping etc
RWL Jones Feb Mumbai

22. TAG Access TAG is a keyed list of variables/event Two roles
Direct access to event in file via pointer
Data collection definition function
Two formats, file and database
Now believe large queries require full database
Restricts it to Tier1s and large Tier2s/CAF
Ordinary Tier2s hold file-based TAG corresponding to locally-held datasets
RWL Jones Feb Mumbai

23. BTeV Talk
5/3/2019
Conclusions
Computing Model Data Flow understood for placing Raw, ESD and AOD at Tiered centers
Still need to understand data flow implications of Physics Analysis
SC4/Computing System Commissioning in 2006 is vital.
Some issues will only be resolved with real data in
RWL Jones Feb Mumbai
For January Collab Meeting

24. Backup Slides
RWL Jones Feb Mumbai

25. Heavy Ion Running RWL Jones 13 Feb. 2006 Mumbai BTeV Talk 5/3/2019
For January Collab Meeting