1. Analysis Trains - Reloaded
Andrei Gheata
Costin Grigoras
Jan Fiete Grosse-Oetringhaus

2. Idea
Presented in the offline meeting in June and the offline week that followed
Manage trains using MonALISA
Users register wagons
Train operators compose trains
Automatic testing per wagon
Train file generation
Submission managed by ML (existing LPM infrastructure)
Jan Fiete Grosse-Oetringhaus

3. Configuration & Testing
Train Configuration
New class AliAnalysisTaskCfg
Contains description of wagons (add task macro, libraries, dependencies)
Reads/writes to a text file format (used to read train configuration from ML)
Testing
Uses alientest04 machine
Downloads AliEn packages (ROOT, AliRoot)
Copies a part of the input data set local
Runs tests per wagon
Uses syswatch to extract mem/cpu information
Tests also "base line" task which is empty
Phys Sel
Centr Sel
User A
User B
User C
Jan Fiete Grosse-Oetringhaus

4. Workflow User 1. adds wagons LPM MonALISA Train operator AliEn config
2. composes train
4. recompose after test
test results
6. runs train
Test machine
train files
3. generates test files + executes test
5. generates train jdl + scripts
Jan Fiete Grosse-Oetringhaus

5. Screenshot Handler configuration Wagon configuration
Data configuration
Testing and running status
Jan Fiete Grosse-Oetringhaus

6. Handler
Jan Fiete Grosse-Oetringhaus

7. Wagon
Jan Fiete Grosse-Oetringhaus

8. Dataset
Jan Fiete Grosse-Oetringhaus

9. Run
Jan Fiete Grosse-Oetringhaus

10. Syswatch
Jan Fiete Grosse-Oetringhaus

11. Demo… Enough theory, let's do some clicking…
Jan Fiete Grosse-Oetringhaus

12. Some More Details Train runs with an analysis tag
All code + "AddTask" macro has to be in the tag (no par file!)
Output stored in the input data directory (like AOD, QA trains). E.g.: /alice/data/2010/LHC10h/ /ESDs/pass2/PWG4/CorrelationTrain/7_ _1350
Current infrastructure only allows per-run merging
Jan Fiete Grosse-Oetringhaus

13. Open Issues ROOT AOD analysis
Fix in TGridJDL was required. Is in v patches, but not yet deployed in the Grid. Needed for train operation
AOD analysis
Found huge leak even in empty analysis (20kb/event)
Jan Fiete Grosse-Oetringhaus

14. Old Slides
Jan Fiete Grosse-Oetringhaus

15. Idea Setting up and operating analysis trains is a lot of work
Specific settings for each wagon
Wagons have bugs, leaks etc.
 Automatic configuration needed
 Automatic testing needed (on a subset of the same data the train will run on)
We have collected some ideas which we want to give a try
using at the beginning the PWG4 train
Jan Fiete Grosse-Oetringhaus

16. High Level Description
Train runs on analysis tag (no modifications allowed)
User registers task
Train operator triggers train test
Test results are fed back to Monalisa where the user & operator can see them
Operator starts train with tasks that succeeded and have no (significant) leaks
These steps are operated from MonaLisa
Jan Fiete Grosse-Oetringhaus

17. Some Technical Details
Container that contains task configuration (already shown by Andrei)
Currently identified configuration items
Location of AddTask macro + parameters
Required libraries
Tasks that have to run before
Train testing
Tasks are tested one by one
On subset of data on which the train will run
CPU/Real time, memory extracted w.r.t baseline
Baseline from test with just PhysSel + Centrality
Train macro generation
By analysis framework using the wagons selected by the operator
Macros for testing (wagon by wagon)
Macros for full train (all wagons)
Overall train submission
Using the already existing ML submission framework (including merging jobs)
Phys Sel
Centr Sel
User A
User B
User C
Jan Fiete Grosse-Oetringhaus