1. The LEGO Train Framework
Andrei Gheata
Costin Grigoras
Jan Fiete Grosse-Oetringhaus

2. Idea Manage trains using MonALISA Automatic testing per wagon
Users register wagons
Train operators compose trains
Automatic testing per wagon
Train file generation
Submission managed by ML (existing LPM infrastructure)
Merging managed by LPM
Aim: allow operators easy running of analysis trains (~weekly) getting output on the scale of 1-2 days
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

3. Configuration & Testing
Base line
Phys Sel
Centr Sel
User A
User B
User C
Train Configuration
New class AliAnalysisTaskCfg
Contains description of wagons (add task macro, libraries, dependencies)
See talk by Andrei on Monday
Testing
Uses alientest04 machine
Downloads AliEn packages (ROOT, AliRoot)
Copies a part of the input data set to the local machine
Runs tests per wagon
Uses syswatch to extract mem/cpu information
Tests also "base line" task which is empty
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

4. Workflow User 1. adds wagons LPM MonALISA Train operator AliEn config
2. composes train
4. recompose after test
test results
6. runs train
Test machine
train files
3. generates test files + executes test
5. generates train jdl + scripts
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

5. Screenshot Handler configuration Wagon configuration
Data configuration
Testing and running status
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

6. Handler
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

7. Wagon
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

8. Dataset
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

9. Run
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

10. Syswatch
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

11. Operator Workflow Select dataset Select wagon Start testing
Inspect output
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

12. Operator Workflow (2) status of analysis status of merging
intermediate
merging steps
Submit final
merge job
(to be automatized)
final merging
status
check output
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

13. Demo… Enough theory, let's do some clicking…
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

14. Some More Details Train runs with an analysis tag
All code + "AddTask" macro has to be in the tag (no par file!)
Output per run stored in the input data directory (like AOD, QA trains). E.g.: /alice/data/2010/LHC10h/ /ESDs/pass2/PWG4/ CorrelationTrain/7_ _1350
All merged runs found in /alice/cern.ch/user/a/alitrain/PWG4/CorrelationTrain/ 7_ _1350/merge
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

15. Operations After 10-12h most jobs are done (~90-98%)
Few running, few waiting
This situation can persist for days  killer for merging the output
Solutions
Kill jobs that have waited longer than X (being tested on the level of the LPM, better as a JDL tag)
Remove CE requirement after a certain time (thx Latchezar for this idea), to be implemented
Merge jobs have the same tails of few jobs that wait a long time
Ideas: same as above or run them on any CE (problem with splitting, Pablo is investigating)
Output available after ~2 days
25% (real time) spend in running
75% in merging
I believe this can still be improved!
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

16. Operations (visually…)
Analysis jobs
Waiting
Running
Done
Error
Analysis jobs
Waiting
Running
Done
Error
80% done
in 4 hours
hours since submission
Merging jobs
Waiting
Running
Done
Error
hours since submission
here we kill the remaining ones
The LEGO Framework - Jan Fiete Grosse-Oetringhaus
hours since submission

17. Current Trains Four active beta testers
Jets (Christian KB)
D2H (Zaida)
Correlations in pp (Eva)
Correlations in PbPb (JF)
We got a lot of feedback, improved the system
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

18. TODO
Graphs for CPU/Wall/Mem consumption of user tasks as function of AliRoot tag
Some improvements in the web interface
Automatic launching of final job
The LEGO Framework - Jan Fiete Grosse-Oetringhaus

19. Documentation Mailing list (for operators) TWiki (Users + operators)
TWiki (Users + operators)
The LEGO Framework - Jan Fiete Grosse-Oetringhaus