1. Connecting LRMS to GRMS
Jeff Templon
PDP Group, NIKHEF
LCG/EGEE Workshop
Bologna
25 May 2005

2. Example Site Scenario Computer cluster at NIKHEF:
50% guaranteed for SC-Grid
50% guaranteed for LHC experiments
Allow either group to exceed 50% if other group not active
Allow D0 experiment to scavenge any crumbs
Give ‘dteam’ (operations group) extremely high priority; but limit to 2 concurrent jobs
Limit running jobs from production groups to ~ 95% of capacity (always keep a few CPUs free for e.g. operations checks)

3. Example User Scenarios
“polite user”
Uses grid job submission tools ‘bare’ & lets grid figure it out
“high-throughput user”
Ignores grid suggestions on sites; blast each site until jobs start piling up in ‘waiting’ state, then go to next site
“sneaky high-throughput user”
Like above but doesn’t even look at whether jobs pile up … jobs aren’t real jobs, they are ‘pilot’ jobs (supermarket approach)
“fast turnaround user”
Wants jobs to complete as soon as possible (special priority)

4. Connect Users to Sites with “Maximal Joint Happiness”
Users: work finished ASAP
Sites: always full and usage matches fair-share commitments

5. Current Strategy: Shortest Time to Start
Users: want to submit to sites that will complete job as fast as possible
Sites: site may be “full” i.e. no free CPUs BUT:
HIKHEF 100% full for ATLAS means that
Any ‘SC-Grid’ jobs submitted will run as soon as a free CPU appears
Must get this message to WMS, else won’t get any SC-Grid jobs
Should be clear from this that answer to “how long” depends on who is asking!

6. This Can Be Done Support Will Be in Next Glue Schema dteam ATLAS
Time to start (sec)
Real Time -> (sec)
Black lines are measured, blue triangles are statistical predictions
Support Will Be in Next Glue Schema

7. How Long to Run not yet included
Need reasonable normalized estimates from users
Need normalized CPU units
Need solution for heterogeneous CPU population behind most site’s grid entry points (most of us have these)
Glue Schema support (I think) is also in next version
Added value: good run-time estimates helps LRMS scheduling (eg MPI jobs & backfill)

8. But does it matter?

9. Sneaky HT vs Polite Users
Sneaky Ignores ERT; Polite almost always loses
Sneaky good for sites to 0th order – mix of waiting jobs allows good scheduling … but only if various groups are all sneaky
However
Templon needs to run 10 jobs
Submits 10 jobs to each of 100 sites in grid
First ten to start grab the ‘real’ jobs
Other 990 look exactly like black hole jobs
Waste ~ 16 CPU hrs (2 min scheduling cycle * 500 passes)

10. Polite Users Also Lose in WMS
High overhead in WMS: avg 250 sec life cycle for 20 sec job!
Two hour job
Single user
Single RB
Best RB perf
Sched cycle is only delay at site
Grid Speedup
Number of Jobs Submitted

11. Alt: Schedule On Slots? Advertise EJS instead of ERT
What is more important:
Run quickly -> preempt running over-fair-share jobs, or
No preemption … new jobs will be scheduled as resources freed
WMS could work more like negotiator : “submit 20 jobs to NIKHEF” … (about two per second)
Might solve fair-share target problems & slow WMS problems
Will certainly have its own problems (for SB)

12. Pull Model?
Turns problem around: TQs need to advertise how many of which jobs they each have
Sites can ask for jobs they want
Not clear how scaling goes …
Get list of jobs per VO at TQs
Look at fair-share
Try to get jobs of desired type
If fail, go to next type …
Is it any quicker?

13. High Priority Users
Sol’n 1: dedicated CPUs (standing reservations) (expensive!)
Soln’ 2: virtualization w/preemption (long way off?)

14. Other Issues Transferring Info to LRMS
Run-time estimate
helps enormously in e.g. scheduling MPI jobs
Also may help in answering “the question”
Memory usage, disk space needs, etc etc
MPI & accounting – what about “the dip”?
Self-disabling sites (avoid hundreds of lost jobs and tens of lost person-hours)
“Circuit breakers”? (Miron Livny)