1. VGrADS Tools Activities
Chuck Koelbel VGrADS Workshop, February 23, 2006

2. Tools: Where We Are Achieved Working On It
Initial workflow scheduling methods
“Anirban scheduler” [Rice, UH, UCSD, ISI]
Supported by performance prediction, NWS
Initial fault tolerance implementations
FT-MPI [UTK]
Optimal checkpoint scheduling [UCSB]
Platform-independent application launch and optimization
LLVM, run-time reoptimization experiments [Rice]
Working On It
Virtual Grid scheduling methods
Building workflow DAGs

3. Ongoing Tools Thrusts Scheduling methods Performance prediction Other
Most of rest of talk [Rice, UCSD, UCSB, UTK, ISI]
All based on pre-scheduling (aka off-line scheduling) of workflow (aka datflow, aka DAGs) using performance prediction
Performance prediction
Queue delay model [UCSB]
Other
Launching and reoptimization [Rice]
DAG construction [Rice]

4. Scheduling Methods Two-level (choose VG, map onto it)
Richard Huang (UCSD), Anirban Mandal & Ryan Zhang (Rice)
Batch queue (include est. queue delay in cost model)
Anirban Mandal (Rice), Dan Nurmi (UCSB)
Cluster (assign block of tasks to cluster)
Anirban Mandal (Rice)
Provisioning (minimize reservation time + execution time)
Gurmeet Singh (ISI)
Robust (schedule to reduce sensitivity to variability)
Zhiao Shi (UTK)

5. Scheduling Comparison
Objective function
Min / Max
Costs
Important Assumptions
Two-level
Scheduler + Makespan
DAG nodes & edges; (Rice) Proc type
Dedicated resources available for duration of application
Batch queue
Scheduler + Makespan + Queue delays
DAG nodes & edges; Proc type, Queue delay
Resources controlled by queues, but no relevant allocation limits
Cluster
DAG nodes & edges; Proc type
Dedicated resources available for duration of application; homogeneous DAG nodes on each level
Provisioning
Reservation time and Sched quality
Reservation, Schedule costs
Resources controlled by provisioning mechanism
Robust
Robustness subject to makespan
DAG edges(?)
Shared resources resulting in runtime performance variability

6. Results Maximize f Huang - 2-level scheduler, Montage DAG
Mandal - cluster scheduler, EMAN DAG
Maximize f
Shi - robust scheduler, ??? DAG

7. Tools Research Going Forward
Interface between vgES and schedulers
What capabilities can schedulers expect from vgES?
How can schedulers exploit this capability?
How can schedulers work around this capability?
Some interesting operating points
vgES provisions VG / application takes what’s given
vgES returns shared VG nodes / application adapts to perf variance
vgES returns queued VG resources / application manages queues
vgES provisions VG, monitors for additional resources / application starts immediately, adapts to changes

8. Tools Research Going Forward
Generating vgDL request for 2-level methods
Balance request complexity vs. difficulty scheduling onto VG
VG1 = ClusterOf (node) [1:N] [Rank=Cluster.nodes] {node = [CPU=Opteron]}
VG2 = ClusterOf (node) [1:N] [Rank=Cluster.nodes*node.clock] {node = [CPU=Opteron]}
VG3 = ClusterOf (node) [1:N] [Rank=PerfModel(Cluster.nodes,Cluster.bw,node.clock,node.mem)] {node = [CPU=Opteron]}
Automatic vgDL generation from DAGs
Template-driven? Heuristic-driven?
Extended vgDL capabilities
Global constraints (e.g. total # of nodes)
Temporal constraints (e.g. available within 60 min)
Probabalistic constraints (e.g. 95% likely to succeed)

9. Tools Research Going Forward
New scheduling criteria
Deadline scheduling
Economic scheduling
Real-time scheduling
New scheduling situations
Rescheduling
Adapting to new resources
Adapting to resource failures
Incremental scheduling
Managing dynamic applications
“Horizon scheduling” for limited-time predictions
Hybrid static / dynamic scheduling
Contingency scheduling
Static planning for dynamic optimizations

10. Backup Slides Beyond This Point

11. Two-Level Scheduling (Huang)
Target Application
Workflows represented by DAG
Performance Metrics
Application Turn-Around Time
Resource Selection
Scheduling Time
Application Makespan
Major Assumptions of the Scheduler
Resources are dedicated
Resources available for duration of application
Scheduling Algorithms (so far)
Greedy
Modified Critical Path

12. Experimental Setup
Use synthetic resource generator to generate 1000 clusters (33,667 hosts)
Execute one “simple” (greedy) and one “complex” (Modified Critical Path) scheduling heuristic
Tests on Montage DAG
Scheduling
Heuristic
Resources
Complex
Resource Universe
Top x percent Fastest Hosts
Appropriate Virtual Grid
Simple

13. Initial Results
Original CCR
CCR = 0.1
Two-phase scheduling necessary to avoid excessive scheduling time
Appropriate virtual grids necessary for better performance
Using more complex heuristic did not improve performance if you have the appropriate resource abstractions!

14. Batch Queue Scheduling (Mandal)
Make batch-queue predictions on-the-fly from the ``live’’ systems
New NWS functionality
Parameterize the performance models using the 95% upper bound on the median prediction as a prediction of delay
The performance models can take into account the amount of time needed to start a computation
Run a top-down (heuristic) scheduler to choose a resource set
Scheduler is smart enough to understand that the start-up delay can be amortized
Joint work with Dan Nurmi and Rich Wolski

15. Top-Down Scheduling Top-Down For each heuristic
Until all components mapped
Map available components to resources
Select mapping with minimum makespan
While all available components not mapped
For each (component, resource) pair
ECT(c,r) = rank(c,r) + EAT(r)
End For each
Run min-min, max-min and sufferage
Store mapping
End while
Top-Down

16. Scheduling onto Batch-Queue Systems
Details: Modification of Top-Down scheduler
At every scheduling step, take into account the estimated time the job has to wait in the queue in the estimated completion time for the job [ECT(c,r) in the algorithm]
Keep track of the queue wait times for each cluster and the number of nodes that correspond to the queue wait time
With each mapping, update the estimated availability time [EAT in the algorithm] with the queue wait time, as required

17. Scheduling onto Batch-Queue Systems: Example
Cluster 0
Cluster 1
Input DAG R0 R1 R2 R3
Queue Wait Time [Cluster 0] = 20
# nodes for this wt. time = 1
Queue Wait Time [Cluster 1] = 10
# nodes for this wt. time = 2 T

18. Scheduling onto Batch-Queue Systems: Example
Cluster 0
Cluster 1
Input DAG R0 R1 R2 R3
Queue Wait Time [Cluster 0] = 20
# nodes for this wt. time = 1
Queue Wait Time [Cluster 1] = 10
# nodes for this wt. time = 2 T

19. Discussions Experiments to evaluate EMAN scheduling with batch-queues
Control experiment
Schedule with and without queue-wait estimates, run application with the two schedules on Teragrid and compare turnaround times
Accuracy of the results - how close to actual
Other future issues
Predictive/opportunistic approach
Submit to queues even before data arrives with hope that data arrives by the time job moves to the front of the queue
Point-valued predictions of probabilistic systems are problematic
Need to schedule based on ranges or distributions
Probabilistic deadline scheduling

20. Cluster Scheduling (Mandal)
Motivation: Scheduler scaling problem for ‘large’ Grids
Idea: Schedule directly onto clusters
Input:
Workflow DAG with restricted structure - nodes at the same level do the same computation
Set of available Clusters (numNodes, arch, CPU speed etc.) and inter-cluster network connectivity (latency, bandwidth)
Per-node performance models for each cluster
Output:
Mapping: for each level the number of instances mapped to each cluster
Objective:
Minimize makespan

21. Scheduling onto Clusters: Modeling
Abstract modeling of mapping problem for a DAG level
Given:
N instances
M clusters
r1..rM nodes/cluster
t1..tM - rank value per node per cluster (incorporates both computation and communication)
Aim:
To find a partition (n1, n2,… nM) of N such that overall time is minimized with n1+n2+..nM = N
Analytical solution:
No ‘obvious’ solution because of discrete nature of problem

22. Scheduling onto Clusters
Iterative solution
Big picture:
Iterative assignment of tasks to clusters
DP approach
For each instance, i from 1 to N
For each cluster, j from 1 to M
Tentatively map i onto j
Record makespan for each j by taking care of round(j)
End For each
Find cluster, p with minimum makespan increase
Map i to p
Update round(p), numMapped(p)
O(#instances * #clusters)

23. Scheduling onto Clusters: Evaluation
Application
Representative DAGs from Montage and EMAN with varying widths
Known performance models
Simulation Platform
Resource Model: Synthetic cluster generator (Kee et al SC’04)
Network Model: BRITE to generate network topology, generate latency/bandwidth following a truncated normal distribution
Experiment
Varying number of clusters (nodes)
250 to 1000 clusters (8.5K to 36K nodes)
Ran three scheduling approaches
Heuristic (min-min/max-min/sufferage heuristics based)
Greedy (simple greedy heuristic based)
Simple (the Cluster level scheduler)
Compared turnaround time (Makespan + Scheduling Time)

24. Scheduling onto Clusters: Results
Montage Application
103 node Montage DAG
717 node Montage DAG
Cluster level Scheduler (Simple) offers
Scalability - scales to ‘large’ Grids
Improved turnaround time
No significant degradation of application makespan quality

25. Scheduling onto Clusters: Results
EMAN Application
171 node EMAN DAG
666 node EMAN DAG
Cluster level Scheduler (Simple) offers
Scalability - scales to ‘large’ Grids
Improved turnaround time
No significant degradation of application makespan quality

26. Robust Task Scheduling (Shi)
Task scheduling: Assigning tasks of a meta-task (workflow-type application) to a set of resources, and achieving certain goals, e.g. minimizing the schedule length
NP-complete, finding an optimal solution is either impossible or impractical.
Heuristics (list scheduling, duplication, clustering) , optimization (Genetic algorithm, Simulated annealing etc.)
Previously we focused on list scheduling algorithm considering the case that processors has different capabilities.

27. Non-deterministic environment
Actual resource environment is non-deterministic inherently due to resource sharing.
Previously we used expected values of execution times of tasks, network speed.
The optimal solution for task scheduling problem with expected values of resource characteristics is NOT optimal for the corresponding problem with non-deterministic values.
We focus on variable execution time in this work.

28. Possible Solutions Static scheduling Dynamic scheduling
Overestimate the execution time to avoid exceeding the allotted use of machine at the expense of machine utilization
Compute the schedules for various scenarios and at run time adopt the one which fits the current status.
Find schedules more robust to variable execution time.
Dynamic scheduling
at each point of scheduling (when a task is ready to be executed), gather current resource information and compute a new schedule for unscheduled tasks

29. Robustness Schedule delay Robustness
M0(s): Makespan of the schedule s obtained with expect values (execution time)
M(s): Makespan of schedule s with real execution time
Robustness
Each realization of expect values gives different schedule delay.

30. Slack slack(ni) = makespan – [b_level(ni)+t_level(ni)]
Slack of a task node is defined as follow:
Slack is closely related to robustness.
large slack means a task node can tolerate large increase of execution time without increasing the makespan
slack(ni) = makespan – [b_level(ni)+t_level(ni)]

31. slack(ni) = makespan – [b_level(ni)+t_level(ni)]
Robustness and slack
Disjunctive graph P1 1 3 8 P2 2 5 7 P3 4 6 9 10
Disjunctive graph is used to calculate expected makespan and real makespan
slack(ni) = makespan – [b_level(ni)+t_level(ni)]

32. Task execution time modeling
Least Time to Compute (LTC) matrix : {ltcij}
time to compute task i on processor j
generated from a single number with twice gamma distribution on 2 dimensions (machine, task)
different values of gamma parameters represent different heterogeneities of machine or task
Uncertainty level: {ulij}
expected actual time to compute / least time to compute
Actual computation time: actij = ltcij * ulij

33. Genetic Algorithm
1.[Start] Generate initial population of n chromosomes (suitable solutions for the problem)
2.[Fitness] Evaluate the fitness f(x) of each chromosome x in the population
3.[New population] Create a new population by repeating following steps until the new population is complete
[Selection] Select two parent chromosomes from a population according to their fitness
[Crossover] With a crossover probability cross over the parents to form new offspring (children). If no crossover was performed, offspring is the exact copy of parents.
[Mutation] With a mutation probability mutate new offspring at each locus (position in chromosome).
[Accepting] Place new offspring in the new population
4.[Replace] Use new generated population for a further run of the algorithm
5.[Test] If the end condition is satisfied, stop, and return the best solution in current population
6.[Loop] Go to step 2

34. Single objective optimization
makespan optimization
robustness optimization

35. Multi-objective optimization
Goal: Minimize makespan and maximize robustness at the same time.
Conflict - there cannot be a single optimum solution which simultaneously optimizes both objectives.
Solution – seek balance between the 2 objectives.

36. Multi-objective optimization
Classical methods
weighted sum
-constraint
Weigthed sum
scalarizes multiple objectives into a single objective
optimize one of the objectives , subject to some constraints imposed on the other objectives

37. Weighted sum Objective function aws: average weighted slack:
ni is scheduled on pj

38. -constraint Objective: Solutions: Fitness:
maximize aws (average weighted slack)
subject to: ms < ε*ms0
Solutions:
feasible (ms < ε*ms0)
infeasible (ms ≥ ε*ms0)
Fitness:

39. Summary
Studied robust scheduling in non-deterministic environment using GA.
Provided a measurement of robustness.
Robust schedule could be generated through the optimization of average weighted slack (AWS) of a task graph.
Makespan and robustness are two conflicting objectives.
Multi-objective optimization methods are employed.
Weighted sum method is easy to use and intuitive. Setting up an appropriate weight vector depends on the scaling of each objective function. Normalization of objectives is usually required.
-constraint methods let user optimize one objective while imposing constraints on other objectives.