October 18, 2005 Charm++ Workshop Faucets A Framework for Developing Cluster and Grid Scheduling Solutions Presented by Esteban Pauli Parallel Programming Lab, UIUC

October 18, 2005 Charm++ Workshop Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

October 18, 2005 Charm++ Workshop Motivation Clusters are becoming ubiquitous Clusters are becoming ubiquitous Workloads come in bursts, resulting in alternation between low and high utilization Workloads come in bursts, resulting in alternation between low and high utilization Need framework for sharing computing power Need framework for sharing computing power Traditional schedulers care about throughput, not deadlines, priorities, etc. Traditional schedulers care about throughput, not deadlines, priorities, etc.

October 18, 2005 Charm++ Workshop Goals Provide technical and economic framework for allowing organizations to share their resources (clusters) Provide technical and economic framework for allowing organizations to share their resources (clusters) Provide new cluster scheduler which facilitates the above Provide new cluster scheduler which facilitates the above Provide platform for implementing new scheduling strategies Provide platform for implementing new scheduling strategies

October 18, 2005 Charm++ Workshop System Overview Faucets consists of two main components: meta scheduler and cluster scheduler Faucets consists of two main components: meta scheduler and cluster scheduler Meta scheduler provides mechanism for discovering and sharing resourcesMeta scheduler provides mechanism for discovering and sharing resources Cluster scheduler makes scheduling decisions based on local and global workloadCluster scheduler makes scheduling decisions based on local and global workload Components interact to meet users’ needs Components interact to meet users’ needs

October 18, 2005 Charm++ Workshop Central Server Database Cluster Cluster Daemon Scheduler System Architecture User Cluster Cluster Daemon Scheduler User

October 18, 2005 Charm++ Workshop Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Job Monitor Job Submission Job Specs Bids Job Specs Job Id Cluster

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Users provide job requirements Users provide job requirements System requirements: architecture, number of processors, minimum memory, etc.System requirements: architecture, number of processors, minimum memory, etc. Software requirements: utilities, dynamic libraries, packages, etc.Software requirements: utilities, dynamic libraries, packages, etc. Contract requirements: deadline, reliability, maximum price, etc.Contract requirements: deadline, reliability, maximum price, etc. Use XML, easily expandableUse XML, easily expandable Clusters bid on job Clusters bid on job Winning bidder executes job Winning bidder executes job

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Bidding requires no user intervention Bidding requires no user intervention Clusters bid on jobs based on current conditions Clusters bid on jobs based on current conditions Local utilizationLocal utilization Account balancesAccount balances Depending on scheduling strategy, might not be able to accept all jobs Depending on scheduling strategy, might not be able to accept all jobs

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (10000) Bob (100) Cluster 2 Shared (- 200) John (0)Joan (0)

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (10000) Bob (100) Cluster 2 Shared (- 200) John (0)Joan (0) Bob runs job worth 1000 units on Cluster 2

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (- 200) John (0)Joan (0) Bob’s account drained, remaining 900 units come from shared pool

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (- 200) John (0)Joan (0) Cluster 2’s policy: 50% to shared, rest divided equally

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Cluster 2’s shared balance up 500, John & Joan get 250 each

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Global balance remains unchanged

October 18, 2005 Charm++ Workshop The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Have limits to negative balances to prevent freeloading

October 18, 2005 Charm++ Workshop Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

October 18, 2005 Charm++ Workshop Cluster Scheduler Traditional schedulers concerned only with throughput – try to have highest possible utilization Traditional schedulers concerned only with throughput – try to have highest possible utilization Faucets cluster scheduler provides different strategies to allow more efficient bidding Faucets cluster scheduler provides different strategies to allow more efficient bidding Leverage run-time systems Leverage run-time systems Flexible design allows for easy implementation of new strategies Flexible design allows for easy implementation of new strategies

October 18, 2005 Deadline-Driven Scheduling (Gantt Chart)   Schedule based on #processors, deadline, wall-time   As new jobs arrive, reschedule meeting all demands  Allows bidding based on deadline – can charge different amounts based on user’s flexibility  Can leverage Charm++ runtime system to shrink and expand jobs Job 1, 4 PE’s, 4 time slices Job 2, 2 PE’s, 6 time slices Job 3, 3 PE’s, 3 time slices New Job, 2 PE’s, 7 time slices P1 P2 P3 P Original schedule P1 P2 P3 P New schedule

October 18, 2005 Priority-Driven Scheduling Leverage Charm++ and other checkpoint/restart mechanisms Leverage Charm++ and other checkpoint/restart mechanisms Priority can be based on rank (military, institutional, etc), price paid, or other factors Priority can be based on rank (military, institutional, etc), price paid, or other factors P1 P2 P3 P4 Original Schedule Job 1, normal priority Job 2, normal priority Job 3, high priority Job 3 arrives Job 3 terminates P1 P2 P3 P4 New Schedule

October 18, 2005 Charm++ Workshop Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

October 18, 2005 Charm++ Workshop Conclusions Clusters becoming more common, Faucets provides economic and technical framework for sharing Clusters becoming more common, Faucets provides economic and technical framework for sharing Flexible cluster scheduler allows scheduling based on deadlines, priorities, etc. Flexible cluster scheduler allows scheduling based on deadlines, priorities, etc. Cluster scheduler leverages run-time systems to increase functionality Cluster scheduler leverages run-time systems to increase functionality

October 18, 2005 Charm++ Workshop Future Work NCSA Faculty Fellowship – on- demand access NCSA Faculty Fellowship – on- demand access How do we control anonymous access? How do we control anonymous access? Only allow pre-selected applicationsOnly allow pre-selected applications Virtual machines (Xen, VMWare, etc.)Virtual machines (Xen, VMWare, etc.) Leverage virtualization to allow processor sharing Leverage virtualization to allow processor sharing Re-architect Faucets to make more robust, easier to write strategies Re-architect Faucets to make more robust, easier to write strategies

October 18, 2005 Charm++ Workshop Questions?

October 18, 2005 Charm++ Workshop Thanks!