Download presentation
Presentation is loading. Please wait.
1
Distributed Low-Latency Scheduling
Sparrow Distributed Low-Latency Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica
2
Sparrow schedules tasks in clusters
using a decentralized, randomized approach
3
Sparrow schedules tasks in clusters
using a decentralized, randomized approach support constraints and fair sharing and provides response times within 12% of ideal
4
… … … … Scheduling Setting Map Reduce/Spark/Dryad Job Task Task Task
5
Job Latencies Rapidly Decreasing
2012: Impala query 2010: Dremel Query 2010: In-memory Spark query 2009: Hive query 2013: Spark streaming 2004: MapReduce batch job 10 min. 10 sec. 100 ms 1 ms Job Latencies Rapidly Decreasing
6
Scheduling challenges:
Millisecond Latency Quality Placement Fault Tolerant High Throughput
7
26 1.6K 160K 16M 1000 16-core machines Scheduler throughput
2012: Impala query 2010: Dremel Query 2010: In-memory Spark query 2009: Hive query 2013: Spark streaming 2004: MapReduce batch job 10 min. 10 sec. 100 ms 1 ms 26 decisions/second 1.6K decisions/second 160K decisions/second 16M decisions/second Scheduler throughput core machines
8
? Millisecond Latency Quality Placement Fault Tolerant High Throughput
Today: Completely Centralized Sparrow: Completely Decentralized Less centralization Millisecond Latency Quality Placement Fault Tolerant High Throughput ✗ ✓ ? ✓ ✓ ✗ ✓ ✗ ✓
9
Sparrow Decentralized approach Existing randomized approaches Batch Sampling Late Binding Analytical performance evaluation Handling constraints Fairness and policy enforcement Within 12% of ideal on 100 machines
10
Scheduling with Sparrow
Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker
11
Random Job Worker Scheduler Worker Scheduler Worker Worker Scheduler
12
Simulated Results Omniscient: infinitely fast centralized scheduler
100-task jobs in 10,000-node cluster, exp. task durations
13
Per-task sampling Power of Two Choices Job Worker Scheduler Worker
14
Per-task sampling Power of Two Choices Job Worker Scheduler Worker
15
100-task jobs in 10,000-node cluster, exp. task durations
Simulated Results 100-task jobs in 10,000-node cluster, exp. task durations
16
Response Time Grows with Tasks/Job!
70% cluster load
17
Per-Task Sampling Job Per-task Worker Scheduler Task 1 Worker
✓ Scheduler Task 1 Worker Job Scheduler Worker Task 2 Worker Scheduler Worker Scheduler Worker ✓
18
Place m tasks on the least loaded of dm slaves
Per-task Sampling Batch Per-task 4 probes (d = 2) Worker ✓ Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker ✓ Place m tasks on the least loaded of dm slaves
19
Per-task versus Batch Sampling
70% cluster load
20
100-task jobs in 10,000-node cluster, exp. task durations
Simulated Results 100-task jobs in 10,000-node cluster, exp. task durations
21
Queue length poor predictor of wait time
80 ms 155 ms Worker Worker 530 ms Poor performance on heterogeneous workloads
22
Place m tasks on the least loaded of dm slaves
Late Binding 4 probes (d = 2) Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dm slaves
23
Place m tasks on the least loaded of dm slaves
Late Binding 4 probes (d = 2) Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dm slaves
24
Place m tasks on the least loaded of dm slaves
Late Binding Worke r reques ts task Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dm slaves
25
100-task jobs in 10,000-node cluster, exp. task durations
Simulated Results 100-task jobs in 10,000-node cluster, exp. task durations
26
What about constraints?
27
Restrict probed machines to those that satisfy the constraint
Job Constraints Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker Restrict probed machines to those that satisfy the constraint
28
Probe separately for each task
Per-Task Constraints Worker Scheduler Worker Job Scheduler Worker Worker Scheduler Worker Scheduler Worker Probe separately for each task
29
Technique Recap Batch sampling + Late binding Constraint handling
Worker Scheduler Batch sampling + Late binding Constraint handling Worker Scheduler Worker Worker Scheduler Worker Scheduler Worker
30
How does Sparrow perform on a real cluster?
31
Spark on Sparrow Sparrow Scheduler Job Worker Worker Worker
Query: DAG of Stages Worker Job Worker Worker Worker
32
Spark on Sparrow Sparrow Scheduler Job Worker Worker Worker
Query: DAG of Stages Worker Worker Worker Job Worker
33
Spark on Sparrow Sparrow Scheduler Job Worker Worker Worker
Query: DAG of Stages Worker Worker Worker Job Worker
34
How does Sparrow compare to Spark’s native scheduler?
core EC2 nodes, 10 tasks/job, 10 schedulers, 80% load
35
TPC-H Queries: Background
TPC-H: Common benchmark for analytics workloads Shark: SQL execution engine Spark: Distributed in-memory analytics framework Sparrow
36
100 16-core EC2 nodes, 10 schedulers, 80% load
TPC-H Queries Percentiles 95 75 50 25 5 core EC2 nodes, 10 schedulers, 80% load
37
TPC-H Queries Within 12% of ideal Median queuing delay of 9ms
core EC2 nodes, 10 schedulers, 80% load
38
Timeout: 100ms Failover: 5ms Re-launch queries: 15ms
Fault Tolerance ✗ Spark Client 1 Scheduler 1 Spark Client 2 Scheduler 2 Timeout: 100ms Failover: 5ms Re-launch queries: 15ms
39
When does Sparrow not work as well?
High cluster load
40
Related Work Centralized task schedulers: e.g., Quincy Two level schedulers: e.g., YARN, Mesos Coarse-grained cluster schedulers: e.g., Omega Load balancing: single task
41
Worker Scheduler Batch sampling + Late binding Constraint handling Worker Scheduler Worker Worker Scheduler Worker Scheduler Worker Sparrows provides near-ideal job response times without global visibility
42
Backup Slides
43
Priorities Fair Shares
Policy Enforcement Priorities Serve queues based on strict priorities Fair Shares Serve queues using weighted fair queuing Slave Slave High Priority User A (75%) Low Priority User B (25%) Can we do better without losing simplicity?
Similar presentations
