1. Scheduling

2. Main Points
Scheduling policy: what to do next, when there are multiple threads ready to run
Or multiple packets to send, or web requests to serve, or …
Definitions
response time, throughput, predictability
Uniprocessor policies
FIFO, round robin, optimal
multilevel feedback as approximation of optimal
Multiprocessor policies
Affinity scheduling, gang scheduling
Queueing theory
Can you predict/improve a system’s response time?

3. Example
You manage a web site, that suddenly becomes wildly popular. Do you?
Buy more hardware?
Implement a different scheduling policy?
Turn away some users? Which ones?
How much worse will performance get if the web site becomes even more popular?

4. Definitions Task/Job Latency/response time Throughput Overhead
User request: e.g., mouse click, web request, shell command, …
Latency/response time
How long does a task take to complete?
Throughput
How many tasks can be done per unit of time?
Overhead
How much extra work is done by the scheduler?
Fairness
How equal is the performance received by different users?
Predictability
How consistent is the performance over time?
1. Minimize response time: elapsed time to do an operation (or job)
Response time is what the user sees: elapsed time to
echo a keystroke in editor
compile a program
run a large scientific problem
2. Maximize throughput: operations (or jobs) per second
Throughput is related to response time, but they're not identical -- for example, I’ll show that minimizing response time will lead you to do more context switching than you would if you were only concerned with throughput.
Two parts to maximizing throughput
a. Minimize overhead (for example, context switching)
b. Efficient use of system resources (not only CPU, but disk, memory, etc.)
What does CPU scheduling have to do with efficient use of the disk? A lot! Have to have CPU to make a disk request.
3. Fair: share CPU among users in some equitable way
Tradeoff: will argue you can get better average response time by making system less fair.
What does fairness mean?
Minimize # of angry phone calls? Minimize my response time?
Minimize average response time? We will argue fairness is a tradeoff against average response time; can get better average response time by making system less fair. Sort of like capitalism.
Anecdote: Response time has a lot to do with perceived effectiveness.
IBM keystroke experiment -- consistency is better than speed.
Might believe that since have PC's, CPU scheduling is less important -- usually, only one thing running at a time, for a single user. But! Face similar problems in networks -- how do you allocate scarce resources among users? Do you optimize for response time, throughput, fairness? In networks, fairness is often suboptimal.

5. More Definitions Workload Preemptive scheduler Work-conserving
Set of tasks for system to perform
Preemptive scheduler
If we can take resources away from a running task
Work-conserving
Resource is used whenever there is a task to run
For non-preemptive schedulers, work-conserving is not always better
Scheduling algorithm
takes a workload as input
decides which tasks to do first
Performance metric (throughput, latency) as output
Only preemptive, work-conserving schedulers to be considered

6. First In First Out (FIFO)
Schedule tasks in the order they arrive
Continue running them until they complete or give up the processor
Example: memcached
Facebook cache of friend lists, …
On what workloads is FIFO particularly bad?
Always happens to me -- you go to the store to buy a carton of milk, you get stuck behind someone with a huge basket? And insists on paying in pennies. Feature -- gives you time to read the National Enquirer. Computer science professor has space alien's baby.

7. Shortest Job First (SJF)
Always do the task that has the shortest remaining amount of work to do
Often called Shortest Remaining Time First (SRTF)
Suppose we have five tasks arrive one right after each other, but the first one is much longer than the others
Which completes first in FIFO? Next?
Which completes first in SJF? Next?
Need to be careful: optimal wrt average response time.
Recall: only preemptive schedulers.

8. FIFO vs. SJF
Now you can see why SJF improves average response time – it runs short jobs first. Effect on the short jobs is huge; effect on the long job is small.

9. Question Claim: SJF is optimal for average response time
Why?
Does SJF have any downsides?
Time to practice your CSE Consider a hypothetical alternative policy that is not SJF, but that we think might be optimal. Because the alternative is not SJF, at some point it will choose to run a task that is longer than something else in the queue. If we now switch the order of tasks, keeping everything the same, but doing the shorter task first, we will reduce the average response time.
Downsides: starvation, and variance in response time. Some task might take forever!
Imagine a supermarket that used SJF – would it work?
What would you do if you went to the supermarket and they were using SJF? Clever person would go through with one item at a time…

10. Question
Is FIFO ever optimal?
Pessimal?
Optimal = SJF
Pessimal = LJF

11. Starvation and Sample Bias
Suppose you want to compare two scheduling algorithms
Create some infinite sequence of arriving tasks
Start measuring
Stop at some point
Compute average response time as the average for completed tasks between start and stop
Is this valid or invalid?
SJF and FIFO would complete a different set of tasks – so they average response time isn’t directly comparable.
For example, in previous graphic – suppose you stopped at any point.
Worse, it tells you what you wanted to know – so you’d stop and say see!
Solution?

12. Sample Bias Solutions
Measure for long enough that # of completed tasks >> # of uncompleted tasks
For both systems!
Start and stop system in idle periods
Idle period: no work to do
If algorithms are work-conserving, both will complete the same tasks

13. Round Robin
Each task gets resource for a fixed period of time (time quantum)
If task doesn’t complete, it goes back in line
Need to pick a time quantum
What if time quantum is too long?
Infinite?
What if time quantum is too short?
One instruction?
Can we combine best of both worlds? RR approximates SJF by moving long tasks to the end of the line.

14. Round Robin
Approximation depends on granularity

15. Round Robin vs. FIFO
Assuming zero-cost time slice, is Round Robin always better than FIFO?
Show of hands!
What’s the worst case for RR? Same sized jobs – then you are time-slicing for no purpose. Worse, this is nearly pessimal for average response time.

16. Round Robin vs. FIFO
Everything is fair, but average response time in this case is awful – everyone finishes very late! In fact, this case is exactly when FIFO is optimal, RR is poor.
On the other hand, if we’re running streaming video, RR is great – everything happens in turn. SJF maximizes variance. But RR minimizes it.

17. Round Robin = Fairness? Is Round Robin always fair? What is fair?
FIFO?
Equal share of the CPU?
What if some tasks don’t need their full share?
Minimize worst case divergence?
Time task would take if no one else was running
Time task takes under scheduling algorithm
Show of hands! After all, round robin ensures we don’t starve, and gives everyone a turn, but lets short tasks complete before long tasks.

18. Mixed Workload One task does I/O repetively
The other tasks consume the CPU.
I/O task has to wait its turn for the CPU, and the result is that it gets a tiny fraction of the performance it could get.
By contrast the compute bound job gets almost as much as it would if the I/O task wasn’t there.
We could shorten the RR quantum, and that would help, but it would increase overhead.
what would this do under SJF? Every time the task returns to the CPU, it would get scheduled immediately!

19. Max-Min Fairness How do we balance a mixture of repeating tasks:
Some I/O bound, need only a little CPU
Some compute bound, can use as much CPU as they are assigned
One approach: maximize the minimum allocation given to a task
If any task needs less than an equal share, schedule the smallest of these first
Split the remaining time using max-min
If all remaining tasks need at least equal share, split evenly
On previous slide, what would happen if we used max-min fairness? Then I/O task would be scheduled immediately – its always the one using less than its equal share.

20. Multi-level Feedback Queue (MFQ)
Goals:
Responsiveness
Low overhead
Starvation freedom
Some tasks are high/low priority
Fairness (among equal priority tasks)
Not perfect at any of them!
Used in Linux (and probably Windows, MacOS)

21. MFQ Set of Round Robin queues
Each queue has a separate priority
High priority queues have short time slices
Low priority queues have long time slices
Scheduler picks first thread in highest priority queue
Tasks start in highest priority queue
If time slice expires, task drops one level

22. MFQ
What strategy should you adopt if you knew you were running on an MFQ system?
Grad school: wrote an AI to play Othello. Both sides run on a uniprocessor – so if I could slow down the other task, I’d win! Periodically print a dot to the console screen (back when printing to the screen was I/O).

23. Uniprocessor Summary (1)
FIFO is simple and minimizes overhead.
If tasks are variable in size, then FIFO can have very poor average response time.
If tasks are equal in size, FIFO is optimal in terms of average response time.
Considering only the processor, SJF is optimal in terms of average response time.
SJF is pessimal in terms of variance in response time.

24. Uniprocessor Summary (2)
If tasks are variable in size, Round Robin approximates SJF.
If tasks are equal in size, Round Robin will have very poor average response time.
Tasks that intermix processor and I/O benefit from SJF and can do poorly under Round Robin.

25. Uniprocessor Summary (3)
Max-Min fairness can improve response time for I/O-bound tasks.
Round Robin and Max-Min fairness both avoid starvation.
By manipulating the assignment of tasks to priority queues, an MFQ scheduler can achieve a balance between responsiveness, low overhead, and fairness.

26. Multiprocessor Scheduling
What would happen if we used MFQ on a multiprocessor?
Contention for scheduler spinlock
Cache slowdown due to ready list data structure pinging from one CPU to another
Limited cache reuse: thread’s data from last time it ran is often still in its old cache
Two problems:
Single lock on MFQ
Cache effects

27. Per-Processor Affinity Scheduling
Each processor has its own ready list
Protected by a per-processor spinlock
Put threads back on the ready list where it had most recently run
Ex: when I/O completes, or on Condition->signal
Idle processors can steal work from other processors

28. Per-Processor Multi-level Feedback with Affinity Scheduling

29. Scheduling Parallel Programs
What happens if one thread gets time-sliced while other threads from the same program are still running?
Assuming program uses locks and condition variables, it will still be correct
What about performance?

30. Bulk Synchronous Parallelism
Loop at each processor:
Compute on local data (in parallel)
Barrier
Send (selected) data to other processors (in parallel)
Examples:
MapReduce
Fluid flow over a wing
Most parallel algorithms can be recast in BSP
Sacrificing a small constant factor in performance
A model for writing parallel programs.
Eg., PageRank

31. Tail Latency
Performance of BSP is the load balancing cost – computation takes the time of the slowest participant.
Called “tail latency” – laggard defines performance.

32. Scheduling Parallel Programs
Oblivious: each processor time-slices its ready list independently of the other processors
What happens to tail latency here? Pretty much the worst case – the last to get scheduled.

33. Gang Scheduling Schedule the same task at the same time everywhere.
Problems? What about single threaded processes? You would waste time on the other processors.

34. Parallel Program Speedup
Is it possible to be above the perfect parallelism line? That is – more than N times faster on N processors?
Impact of overhead – creating extra threads, synchronization, communication that’s not needed in the single processor case. Overhead shifts the curve down, but still can be linear.
But we might also have non-perfect parallelism – e.g., for non-equal BSP tasks
Or if you have a single lock that’s contended.
For these last two cases, you’d rather give the system fewer processors and avoid time-slicing.

35. Space Sharing
Scheduler activations: kernel tells each application its # of
processors with upcalls every time the assignment changes

36. Queueing Theory Can we predict what will happen to user performance:
If a service becomes more popular?
If we buy more hardware?
If we change the implementation to provide more features?

37. Queueing Model Assumption: average performance in a stable system,
where the arrival rate (ƛ) matches the departure rate (μ)

38. Definitions Queueing delay (W): wait time
Number of tasks queued (Q)
Service time (S): time to service the request
Response time (R) = queueing delay + service time
Utilization (U): fraction of time the server is busy
Service time * arrival rate (ƛ)
Throughput (X): rate of task completions
If no overload, throughput = arrival rate

39. Little’s Law N = X * R N: number of tasks in the system
Applies to any stable system – where arrivals match departures.

40. Question
Suppose a system has throughput (X) = 100 tasks/s, average response time (R) = 50 ms/task
How many tasks are in the system on average?
If the server takes 5 ms/task, what is its utilization?
What is the average wait time?
What is the average number of queued tasks?

41. Question From example: Why is W = 45 ms and not 4.5 * 5 = 22.5 ms?
X = 100 task/sec
R = 50 ms/task
S = 5 ms/task
W = 45 ms/task
Q = 4.5 tasks
Why is W = 45 ms and not 4.5 * 5 = 22.5 ms?
Hint: what if S = 10ms? S = 1ms?

42. Queueing What is the best case scenario for minimizing queueing delay?
Keeping arrival rate, service time constant
What is the worst case scenario?

43. Queueing: Best Case

44. Response Time: Best vs. Worst Case

45. Queueing: Average Case?
What is average?
Gaussian: Arrivals are spread out, around a mean value
Exponential: arrivals are memoryless
Heavy-tailed: arrivals are bursty
Can have randomness in both arrivals and service times
In gaussian, longer you wait, the more likely you are to be done
In exponential, memoryless
In bursty, longer you wait, longer you still need to wait

46. Exponential Distribution

47. Exponential Distribution
Permits closed form solution to state probabilities,
as function of arrival rate and service rate

48. Response Time vs. Utilization
R = S/(1-U)
Better if gaussian
Worse if heavy-tailed
Variance in R = S/(1-U)^2

49. Question Exponential arrivals: R = S/(1-U)
If system is 20% utilized, and load increases by 5%, how much does response time increase?
If system is 90% utilized, and load increases by 5%, how much does response time increase?

50. Variance in Response Time
Exponential arrivals
Variance in R = S/(1-U)^2
What if less bursty than exponential?
What if more bursty than exponential?

51. What if Multiple Resources?
Response time =
Sum over all i
Service time for resource i /
(1 – Utilization of resource i)
Implication
If you fix one bottleneck, the next highest utilized resource will limit performance

52. Overload Management
What if arrivals occur faster than service can handle them
If do nothing, response time will become infinite
Turn users away?
Which ones? Average response time is best if turn away users that have the highest service demand
Example: Highway congestion
Degrade service?
Compute result with fewer resources
Example: CNN static front page on 9/11

53. Highway Congestion (measured)

54. Why Do Metro Buses Cluster?
Suppose two Metro buses start 15 minutes apart
Why might they arrive at the same time?