1. Recursive dimensionality reduction
for the analysis of multiserver scheduling policies
Taka Osogami
Joint with:
Hi, I am Taka, and this is a joint work with Mor Harchol-Balter, Adam Wierman, Alan Scheller-Wolf from Tepper school.
I am going to present a new technique that allows us to analyze the performance of various multiserver scheduling policies.
Mor Harchol-Balter (CMU, CS)
Adam Wierman (CMU, CS)
Alan Scheller-Wolf (CMU, Tepper School)

2. multiserver scheduling problems
Here are some examples of what we mean by multiserver systems.
In the first example, we have a stream of job arrivals from outside, and we need to dispatch these jobs to one of the servers as soon as they arrive at the dispatcher.
A possible scheduling policy is to send short jobs to one server and send long jobs to the other server,
so that short jobs do not have to wait behind long jobs.
In the second example, we have a central queue.
The jobs arriving from outside once wait in the central queue.
When a server becomes available, it grabs a job from the central queue.
A possible scheduling policy is a two-class priority scheduling,
where low priority jobs receive service only when a server has no high priority jobs that it can work on.
In the last example, we have two types of jobs arriving from outside.
Different types of jobs may have different affinities with different servers.
For example, jobs in the first queue may be processed more efficiently on server 1 than on server 2.
A possible scheduling policy is a threshold-based policy,
where each server usually works on its own queue,
but when the number of jobs in queue 1 exceeds a threshold value,
both servers work on jobs in queue 1.
Our goal is to analyze the mean response time under various scheduling policies for various multiserver architectures.
Here the response time of a job is defined to be the time between its arrival and its completion.
We assume that the arrival process is a certain stochastic process and job size has a certain probability distribution.
So, our goal is to get an analytical expression for the mean response time for each scheduling policy as a function of arrival process and job size distributions.
This is a difficult problem, but we are going to provide a partial solution to this problem.
To understand the difficulty of the problem and how our solution works,
we are going to first step back and see how we can analyze the performance under single server systems.
Goal:
Mean response time
= f(scheduling policy, arrival processes,
job size distributions)

3. Analysis of single-server FCFS
Poisson arrival &
exponential job size
(M/M/1) m 1 2 l
Phase-type job size (M/PH/1) 1 2 l m1 m2 m3
MAP/PH/1 1 2
Markovian arrival process (MAP/M/1) 1 2 m l1 l2 l3 l4
In this slide, we are going to look at how we can analyze the performance of first come first serve scheduling policy in singe server systems.
We are going to start with the simplest case of Poisson arrivals and exponential job size distributions.
This is a Markovian system.
During delta interval, we have an arrival with probability lambda times delta, and we have a job completion with probability mu times delta.
This is a simple birth and death process, and pi i, which is the probability that there are i jobs in system, has a nice geometric form of pi 0 times rho to the i,
where rho is some constant.
When the job size distribution is not an exponential distribution, we can model the job size distribution by a phase type distribution.
In fact, any job size distribution can be approximated arbitrarily closely by a phase type distribution.
Technically, the phase type distribution is dense in the class of all the nonnegative probability distributions.
That is for any given nonnegative distribution, there is a sequence of phase type distributions that converges to the given distribution.
In this example, job size consists of two phases.
In the first phase, the job is completed with rate mu1 but with rate mu2 it just moves to the second phase.
In the second phase, the job is completed with rate mu3.
In a general phase type distribution, we can have an arbitrary number of phases and arbitrary transitions among the phases.
This time, pi i is a vector of probabilities that there are i jobs in system, but it has a nice matrix geometric form like this,
where R is some matrix.
When the arrival process is not Poisson, we can model the arrival process by a Markovian arrival process.
Again, any stationary arrival process can be approximated arbitrarily closely by a Makovian arrival process,
since the Markovian arrival process is dense in the class of all the stationary point processes.
In this example, the arrival process has two phases.
In the first phase, a job arrives with rate lambda 1, and in the second phase, a job arrives with rate lambda 2.
The phase is changed with rate lambda 3 and lambda 4.
In a general Markovian arrival process, we can have an arbitrary number of phases and arbitrary transitions among phases.
Again, probability vector, pi, has a nice matrix geometric form.
When the arrival process is a Markovian arrival process and the job size has a phase type distribution,
we can take the superposition of these two Markov chain, which looks like this.
So, we can analyze its performance very easily.
There is one point that you should remember from this slide.
One of the reasons that we can analyze these Markov chains efficiently is that all the Markov chains are infinite only in one dimension.
We may have multiple phases, but they are finite.

4. Common Problem: 2D-infinite Markov chain (or nD-infinite)
Goal:
Mean response time
= f(scheduling policy, MAP arrival processes,
PH job size distributions) H L
Now let’s go back to our problem of multiserver systems.
Our goal is to obtain the mean response time for each scheduling policy as a function of Markovian arrival process and phase type job size distributions.
There is a common problem that comes up in the analysis of many scheduling policies in many multiserver architectures.
The problem is that the Markov chain grows infinitely in 2 dimensions or even higher dimensions.
In this talk, I am going to illustrate this common problem and our solution through the analysis of just one example of multiserver scheduling policy.
We are going to consider a priority scheduling, where high priority jobs have preemptive priority over low priority jobs.
That is, if a high priority job arrive in system where all the servers are busy and some low priority jobs are getting service,
a low priority job is preempted and the high priority job can receive service immediately.
H has preemptive priority over L
Common Problem: 2D-infinite Markov chain
(or nD-infinite)

5. M/M/2 with 2 priority classes
2mL
0,0
1,0
2,0
0,1
1,1
2,1
0,2
1,2
2,2 lL lH mH
2mH mL #L #H
3,0
3,1
3,2 H L
We are going to consider the simplest case of Poisson arrivals and exponential job sizes.
We can draw a Markov chain but it is infinite in 2 dimensions.
Here, we keep track of the number of low priority jobs and the number of high priority jobs.
There are transitions due to low priority jobs arrival and completion, and they are shown in blue.
There are also transitions due to high priority jobs arrival and completion, and they are shown in red.
In fact, high priority jobs can be analyzed independently of low priority jobs,
since the behavior of high priority jobs does not depend on low priority jobs.
However, the behavior of low priority jobs does depend on the behavior of high priority jobs,
so we need to analyze them at the same time.
In particular, the completion rate of low priority jobs depends on the number of high priority jobs.

6. Prior work: 2D infinite Markov chain
transform
methods
(’80s-’90s)
Boundary
value problems
(’70s-’00s)
matrix analytic
methods
(’80s-’00s)
Gail
Hunter
Taylor
Mitrani
King
Kao
Wilson :
Cohen
Boxma
King
Mitrani
Fayolle
Iasnogorodski
Borst
Jelenkovic
Uitert
Nain :
truncation (1D)
no truncation
Rao
Posner
Kao
Narayanan
Leemans
Miller
Ngo
Lee : 2D
2D→1D
nD→1D
A few approaches have been proposed to deal with 2D infinite state space.
Transform methods and reduction to boundary value problems lead to mathematically elegant expressions,
but we often experience numerical instability in evaluating these expressions.
Matrix analytic method is an algorithmic approach for solving a broad class of Markov chain with certain structures.
However, because matrix analytic method is most efficient when it is applied to a 1D infinite Markov chain,
most approaches that use matrix analytic methods first truncate the state space so that the state space becomes only one-dimensionally infinite.
In our model, this corresponds to placing an upper bound on the number of high priority jobs or the number of low priority jobs.
Although theory exists as to how matrix analytic method can be directly applied to 2D infinite Markov chain,
it is much less efficient for 2D infinite Markov chains.
Our approach reduces a n-dimensionally infinite state space to a 1D infinite state space without truncation
and solve the resulting 1D infinite Markov chain via the matrix analytic method efficiently.
Latouche
Ramaswami
Bright
Taylor
Sleptchenko
Squilante :
Our
approach

7. 2D infinite Markov chain#L L #H lL lL lL
0,0
0,1
0,2 L mL
2mL
2mL mH lH mH lH mH lH H lL lL lL
1,0
1,1
1,2 L L mL mL mL
2mH lH
2mH lH
2mH lH H
2,0 lL
2,1 lL
2,2 lL L L
This is the 2D infinite Markov chain that we want to solve.
The Markov chain can be divided into 3 regions, depending on the number of high priority jobs.
To know the behavior of low priority jobs, we just need to know which of the three regions we are in.
A problem is the third region which is infinite in two dimensions.
The point is that we don’t care about the exact number of high priority jobs in the third region.
All we want is the distribution of the sojourn time in the third region.
So, we are going to replace this region by something that is infinite only in one dimension.
That is, we first analyze the distribution of the sojourn time in this region,
we then approximate the sojourn time distribution by a phase type distribution,
and we put the phase type distribution in this region. H
2mH lH
2mH lH
2mH lH H L
3,0 lL
3,1 lL
3,2 lL

8. 2D 1D #L L #H lL lL lL
0,0
0,1
0,2 L L mL
2mL
2mL mH lH mH lH mH lH H lL lL lL
1,0
1,1
1,2 L L L mL mL mL b3 lH b3 lH b3 lH
2+,0 lL
2+,1 lL
2+,2 lL H L
So, the resulting Markov chain looks like this.
Now, in the third region, we don’t know the exact number of high priority jobs
All we know is that the number is at least two.
However, the distribution of time that we are in the third region is the same as, or similar to, the distribution of time in the third region of the 2D infinite Markov chain.
As you can see, our approach does not involve truncation,
but our Markov chain is infinite only in one dimension, which we can analyze very efficiently. b2 b1 b2 b1 b2 b1
2+,0 lL
2+,1 lL
2+,2 lL

9. M/M/2 with 3 priority classes
We can extend our analysis to more than two priority classes.
Now we are going to consider the case of three priority classes, high, medium and low.
As you can imagine now the Markov chain grows infinitely in three dimensions,
but we can reduce its dimension recursively until we get 1D infinite Markov chain.
In this dimension, we track the number of high priority jobs.
We can make this dimension finite, in exactly the same way as we did in the two-class case.
Then, we get a Markov chain that is infinite only in two dimensions.
In one dimension we track the number of low priority jobs,
and in the other dimension we track the number of medium priority jobs.
We can then make the medium jobs dimension finite.
The resulting 1D infinite Markov chain looks like this.
Now chain grows infinitely in 3 dimensions!

10. lL Transitions within a level: #L = 2M lH mH lM mM
Transitions within a level: #L = 3 lL
Transitions within a level: #L = 2
This picture shows the transition diagram within a single level, where the number of low priority jobs is three.
In this region, we have no high and medium priority jobs.
In this region, we have one high priority job or we have one medium priority job.
In these three regions, we have at least two high or medium priority jobs.
These three regions are differentiated only by how you enter these regions.
If you get a high priority job arrival when you have on high priority jobs in system, you come to this region.
If you get a medium priority job arrival, you come to this region.
We need to differentiate these regions, because the distributions of the sojourn time are different in these two cases.
These transitions from the bottom regions to the middle regions need to be replaced by phase type distributions.
This is the transition within a level, where there are three low priority jobs.
If a low priority job is completed, we are going to transition to a level next to it.
Here, we have two low priority jobs, and the transition within this level is exactly the same as the transitions when we have three low priority jobs.
We get a low priority job arrival with rate lambda L, and we transition back to this level.

11. Accuracy of recursive DR
Mean delay
Relative error (%) r
Our analysis involves approximation of sojourn time distributions by phase type distributions,
but this approximation can be made as accurate as desired.
In our work, we match the first three moments of the distribution, that is we match mean, variance, and one higher moment, and this gives a pretty good accuracy in our models.
This plot shows the mean delay calculated by our analysis and by simulation in the case of four priority classes with Poisson arrivals and exponential job size distributions.
Simulation values are shown in x’s.
It looks like our analysis perfectly matches simulation.
By taking a closer look, our analysis is within a few percent of what simulation predicts. r
Simulation
Class 1
Class 2
Class 3
Class 4
Analysis via RDR
Low
prio
High
prio

12. Impact of our new analysis
Evaluation of existing approximation
New approximation
Design heuristics for multiserver priority systems
n slow servers sometimes better than 1 fast servers
variability of H jobs
– big impact on L jobs
relative priority among higher priority jobs
– little impact on L jobs
What can we do with our analysis?
RDR provides the first near exact analysis of more than two priority classes, or nonexponential job sizes with two priority classes, in multiserver systems.
Only coarse approximations had existed.
We have evaluated the existing approximations extensively,
and characterized exactly when these approximations works or does not.
The study of these existing approximations led us to propose a new approximation,
which has better accuracy than existing approximations,
and which is simper and more efficient than our analysis via RDR.
We have also used our analysis to provide design heuristics that can be used to design multiserver systems with priority.
For example, there are some situations where n slow servers perform better than one server that is n times faster.
Since n slow servers are cheaper than 1 fast server, this is a good news.
We also observed that the variability of high priority jobs can have significant impact on the performance of low priority jobs.
So, if we could, we should design high priority jobs so that its service time has less variability.
Finally, we observed that relative priority among high priority jobs are usually not very important.
So, when you determine the priority among high priority jobs,
we do not need to care about the performance of lower priority jobs,
since it does not affect their performance so much.

13. nD infinite Markov chain
Summary H L
Multiserver
scheduling
Common
problem
nD infinite Markov chain
In summary, we are interested in analyzing the performance of various scheduling policies in various multiserver architectures.
However, there is a common problem that comes up in the analysis of multiserver scheduling policies.
We proposed a partial solution that works for many of these problems.
Our solution reduces an n dimensionally infinite Markov chain into a 1 dimensionally infinite Markov chain, which we can analyze efficiently.
Recursive
dimensionality
reduction
nD 1D

14. Recursive dimensionality reduction
Thank you!
Recursive dimensionality reduction
for the analysis of multiserver scheduling policies
Taka Osogami