1. Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Introduction to Continuous-Time Markov Chains and Queueing Theory

2. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis DTMC 1. Transitions at discrete time steps n=0,1,2,… 2. Past doesn’t matter (Markov property) P(X n+1 |X n,X n-1,…X 1 ) = P(X n+1 |X n ) 3. P(X n+1 = j|X n = i) = p ij (stationarity) 2 0 p 00 p 01 1 2 p 10 p 21 p 12 … k k-1 p k,k-1 p k-1,k k+1 p k,k-1 p k-1,k … CTMC 1. Transitions between states can happen any time 2. Past still doesn’t matter 3. Transition still independent of time t

3. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A CTMC is continuous time stochastic process {X(t), t ≥ 0}, s.t. for all s,t ≥ 0, and any i,j,x(u) P{X(t+s) = j|X(s) = i,X(u) = x(u), 0≤ u≤ s} = P{X(t+s) = j|X(s) = i} (Markov property) = P{X(t) = j| X(0) = i} (stationarity)  τ i : time until CTMC leaves state i, if currently in i  Above def. implies: P{τ i > t+s | τ i > s} = P{τ i > t}  Q: What does this imply?  A: τ i is exponentially distributed! 3

4. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Step 1: When at state i, wait an exponentially distributed amount of time  denote rate out of: v i  Step 2: When the process leaves i, it will enter j with probability p ij 4 i exp(v i ) l k p il j p ik p ij

5. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  When at state i :  move to state j with rate λ ij (independently) Q: Time to leave state i?  first move to any state  first phone rings A: Exponential with rate ( v i = λ i1 + λ i2 +…+λ in = Σ j λ ij ) Q: Probability to go to state k, after leaving i ? A: 5 0 3 2 λ 03 1 λ 02 λ 01 exp(λ 01 ) exp(λ 02 ) exp(λ 03 ) View 2 => View 1

6. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 6 Poisson (λ) e.g. 3 jobs/sec service: exp(μ) e.g. 5 jobs/sec 0 λ 1 2 μ … λ μ λ μ  Let π j = lim t  ∞ P ij (t) : limit probability of being in state j Q: How do we determine π j ?  how do we “solve” the chain? A: Assume an equivalent DTMC  “half-rigorous” argument

7. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Divide time into very small intervals δ  Prob. of an arrival in the next interval δ : λδ + o(δ)  Prob. of a departure in the next interval δ : μδ + o(δ)  P{i  i+1} = λδ (1-μδ) = λδ + o(δ) (1 arrival + 0 departure)  P{i  i-1} = μδ (1- λδ) = μδ + o(δ) (0 arrival + 1 departure)  P{i  i} = 1- λδ -μδ +o(δ) (no event with almost prob 1)  we can ignore self-loops in the balance equations 7 0 λ 1 2 μ … λ μ λ μ 0 λδ+o(δ) 1 2 μδ+o(δ) … λδ+o(δ) μδ+o(δ) CTMC DTMC (time step δ  0)

8. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Methodology  Step 1) Write balance equations  Step 2) Take limit as δ  0  For state 0 rate out = rate in 8 0 λδ+o(δ) 1 2 μδ+o(δ) … λδ+o(δ) μδ+o(δ)  For state 1 rate out = rate in

9. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Lesson from previous example: can simply solve the balance equations, rate in = rate out  See book (MHB, CH.13) for generalization  Can also try local balance equations  Need to check for irreducibility (for π i to exist) -What about periodicity??  Back to our queue)  Define ρ = λ\μ (utilization) (we derived earlier) (normalize to get π 0 ) 9 Poisson (λ) service: exp(μ) Our first continuous time queueing result!!

10. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  The example so far was an M/M/1 system  “M” stands for memoryless --- both arrivals and departures  “1” for single server  π n : number of customers in system (in service or queue) Q: What is the probability that the server is empty? A: π 0 =1-ρ Q: What is the expected number of customers? A: 10 Poisson (λ) service: exp(μ)

11. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q1: What happens when ρ  1 ? (i.e. system very busy) A: (queue goes to infinity!) Q2: What happens to the Markov Chain when ρ ≥ 1 ? A: All states become transient  π i = 0 (for all i)  Null-recurrent for ρ = 1  But this means we cannot apply balance equations (no balance!)  Could we have guessed it from the MC?? (homework!) Q3: We know E[N]; What is the expected delay E[T]? A: Little’s Law will tell us that E[T] = E[N]/ λ 11

12. Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Introduction to Queueing Theory 12

13. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A queuing system is described by A/D/n/B/S  A: the arrival process  M: for Memoryless  Poisson arrivals  D: Deterministic  arrivals at fixed intervals  G: generic distribution  D: service process  M: Memoryless  duration of service is exponentially distributed  D: Deterministic  each service last the same fixed amount of t  G: generic distribution (e.g. uniform, pareto, gamma, etc.)  n: number of servers  B: size of queue/buffer  If queue is full  additional arrivals are rejected  No B in notation  infinite queue (e.g. M/M/1, M/G/1, G/G/4)  S: Scheduling policy  FCFS (default assumed), preemptive, random, etc. 13

14. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis For the system to be stable: λ < μ Q: What happens if λ > μ ? A: Queue size goes to infinity (and so does delay!) Q: Can you give the intuition? A: Consider an interval [0,t] (large t)  Number in system = (arrivals in [0,t]) – (departures in [0,t])  Number in system = λt – μt = (λ- μ)t  t -> ∞  queue size -> ∞ 14 Poisson (λ) service: exp(μ)

15. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: Which of the two systems has a higher throughput? A: The throughput is the same for both: X = λ = 1/6  Consider a busy period of the system  Def: from time an arrival finds system empty, until it empties again  Assume A arrivals during a busy period Q: How many departures during that busy period? A: Exactly A!  All arrived jobs should complete for the system to empty 15 Throughput X: jobs/min leaving the system Open (stable) system: Throughput X = Input Rate

16. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  E[N] : number of customers in the system (queue + server)  E[T]: time a customer spends in the system (queue+server) Little’s Law: E[N] = λ E[T]  M/M/1:  (some) Intuition behind Little’s Law  Customer X arrives at the end of a queue  on average stays E[T]  Q: how many customers have arrived while X was in the system?  A: λ E[T]  => on average a departure leaves E[N] = λ E[T] customers in the system 16 Poisson (λ) service: exp(μ) M/M/1:

17. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  DEF: A is the area inside boxes  A(t): num of arrivals by t  D(t): num of departures by t 1. Sum times horizontally 2. Sum times vertically: 17 area A N(t 1 )=2 t1t1 N(t 2 )=3 t2t2

18. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: Did we assume anything about the number of servers? A: No! Q: Did we assume that scheduling is first-come first-serve (FCFS)? A: No! In fact, in the previous example 3 finishes before 2! Little’s Law applies to ANY system and ANY scheduling  Many servers, many queues, queueing network, scheduling policy 18

19. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: What about E[T Q ]?  number of jobs in queue – not server(s) A: λE[T Q ] = E[N Q ] (similar proof) Q: What about ρ (number of customers in service)? A: λE[S] = ρ  ρ = λ/μ Q: What is ρ? A: System utilization -> fraction of time server(s) is busy (time average) A: Ergodicity  Prob{server is busy at a random time} (ensemble average) 19

20. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Consider a call center with k total lines  E.g. an actual call center, or switch with k circuits, or base station with k channels  If all servers/lines busy  call/job is rejected  (Key Question): What is the probability of “blocking”? 20

21. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 21 Q: How many states do we need? A: k states Q:What is the rate from i to i+1? A: λ Q:What is the rate from i to i-1? A: kμ

22. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: Is the chain time-reversible (to apply local balance)?  i.e. is rate from 1  2 = rate from 2  1? A: Yes, it is  so, we can apply local balance 22 Erlang B

23. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: Job size/duration is not exponential? A: Turns out that the state probabilities π i is as in M/M/k/k  The block probability is the same and given by Erlang B  This is an important insensitivity result  Result is insensitive to service distribution (only depends on mean μ)  Insensitivity results often arise when there is no queueing 23

24. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 24 Dispatch to first available server queue if all busy A very important model  for server farms  Amazon, Google, CNN, pretty much all major web sites  Cloud computing Exponential service with rate μ Poisson (λ)

25. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: How many states? A: Infinite states Q: What is the rate from i  i+1 A: λ Q: What is the rate from i  i-1 (i ≤ k) A: iμ Q: What is the rate from i  i-1 (i > k) A: kμ 25 …

26. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 26 …

27. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: What is the expected number of busy servers? A: λ /μ - Definition: ρ = λ /kμ (utilization of a single server) Q: What is the probability (P Q ) an arriving job must queue? A: P Q = P{arrival finds all servers busy} = P{arrival sees ≥ k jobs in system} (from PASTA property) 27 Erlang C:

28. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: For which system (M/M/k and M/M/k/k) is the probability that all k server are busy higher? A: For the M/M/k Q: Why? A: In M/M/k jobs can arrive when all k servers are busy  these do not disappear but queue up  more work for later  In M/M/k/k these jobs are simply rejected  Can confirm by comparing Erlang B and C (work the math as HW) 28

29. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis …or server organizations…or supermarket lines… Q: Which is better in terms of E[T], FDM or M/M/1 (statistical multiplexing) A: M/M/1  E[T FDM ] = k/(kμ-λ) but E[T M/M/1 ] = 1/(kμ-λ) 29

30. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis …or server organizations…or supermarket lines… Q: What about E[T M/M/1 ] and E[T M/M/k ]? A: 30 ρ ≈ 0 ρ ≈ 1