Fundamental Characteristics of Queues with Fluctuating Load VARUN GUPTA Joint with: Mor Harchol-Balter Carnegie Mellon Univ. Alan Scheller-Wolf Carnegie Mellon Univ. Uri Yechiali Tel Aviv Univ.

2 Motivation Clients Server Farm Requests

3 Motivation Clients Server Farm Requests

4 Motivation Clients Server Farm Requests

5 Motivation Clients Server Farm Requests

6 Motivation Clients Server Farm Requests

7 Motivation Clients Server Farm Requests

8 Motivation Clients Server Farm Requests

9 Motivation Clients Server Farm Requests Real World  Fluctuating arrival and service intensities

10 A Simple Model HH LL exp(  H ) exp(  L ) High Load Low Load

11 Poisson Arrivals Exponential Job Size Distribution H /  H > L /  L H >  H possible, only need stability A Simple Model High Load Low Load  H,  H L,  L exp(  ) H HH L LL

12 The Markov Chain Phase Number of jobs L H H HH  LL L  H HH LL L...  Solving the Markov chain provides no behavioral insight

13  H HH L LL N = Number of jobs in the fluctuating load system Lets try approximating N using (simpler) non- fluctuating systems

14  H HH L LL Method 1 N mix

15 H HH L LL Q: Is N mix ≈ N? A: Only when   0 Method 1 N mix ½½ ½½ +   ,

16  H HH L LL Method 2

17 avg( H, L ) avg(  H,  L ) Method 2 ≡ N avg Q: Is N avg ≈ N? A: When      ,

18 Example    H =1, H =0.99  L =1, L =0.01 E[N mix ] ≈ 49.5 E[N avg ] = 1 00 

19 Observations Fluctuating system can be worse than non- fluctuating   0 and    asymptotes can be very far apart E[N mix ] > E[N avg ] E[N mix ]  E[N avg ]

20 Questions Is fluctuation always bad? Is E[N] monotonic in  ? Is there a simple closed form approximation for E[N] for intermediate  ’s? How do queue lengths during High Load and Low Load phase compare? How do they compare with N avg ? More than 40 years of research has not addressed such fundamental questions!

21 Outline  Is E[N mix ] ≥ E[N avg ], always?  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

22 Prior Work Fluid/Diffusion Approximations Transforms Matrix Analytic & Spectral Analysis - P. Harrison - Adan and Kulkarni Numerical Approaches Involves solution of cubic - Clarke - Neuts - Yechiali and Naor Involves solution of cubic - Massey - Newell - Abate, Choudhary, Whitt Limiting Behavior But cubic equations have a close form solution… ?

23 Good luck understanding this!

24 Asymptotics for E[N] ( H <  H ) E[N avg ] E[N mix ] E[N]  (switching rate) High fluctuation    H =1, H =0.99  L =1, L =0.01 E[N mix ] > E[N avg ]  Low fluctuation

25 Asymptotics for E[N] ( H <  H )  E[N] E[N mix ] E[N avg ] Agrees with our example (  H =  L ) Ross’s conjecture for systems with constant service rate: “Fluctuation increases mean delay” Q: Is this behavior possible? A: Yes  E[N] E[N avg ] E[N mix ]

26 Our Results  E[N] (  H - H ) > (  L - L ) (  H - H ) = (  L - L ) (  H - H ) < (  L - L ) Define the slacks during L and H as s L =  L - L s H =  H - H  E[N] 

27 Our Results Define the slacks during L and H as s L =  L - L s H =  H - H Not load but slacks determine the response times! s H > s L s H = s L s H < s L KEY IDEA  E[N]  

28 Outline  Is E[N mix ] ≥ E[N avg ], always?  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

29 Outline Is E[N mix ] ≥ E[N avg ], always? No  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

30 Notation N H : Number of jobs in system during H phase N L : Number of jobs in system during L phase N = (N H +N L )/2 H,  H L,  L exp(  ) NHNH NLNL

31 Analysis of E[N] First steps: –Note that it suffices to look at switching points –Express N L = f(N H ) N H = g(N L ) –The problem reduces to finding  Pr{N H =0} and Pr{N L =0} H,  H L,  L NHNH NLNL N L =f(g(N L )) f g

32 –Find the root  of a cubic (the characteristic matrix polynomial in the Spectral Expansion method) –Express E[N] in terms of  E[N] = The simple way forward… H,  H L,  L f g A  A - A  H (  L - L )  0 H +  L (  H - H )  0 L - (  L - L )(  H - H ) 2  (  A - A ) + Where  0 L =  0 H =  (  A - A )   L (  -1)(  H  - H )  (  A - A )   H (  -1)(  L  - L ) NHNH NLNL Difficult to even prove the monotonicity of E[N] wrt  using this!

33 Our approach (contd.) Express the first moment as E[N] = f 1 (  )r+f 0 (  )(1-r) –r is the root of a (different) cubic –r  1 as  0 and r  0 as  KEY IDEA

34 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r 

35 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  Need at least 3 roots for  when r=c 1 but  has at most 2 roots c1c1

36 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  Need at least 2 positive roots for  when r=c 2 but for r>1 product of roots is negative c2c2

37 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  E[N] is monotonic in  !

38 Outline Is E[N mix ] ≥ E[N avg ], always? No  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

39 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

40 Approximating E[N] Express the first moment as E[N] = f 1 (  )r+f 0 (  )(1-r) –r is the root of a (different) cubic –r  1 as  0 and r  0 as  Approximate r by the root of a quadratic KEY IDEA

41 Approximating E[N]  E[N] Exact Approx.  H =  L =1, H =0.95, L =0.2

42 Approximating E[N]  E[N] Exact Approx.  H =  L =1, H =0.95, L =0.2

43 Approximating E[N]  Exact Approx.  H =  L =1, H =1.2, L = E[N]

44 Approximating E[N]  Exact Approx.  H =  L =1, H =1.2, L = E[N]

45 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes  Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

46 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes Simple closed form approximation for E[N]  Application: Capacity Planning  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase Not covered in this talk Please read paper.

47 Scenario Application: Capacity Provisioning  H HH L LL  2 H HH 2 L LL Aim: To keep the mean response times same

48 Scenario Application: Capacity Provisioning  H HH L LL  2 H 2H2H 2 L 2L2L Question: What is the effect of doubling the arrival and service rates on the mean response time?

49 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B:

50 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B:

51 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B: Look at slacks! A: s H = s L B: s H > s L C: s H < s L D: s H < 0,   0  reduces by half  more than half  less than half  remains same

52 Our Contributions Give a simple characterization of the behavior of E[N] vs.  Provide simple (and tight) quadratic approximations for E[N] Prove the first stochastic ordering results for the fluctuating load model (see paper)

53 Bon Appetit!

54 Direction for future research Analysis of higher moments of response time Analysis of bursty arrival process General phase type distributions for phase lengths Analysis of alternating traffic streams – look at the workload process instead of number of jobs in system Conjecture: N H increases stochastically as switching rates decrease

55 Comparison of N L vs. N H Jackpot! Honey, I think we chose the wrong time to go out!

56 Stochastic Ordering refresher Random variable X stochastically dominates (is stochastically larger than) Y if: Pr{X  i}  Pr{Y  i} for all i. If X  st Y then E[f(X)]  E[f(Y)] for all increasing f –E[X k ]  E[Y k ] for all k  0.

57 Comparison of N L vs N H N L ≥ st N M/M/1/L N H ≤ st N M/M/1/H N H ≥ st N L N H ≥ st N avg N L  st N avg

58 Why do slacks matter? Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H H,  H L,  L exp(  ) when  ? H,  H L,  L exp(  ) when  0? HH  H +  L H H + L ?

59 Why do slacks matter? when  ? H,  H L,  L exp(  ) when  0? HH  H +  L H H + L ? Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H H,  H L,  L exp(  )

60 Why do slacks matter? when  ? H,  H L,  L exp(  ) when  0? AA HH ?  Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H As switching rates decrease, larger fraction of customers experience lower mean response times when s H >s L H,  H L,  L exp(  )

61 Q: What happens to E[N] when we double ’s and  ’s? A: System A:, ,  System B: 2, 2 ,  ?

62 Q: What happens to E[N] when we double ’s and  ’s? A: System A:, ,  System B: 2, 2 ,  System C: 2, 2 , 2  E[N] remains same in going from A to C A) s L = s H : remains same B) s L > s H : increases, but by less than twice C) s L < s H : decreases D)  0,  H >1 : queue lengths become twice as switching rates halve, E[N] doubles

63 Example    H =1.9, H =0.99  L =0.1, L =0.01 E[N mix ] ≈ 0.6 E[N avg ] = 1 00  