Join-the-Shortest-Queue (JSQ) Routing in Web Server Farms VARUN GUPTA Joint with: Mor Harchol-Balter Carnegie Mellon Univ. Karl Sigman Columbia Univ. Ward Whitt

Application: Web server farms Timeshare service among current requests Local Router (Immediate Dispatch) JSQ : most popular policy Cisco Local Director IBM Network Dispatcher … Commodity web servers

Model: PS server farm with JSQ Timeshare service among current requests Local Router (Immediate Dispatch) JSQ : most popular policy Cisco Local Director IBM Network Dispatcher … Commodity web servers

Model: PS server farm with JSQ Local Router PS (Immediate Dispatch) K homogenous, processor sharing servers

Model: PS server farm with JSQ Poisson Rate  JSQ / Immed. Dispatch PS K homogenous, processor sharing servers Poisson arrivals Job sizes i.i.d. ~ G ≡ M/G/K/JSQ/PS

Why join the shortest queue? Dynamic load balancing Simple Greedy for PS server farm share server with minimum # of jobs

Prior Analysis of JSQ routing FCFS Limited to FCFS servers and mostly exponential job size distribution 2-server: [Kingman 61] , [Flatto, McKean 77], [Cohen, Boxma 83], [Wessels, Adan, Zijm 91] [Foschini, Salz 78], [Knessl, Makkowsky, Schuss, Tier 87] [Conolly 84], [Rao, Posner 87], [Blanc 87], [Grassmann 80] >2-server approximations: [Nelson, Philips, Sigmetrics 89] [Lin, Raghavendra, TPDS 96] [Lui, Muntz, Towsley 95] OUR GOAL: Analyze JSQ with PS servers and general job size distributions; interested in mean response time, E[T]

GOAL: Analysis of JSQ with PS servers Observe: exponential job sizes How about general job sizes? M/M/K/JSQ/PS JSQ PS JSQ FCFS M/M/K/JSQ/FCFS joint queue length Approximations exist GOAL: Effect of job size variability on JSQ/PS

Goal: Effect of job size variability on JSQ/PS Idea: Look at H2*(,p) distribution 2 degrees of freedom  can fix mean and control variance THEOREM: E[T] insensitive under H2* jobs

( ) THEOREM: E[T] insensitive under H2* job size distribution PROOF: Q: What happens to 0-sized jobs? A: Disappear on arrival M/H2*/K/JSQ/PS JSQ PS  H2*(,p) M/M/K/JSQ/PS JSQ PS  Exp ( )  1-p equal mean size stationary queue length distribution stationary queue length distribution M/M/K/JSQ/PS JSQ PS (1-p) Exp()

Insensitivity for general distributions? Simulate M/G/K/JSQ/PS under following 7 distributions (all with mean 2) 1. Deterministic var=0 2. Erlang2 var=2 3. Exponential var=4 4. Bimodal(1,11) var=9 5. Weibull-1 var=20 6. Weibull-2 var=76 7. Bimodal(1,101) var=99 Heavy-tailed

Simulation results Number of servers = 2 E[T] < 2% deviation from Exp (95% conf intervals) Number of servers = 8 E[T] < 2% deviation from Exp Increasing variability

Goal: Effect of variability on JSQ/PS Conclusion: E[T] is “nearly insensitive” to variability of G

Why is JSQ/PS “near-insensitive”? Maybe just because M/G/1/PS is insensitive. Maybe all routing policies are near-insensitive. Which of the following do you think are insensitive? ??? PS RANDOM – randomly select one of K servers Round Robin – cyclic assignment Least Work Left – join the server with the smallest total remaining work

E[T] Number of servers = 2 ??? PS JSQ Det Exp Bim-1 Weib-1 Weib-2

E[T] Number of servers = 2 ??? PS RANDOM JSQ Det Exp Bim-1 Weib-1

E[T] Number of servers = 2 ??? PS RANDOM R-R JSQ Det Exp Bim-1 Weib-1

“Near-insensitivity” of JSQ is non-trivial (but cool) ! ??? PS Number of servers = 2 RANDOM R-R E[T] LWL JSQ Det Exp Bim-1 Weib-1 Weib-2 Bim-2 “Near-insensitivity” of JSQ is non-trivial (but cool) !

Recap ≈ = JSQ/PS “nearly insensitive” to variability M/G/K/JSQ/PS JSQ M/M/K/JSQ/PS JSQ PS JSQ FCFS M/M/K/JSQ/FCFS E[T] ≈ E[T] = Approximations exist THEOREM: equality for H2*

Outline ≈ = PART I: JSQ/PS “nearly insensitive” to variability M/G/K/JSQ/PS JSQ PS M/M/K/JSQ/PS JSQ PS JSQ FCFS M/M/K/JSQ/FCFS E[T] ≈ E[T] = Approximations exist THEOREM: equality for H2* PART II: Investigate new approaches for M/M/K/JSQ PART III: Is JSQ the best routing policy for PS servers?

Single Queue Approximation (SQA) M/M/K/JSQ/PS Mn/M/1/PS ??/M/1/PS JSQ PS ≈ (n) PS (n)= # arrivals into queue 1 finding n jobs total time there are n jobs in queue 1 Captures the effect of other queues in the JSQ system Model queue 1 as an independent PS queue with state (queue length) dependent arrival rates

Single Queue Approximation (SQA) M/M/K/JSQ/PS Mn/M/1/PS JSQ PS ≈ (n) PS (n)= # arrivals into queue 1 finding n jobs total time there are n jobs in queue 1 Intuition test Q1: Which is true? a. (0) = /K b. (0) < /K c. (0) > /K Q1: Which is true? a. (0) = /K b. (0) < /K c. (0) > /K Q2: Which is true? a. (0) = (1) b. (0) < (1) c. (0) > (1) Q2: Which is true? a. (0) = (1) b. (0) < (1) c. (0) > (1) Q3: (n) as n→ a. 0 /K (/K)K None of the above Q3: (n) as n→ a. 0 /K (/K)K None of the above THEOREM: lim (n) = (/2)2 when K=2. n→

Single Queue Approximation (SQA) M/M/K/JSQ/PS Mn/M/1/PS JSQ PS ≈ ≡ (n) PS (n)= # arrivals into queue 1 finding n jobs total time there are n jobs in queue 1 n = Pr{n jobs in queue 1} xn = Pr{n jobs} THEOREM: n = xn Where is the approximation? Don’t know the exact (n)’s !

Single Queue Approximation (SQA) M/M/K/JSQ/PS Mn/M/1/PS JSQ PS ≈ (n) PS (n)= # arrivals into queue 1 finding n jobs total time there are n jobs in queue 1 Recall: (n) (/K)K n→ Approximations for (0), (1), …, (n) For n≥3, (n) ≈ (/K)K Obtain closed form functional approx for (0), (1), (2)

Results (SQA) E[T] Number of servers (K) per server load = 0.9 Simulation E[T] per server load = 0.9 Number of servers (K)

Results (SQA) E[T] Number of servers (K) Simulation SQA E[T] per server load = 0.9 Number of servers (K) < 2% error for E[T] for up to 64 servers

Outline ≈ = PART I: JSQ/PS “nearly insensitive” to variability M/G/K/JSQ/PS JSQ PS M/M/K/JSQ/PS JSQ PS JSQ FCFS M/M/K/JSQ/FCFS E[T] ≈ E[T] = Approximations exist THEOREM: equality for H2* PART II: Accurate approximation for M/M/K/JSQ PART III: Is JSQ the best routing policy for PS servers?

To JSQ or not to JSQ, that is the question.. ??? PS RANDOM R-R E[T] LWL JSQ Det Exp Bim-1 Weib-1 Weib-2 Bim-2 OPT-0 – minimize average response time given no more arrivals

To JSQ or not to JSQ, that is the question.. ??? PS RANDOM R-R E[T] LWL JSQ OPT-0 Det Exp Bim-1 Weib-1 Weib-2 Bim-2 OPT-0 – minimize average response time given no more arrivals

To JSQ or not to JSQ, that is the question.. ??? PS RANDOM CONJEC: Minimum E[T] over all distributions, routing policies R-R E[T] LWL JSQ OPT-0 Det Exp Bim-1 Weib-1 Weib-2 Bim-2 Compare here for optimality

To JSQ or not to JSQ, that is the question.. Conclusion: JSQ is near optimal, without knowing job sizes or distribution

Conclusions JSQ/PS exhibits near-insensitivity to job size variability M/G/K/JSQ/PS JSQ PS Conclusions JSQ/PS exhibits near-insensitivity to job size variability SQA method to analyze M/M/K/JSQ/PS JSQ is near-optimal for all job size distributions M/G/K/JSQ/PS ≈ M/M/K/JSQ/PS THM: H2* equivalence M/M/K/JSQ/PS = Mn/M/1/PS THM: Single queue equivalence THM: (n) convergence