System Performance: Queuing

Queuing Theory Leonard Kleinrock - “We study the phenomena of standing, waiting, and serving, and we call this study Queuing Theory. Any system in which arrivals place demands upon a finite capacity resource may be termed a queuing system.”

Airport Security Waiting Line

Common Queuing Questions How long will it take to be served? What length of line will build up? How long a line is desirable/tolerable? How many servers do I need? What throughput can I expect?

Queuing Systems Examples Banks Grocery stores Highway traffic Emergency rooms Communications networks Computers Emergency call centers

Queuing Issues Relative Economics Capacity System Architecture

Queuing Systems Processor(s) Queue(s) Transactions Events Management Discipline Transactions Events Arrivals Departures

Queuing Systems

Queuing Algorithm Arrival Departure If the server is idle, the transaction will go into the processor and will depart after it is serviced If the server is busy, the arriving transaction goes into the queue Departure If the queue is empty, the processor goes idle If the queue is non-empty, a transaction is selected from the queue based on the queue discipline and the transaction will go into the processor and will depart after it is serviced

Queuing Experience If traffic is light If traffic is heavy Queues will be small We get served quicker If traffic is heavy Queues will grow It takes longer to get served

Queuing Theory Steady-state performance Mathematical model Systems Networks Mathematical model Mean Value Analysis

Arrivals Exponential inter-arrival times Poisson Distribution Probability of n arriving in the next t time intervals l = mean (expected) arrival rate Pn(t) = (lt)n/n! e-lt Example: Probability of 3 arriving in the next second if the mean arrival rate is 6 per second P3(1) = (6*1)3/3! e-6*1 P3(1) = .08928

Poisson Arrival Distribution

Example An emergency supply dispensing station has an expected arrival rate of 5 people per minute to receive survival kits. What is the probability of 10 arriving in the next minute? What is the probability of 5 or more arriving in the next minute?

Example Solution A. Pn(t) = (lt)n/n! e-lt P10(1) = (5*1)10/10! e-5*1 P10(1) = 9765625/3628800 * 0.00673797 P10(1) = .01813 B. Pn>=l(t) = 1 – Sn-1i=0 Pi(t) Pn>=5(t) = 1 – (P0(t) + P1(t) + P2(t) + P3(t) + P4(t)) Pn>=5(t) = 1 – (0.006738 + 0.033690 + 0.084225 + 0.140374 + 0.175468) Pn>=5(t) = . 5595

Example Observations Actual number of arrivals varies. We need to look at the likelihood of many arrivals occurring How fast must we process these arrivals to avoid long lines and delays?

Queuing Theory System description – Kendall Notation A/B/c/K/m/Z A - arrival distribution B - service time distribution c - number of servers K - system capacity (i.e., the maximum number of jobs waiting in the queue, normally taken to be infinite) m - source population (normally taken to be infinite) Z is the queuing discipline (normally taken to be FIFO)

Kendall Notation Symbols Meaning Usage GI General Independent arrival time A G General service time B M Markov (i.e. exponential) inter-arrival/service time A/B D Deterministic inter-arrival/service time FIFO First In/First Out queue discipline Z LIFO Last In/First Out queue discipline RSS Random Selection for Service queue discipline

Base Queuing Assumptions M/M/1 Exponential arrival and service distributions One server M/D/1 Exponential arrival distribution Deterministic (constant) service distribution M/G/1 General service distribution M/M/n n servers

Queue Management Disciplines FIFO – First-In-First-Out (FCFS) First-Come-First-Served LIFO – Last-In-First-Out RSS – Random Selection for Service Priority Other

Queuing Model Parameters Expected Processing Time a Traffic Intensity c Number of Processors  Mean Arrival Rate  Mean Processing Rate  Average Processor Utilization L Expected Number in the System Lq Expected Number in the Queue W Expected Wait Time in the System Wq Expected Wait Time in the Queue g Average Throughput

Queuing Model Formulas W = Wq + S Expected Wait Time in the System = Expected Wait Time in the Queue + Expected Processing Time L = Lq + a Expected Number in the System = Expected Number in the Queue + Traffic Intensity

Queuing Model Formulas Little's Formula L = W Expected Number in the System = Mean Arrival Rate * Expected Wait Time in the System Lq = Wq

Queuing Model Formulas a = l/  = a/c g = m * r

Queue State Transition Diagram

Queuing Model Development At Steady State pn-1 = pn Where pn is the probability of the system state being n Pn = l/m pn-1 Pn = an p0

Queuing Model Development From probability S pn = 1 S an p0 = p0 S an =1 This is a standard geometric series: S xn =1/(1-x)

Queuing Model Development p0 S an = p0 /(1 – a) = 1 p0 has a special interpretation: The probability that zero is in the system The system is idle P0 = (1 – r) Pn = r (1 - r)

Queuing Model Development L = E(pn ) = r /(1 - r ) s2 = r /(1 - r )2

M/M/1 Formulas Pn(t) = (lt)n/n! e-lt S = 1/ a = l/  = a/c g = m * r L = E(pn ) = r /(1 - r ) s2 = r /(1 - r )2 L = W Lq = Wq W = Wq + S L = Lq + a

Queuing Example A particular processor has information arriving from sensors over a communication line. The sensor information arrives with an exponential inter-arrival time that has a mean of 1000 sensor transactions per second. The processor processes these with an exponential service time with a mean of 1200 sensor transactions per second.

Example Calculations l 1000 m 1200 c 1 a 0.83333333 S 0.00083333 r L 5 Lq 4. 16667 W 0.005 Wq 0.00417 g

Queuing Example Suppose that the mean arrival rate increases or decreases. How does this affect the M/M/1 example queuing model?

Variation of Mean Arrival Rate 100 300 600 900 1100 m 1200 c 1 a 0.083333 0.25 0.5 0.75 0.916667 S 0.000833 r L 0.090909 0.333333 3 11 Lq 0.007576 2.25 10.0834 W 0.000909 0.001111 0.001667 0.003333 0.010000 Wq 0.000076 0.000278 0.002500 0.009167

Queuing Example Observations At low mean arrival rates, most of the time is spent in the processor. At high mean arrival rates, most of the time is spent in the queue. At higher utilizations, increases in the mean arrival rate result in disproportionate increases in mean system wait time.

Explosion of Expected Queue Size with Arrival Rate

Queuing Example L is the expected (mean) number in the system. What are the probabilities of certain numbers being in the system? Pn = r (1 - r )

Probability of n in the System Pn Cumulative Pn P0 0.1667 1 P1 0.1389 0.3056 2 P2 0.1157 0.4213 3 P3 0.0965 0.5177 4 P4 0.0804 0.5981 5 P5 0.0670 0.6651 6 P6 0.0558 0.7209 7 P7 0.0465 0.7674 8 P8 0.0388 0.8062 9 P9 0.0323 0.8385 10 P10 0.0269 0.8654 11 P11 0.0224 0.8878 12 P12 0.0187 0.9065 13 P13 0.0156 0.9221 14 P14 0.0130 0.9351 15 P15 0.0108 0.9459 16 P16 0.0090 0.9549 17 P17 0.0075 0.9624 18 P18 0.0063 0.9687 19 P19 0.0052 0.9739 20 P20 0.0043 0.9783

Probability of n in the System

Supply Distribution Example Revisited An emergency supply dispensing station has an expected arrival rate of 5 people per minute to receive survival kits. What length of queue and what time to service results if service is exponential and has a mean of 7 people per minute?

Example Calculations l 5 m 7 c 1 a 0.714 S .1429 r L 2.5 Lq W Wq g

M/D/1 Queuing Model Development Service is constant L = E(pn ) = r /2(1 - r ) s2 = 0

Example: Emergency Inoculations There are many examples where large numbers of people need to receive vaccinations, injections of antibiotics or other treatments. Medical personnel are trained to perform these actions methodically (e.g. wipe clean, inject, wipe, bandage, etc.). The result is that it requires the same amount of time to treat each patient. Suppose that a medical technician can treat 200 people per hour. If 150 people an hour arrive for treatment, what mean queue size will result and what will be the average service time?

Emergency Inoculations 150 m 200 c 1 a 0.75 S 0.005 r L 1.5 Lq 1.495 W 0.01 Wq 0.009967

Queuing Example Suppose that the mean arrival rate increases or decreases. How does this affect the M/D/1 example queuing model?

Variation of Mean Arrival Rate 75 100 150 175 190 m 200 c 1 a 0.375 0.5 0.75 0.875 0.95 S 0.005 r L 0.3 1.5 3.5 9.5 Lq 0.295 0.495 1.495 3.495 9.495 W 0.004 0.01 0.02 0.05 Wq 0.003933 0.00495 0.009967 0.019971 0.049974

Queuing Example Observations At low mean arrival rates, most of the time is spent in the processor. At high mean arrival rates, most of the time is spent in the queue. At higher utilizations, increases in the mean arrival rate result in disproportionate increases in mean system wait time.

Explosion of Expected Queue Size with Arrival Rate

Queuing Example L is the expected (mean) number in the system. What are the probabilities of certain numbers being in the system? Pn = r (1 - r )

Comparison of M/M/1 and M/D/1 Queuing Models

M/G/1 Queuing Model General Lq = ((ls)2 + r2)/(2 *(1-r)) For Exponential s = 1/m For Deterministic s = 0

Multiple Processor M/M/c Queues Probability of having no items in queue Probability of having n items in queue

M/M/c Queues Probability of being in a queue

M/M/c Queues M/M/c Average length of queue Average time waiting in queue λ = arrival rate μ = departure rate

M/M/c Queues M/M/c Average time spent in system λ = arrival rate μ = departure rate

M/M/c Example Problem l 200 m 250 c 2 r 0.8 P0 0.42857 Lq 0.15238 L 0.95238 Wq 0.00076 W 0.00476

Multiple Processor Arrival Rate 200 250 300 350 400 450 475 m c 2 r 0.8 1 1.2 1.4 1.6 1.8 1.9 P0 0.42857 0.33333 0.25 0.17647 0.11111 0.05263 0.02564 Lq 0.15238 0.675 1.3451 2.84444 7.67368 17.5872 L 0.95238 1.33333 1.875 2.7451 4.44444 9.47368 19.4872 Wq 0.00076 0.00133 0.00225 0.00384 0.00711 0.01705 0.03703 W 0.00476 0.00533 0.00625 0.00784 0.01111 0.02105 0.04103

System Size vs Utilization

Adding Processors l 475 m 250 c 2 3 4 5 6 r 1.9 P0 0.02564 0.16618 0.25418 0.30933 0.33382 Lq 17.5872 0.89488 0.23785 0.0631 0.01479 L 19.4872 2.79488 2.13785 1.9631 1.91479 Wq 0.03703 0.00188 0.0005 0.00013 3.1E-05 W 0.04103 0.00588 0.0045 0.00413 0.00403

Time to Process vs Number of Servers

System Size Probabilities n Pn Probability Cumulative Probability P0 0.42857 1 P1 0.34286 0.77143 2 P2 0.13714 0.90857 3 P3 0.05486 0.96343 4 P4 0.02194 0.98537 5 P5 0.00878 0.99415 6 P6 0.00351 0.99766 7 P7 0.00140 0.99906 8 P8 0.00056 0.99963 9 P9 0.00022 0.99985 10 P10 0.00009 0.99994 11 P11 0.00004 0.99998 12 P12 0.00001 0.99999 13 P13 1.00000 14 P14 0.00000 15 P15 16 P16 17 P17 18 P18 19 P19 20 P20

M/M/c Observations M/M/c is the general case M/M/c behaves similar to M/M/1 Adding processors reduces queue length and time However the System time can never be smaller than the processor time An infinite number of servers will have system time = processor time

Classic Design Question If my system is performing unsatisfactorally, do I add another processor or trade the current processor in for one that is twice as fast?

M/M/1 vs M/M/2 M/M/1 M/M/2 l 240 m 250 500 c 1 2 r 0.96 0.48 P0 0.2 0.038521 Lq 23.04 0.44308 0.03151 L 24 0.92308 0.99151 Wq 0.096 0.00185 0.000131 W 0.1 0.00385 0.004131

Observations Twice as fast is better because it reduces both the queuing time and the mean processing time