1. CSCI1600: Embedded and Real Time Software
Lecture 34: Queuing Theory
Steven Reiss, Fall 2017
This lecture actually ran short. It would help if I knew exactly what the parameters for service and arrival time were and meant (i.e. distiinguish rates from numbers, etc) to make that clear.
Having more examples would also help. Possibly start with a real example (program example) and ask what the queue size/wait time/probability of

2. Queuing Theory History
Agner Krarup Erlang ( )
Analysis of telephone networks ( )
Why does this involve queues?

3. Analysis of Telelphone Networks
Issues
How many operators are needed
How many circuits are needed

4. Queuing Theory Arriving Jobs Server Departing Jobs
A branch of probability theory studying queues
Applications
Computer systems
Communication networks
Call centers
Transportation …
Arriving Jobs
Server
Departing Jobs

5. Computer Systems Disk access queue Http requests
How big it will be
Will it be advantageous to sort it
Http requests
How many jobs waiting on a semaphore
Average and distribution of wait times
Scheduling behavior
Expected wait time to run a job
How long will switch queue be on trains

6. Parameters: Arrivals Arriving Jobs Server Departing Jobs
Arrival rate λ
How fast jobs are arriving
Regular or random
Random = Poisson process
With some distribution

7. Parameters: service Arriving Jobs Server Departing Jobs
Service demand S – amount of service the job needs
Measured in various terms (time, size, …)
Regular or random (with a given distribution)
Service capacity C – how fast server process jobs
Service rate μ = C/S
Service time = S/C = 1/μ

8. Parameter: Service Server 1 Arriving Jobs Departing Jobs Server m
Multiple symmetric servers
m parallel servers constitute a single serving station
Each has capacity C
Overall capacity depends on number of serviced jobs
C(N) = min(N,m)C

9. Parameters: Queue Arriving Jobs Server Departing Jobs
Scheduling discipline
First come, first served (FCFS, FIFO)
Priority queue (Shortest in, first out)
OS I/O scheduler
Preemptive vs. nonpreemptive servicing
Preemptive servicing can be interrupted by higher priority job
Linux real-time thread scheduling

10. Statistical Basis for Queuing Theory
Arrival probability distribution
Service probability distribution
Determine various properties of the system
Average queue size
Maximum expected queue size
Average wait time
Maximum expected wait time

11. Poisson Arrival A completely random arrival process
The number of jobs is very large
Jobs are independent of each other
Job arrivals are not simultaneous
λ: the average arrival rate
Examples
Phone calls to a switchboard
Radioactive decay
Customers going to checkout
Trains needing switches thrown

12. Poisson Process Represented by Poisson distribution PDF CDF
Probablity of seeing i jobs during the time interval T
P[n(t) = i] = (λT)i/i! * e-λT
PDF
CDF

13. Poisson Process Properties
Memoryless
The number of arrivals in the time interval (T;T+Δt)
Does not depend on the number before T
Superposition
Combination of m independent Poisson processes with λ1,λ2,…,λm is a Poisson process with λ= λ1+λ2+…+λm
Interval between job arrivals
Is distributed exponentially fx(x) = λe-λx

14. Exponential Service Distribution
PDF: fs(t) = μe-μt where μ is the service rate
CDF: Fs(t) = 1 - μe-μt
Mean = Std. Deviation = 1/μ = service time

15. Other Distributions Erlang Hyperexponential General
The job visits an exponential server k times
Hyperexponential
k types of jobs, each has probability πi
Each type of job has its own service time
Each type of jobs passed to its own server
General
With known mean and standard deviation

16. A/S/n Notation Proposed by D. Kendeall in 1953 Distribution lables
Arrival process (A) – message arrival distribution
Server process (S) – servicing time distribution
Number of servers (n)
Distribution lables
M : exponential (memoryless, Poisson)
D : deterministic (constant arrival/processing times)
G : general
Ek : k-stage Erlang
Hk : k-stage Hyperexponential

17. Examples M/M/1 M/D/n G/G/n
One server for jobs that arrive and are serviced by Poisson processes
System with many independent jobs sources can be approximated this way
M/D/n
Arrival process is Poisson, service is deterministic
There are n servers
G/G/n
General system with arbitrary arrival and service times

18. Goals of Queuing Theory
Predict the performance of the system in steady state
Not startup or shut down
Assumptions
The population of jobs is infinite
The size of the queue is infinite (can be bounded)
Specific distributions for A and S known

19. Performance Metrics W: waiting time (queuing delay)
Time between receipt of a job and start of processing
Depends on the load on the server
S: service time (processing delay)
The actual processing time of the job
Depends on the serving capacity
T: response time
T = W + S
Throughput
Average number of jobs per time unit

20. Little’s Law Number of jobs in the system N(t)
N = λT (average number of jobs)
λ : the mean arrival rate
T : the mean service time
Queue size Q
Q = max(0,N – m)
Waiting time in the Queue W
Q = λW

21. Little’s Law Example Clothing store
Customers arrive 10/hour and stay 30 minutes
λ = 10, T = 0.5
Average number of customers in the store N = 10*0.5 = 5
Register line subsystem in the store
Average register queue is 2 people
The wait time W = Q/λ = 2/10 = 0.2 hours

22. Other Metrics Traffic intensity p Utilization factor
Single server: p = arrival rate * service time = λ/μ
m servers: p = λ/(mμ)
p is dimensionless, measured in Erlangs
If p > 1, then more jobs arrive than the system can handle
p < 1 needed to prevent overload
Utilization factor
U = min(p,1)

23. Analytic Solutions Some queuing models can be solved analytically
M/M/1 for example
Other variants

24. M/M/1 Performance Average # jobs in the system N = p/(1-p)
Average queue size Q = p2/(1-p)
Average wait time W = 𝑝 μ 1−𝑝
Average response time T = 1 μ 1−𝑝
p = arrival rate * service time = λ/μ
μ :: service rate

25. M/M/1 Example Post office Result Customers arrive 10 per hour (λ=10)
Clerk can handle 16 customer per hour (μ = 16)
𝑝= λ μ = =0.625
Guess the average time a customer spends
Result
Average number of customers N = 1.667
Average queue size Q = 1.042
Average wait time W = hours (6.25 minutes)
Average response time T = 10 minutes

26. How to Fix Overload How can we reduce job delays Obvious:
Shorten μ (speed up the code, profile, optimize)
Lengthen λ (decrease the load, make less bursty)
Move to M/M/m
How many servers do you want
This can be solved as well

27. Problems Classical queuing theory is too restrictive
Infinite number of jobs
Only certain distributions are handled
Job routine is not allowed
Some limitations can be alleviated
But calculations and the math involved are complex
Provides insufficient information

28. Solving Queuing Problems
Use a computer representation of the system
Instead of a mathematical model
Use simulation to learn the properties
This is what is generally used
MATLAB SimEvents
OMNET
Can construct arbitrary models and run them

29. Examples

30. Homework Monday Wednesday / Friday / Monday
Course Review. Bring questions (or it will be short)
Wednesday / Friday / Monday
Project final presentations