CSCI1600: Embedded and Real Time Software Lecture 34: Queuing Theory Steven Reiss, Fall 2017 This lecture actually ran short. It would help if I knew exactly what the parameters for service and arrival time were and meant (i.e. distiinguish rates from numbers, etc) to make that clear. Having more examples would also help. Possibly start with a real example (program example) and ask what the queue size/wait time/probability of

Queuing Theory History Agner Krarup Erlang (1878-1929) Analysis of telephone networks (1908-1922) Why does this involve queues?

Analysis of Telelphone Networks Issues How many operators are needed How many circuits are needed

Queuing Theory Arriving Jobs Server Departing Jobs A branch of probability theory studying queues Applications Computer systems Communication networks Call centers Transportation … Arriving Jobs Server Departing Jobs

Computer Systems Disk access queue Http requests How big it will be Will it be advantageous to sort it Http requests How many jobs waiting on a semaphore Average and distribution of wait times Scheduling behavior Expected wait time to run a job How long will switch queue be on trains

Parameters: Arrivals Arriving Jobs Server Departing Jobs Arrival rate λ How fast jobs are arriving Regular or random Random = Poisson process With some distribution

Parameters: service Arriving Jobs Server Departing Jobs Service demand S – amount of service the job needs Measured in various terms (time, size, …) Regular or random (with a given distribution) Service capacity C – how fast server process jobs Service rate μ = C/S Service time = S/C = 1/μ

Parameter: Service Server 1 Arriving Jobs Departing Jobs Server m Multiple symmetric servers m parallel servers constitute a single serving station Each has capacity C Overall capacity depends on number of serviced jobs C(N) = min(N,m)C

Parameters: Queue Arriving Jobs Server Departing Jobs Scheduling discipline First come, first served (FCFS, FIFO) Priority queue (Shortest in, first out) OS I/O scheduler Preemptive vs. nonpreemptive servicing Preemptive servicing can be interrupted by higher priority job Linux real-time thread scheduling

Statistical Basis for Queuing Theory Arrival probability distribution Service probability distribution Determine various properties of the system Average queue size Maximum expected queue size Average wait time Maximum expected wait time

Poisson Arrival A completely random arrival process The number of jobs is very large Jobs are independent of each other Job arrivals are not simultaneous λ: the average arrival rate Examples Phone calls to a switchboard Radioactive decay Customers going to checkout Trains needing switches thrown

Poisson Process Represented by Poisson distribution PDF CDF Probablity of seeing i jobs during the time interval T P[n(t) = i] = (λT)i/i! * e-λT PDF CDF

Poisson Process Properties Memoryless The number of arrivals in the time interval (T;T+Δt) Does not depend on the number before T Superposition Combination of m independent Poisson processes with λ1,λ2,…,λm is a Poisson process with λ= λ1+λ2+…+λm Interval between job arrivals Is distributed exponentially fx(x) = λe-λx

Exponential Service Distribution PDF: fs(t) = μe-μt where μ is the service rate CDF: Fs(t) = 1 - μe-μt Mean = Std. Deviation = 1/μ = service time

Other Distributions Erlang Hyperexponential General The job visits an exponential server k times Hyperexponential k types of jobs, each has probability πi Each type of job has its own service time Each type of jobs passed to its own server General With known mean and standard deviation

A/S/n Notation Proposed by D. Kendeall in 1953 Distribution lables Arrival process (A) – message arrival distribution Server process (S) – servicing time distribution Number of servers (n) Distribution lables M : exponential (memoryless, Poisson) D : deterministic (constant arrival/processing times) G : general Ek : k-stage Erlang Hk : k-stage Hyperexponential

Examples M/M/1 M/D/n G/G/n One server for jobs that arrive and are serviced by Poisson processes System with many independent jobs sources can be approximated this way M/D/n Arrival process is Poisson, service is deterministic There are n servers G/G/n General system with arbitrary arrival and service times

Goals of Queuing Theory Predict the performance of the system in steady state Not startup or shut down Assumptions The population of jobs is infinite The size of the queue is infinite (can be bounded) Specific distributions for A and S known

Performance Metrics W: waiting time (queuing delay) Time between receipt of a job and start of processing Depends on the load on the server S: service time (processing delay) The actual processing time of the job Depends on the serving capacity T: response time T = W + S Throughput Average number of jobs per time unit

Little’s Law Number of jobs in the system N(t) N = λT (average number of jobs) λ : the mean arrival rate T : the mean service time Queue size Q Q = max(0,N – m) Waiting time in the Queue W Q = λW

Little’s Law Example Clothing store Customers arrive 10/hour and stay 30 minutes λ = 10, T = 0.5 Average number of customers in the store N = 10*0.5 = 5 Register line subsystem in the store Average register queue is 2 people The wait time W = Q/λ = 2/10 = 0.2 hours

Other Metrics Traffic intensity p Utilization factor Single server: p = arrival rate * service time = λ/μ m servers: p = λ/(mμ) p is dimensionless, measured in Erlangs If p > 1, then more jobs arrive than the system can handle p < 1 needed to prevent overload Utilization factor U = min(p,1)

Analytic Solutions Some queuing models can be solved analytically M/M/1 for example Other variants

M/M/1 Performance Average # jobs in the system N = p/(1-p) Average queue size Q = p2/(1-p) Average wait time W = 𝑝 μ 1−𝑝 Average response time T = 1 μ 1−𝑝 p = arrival rate * service time = λ/μ μ :: service rate

M/M/1 Example Post office Result Customers arrive 10 per hour (λ=10) Clerk can handle 16 customer per hour (μ = 16) 𝑝= λ μ = 10 16 =0.625 Guess the average time a customer spends Result Average number of customers N = 1.667 Average queue size Q = 1.042 Average wait time W = 0.1042 hours (6.25 minutes) Average response time T = 10 minutes

How to Fix Overload How can we reduce job delays Obvious: Shorten μ (speed up the code, profile, optimize) Lengthen λ (decrease the load, make less bursty) Move to M/M/m How many servers do you want This can be solved as well

Problems Classical queuing theory is too restrictive Infinite number of jobs Only certain distributions are handled Job routine is not allowed Some limitations can be alleviated But calculations and the math involved are complex Provides insufficient information

Solving Queuing Problems Use a computer representation of the system Instead of a mathematical model Use simulation to learn the properties This is what is generally used MATLAB SimEvents OMNET Can construct arbitrary models and run them

Examples

Homework Monday Wednesday / Friday / Monday Course Review. Bring questions (or it will be short) Wednesday / Friday / Monday Project final presentations