Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Last time on Flows and Networks: Highlights: continuous time Markov chain stochastic process X(t) countable or finite state space S Markov property transition rates independent t irreducible: each state in S reachable from any other state in S Assume ergodic and regular global balance equations (equilibrium eqns) π is stationary distribution solution that can be normalised is equilibrium distribution if equilibrium distribution exists, then it is unique and is limiting distribution

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Birth-death process State space Markov chain, transition rates Bounded state space: q(J,J+1)=0 then states space bounded above at J q(I,I-1)=0 then state space bounded below at I Kolmogorov forward equations Global balance equations

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Example: pure birth process Exponential interarrival times, mean 1/  Arrival process is Poisson process Markov chain? Transition rates : let t0<t1<…<tn<t Kolmogorov forward equations for P(X(0)=0)=1 Solution for P(X(0)=0)=1

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Example: pure death process Exponential holding times, mean 1/  P(X(0)=N)=1, S={0,1,…,N} Markov chain? Transition rates : let t0<t1<…<tn<t Kolmogorov forward equations for P(X(0)=N)=1 Solution for P(X(0)=N)=1

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Simple queue Poisson arrival proces rate , single server exponential service times, mean 1/  Assume initially empty: P(X(0)=0)=1, S={0,1,2,…,} Markov chain? Transition rates :

Simple queue Poisson arrival proces rate , single server exponential service times, mean 1/  Kolmogorov forward equations, j>0 Global balance equations, j>0

Simple queue (ctd)    j j+1   Equilibrium distribution:  <  Stationary measure; summable  eq. distrib. Proof: Insert into global balance Detailed balance!

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Birth-death process State space Markov chain, transition rates Definition: Detailed balance equations Theorem: A distribution that satisfied detailed balance is a stationary distribution Theorem: Assume that then is the equilibrium distrubution of the birth-death prcess X.

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Reversibility; stationarity Stationary process: A stochastic process is stationary if for all t1,…,tn,  Theorem: If the initial distribution is a stationary distribution, then the process is stationary Reversible process: A stochastic process is reversible if for all t1,…,tn,  NOTE: labelling of states only gives suggestion of one dimensional state space; this is not required

Reversibility; stationarity Lemma: A reversible process is stationary. Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), j  S, summing to unity that satisfy the detailed balance equations When there exists such a collection π(j), j  S, it is the equilibrium distribution Proof

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Lemma 1.9 / Corollary 1.10: If the transition rates of a reversible Markov process with state space S and equilibrium distribution are altered by changing q(j,k) to cq(j,k) for where c>0 then the resulting Markov process is reversible in equilibrium and has equilibrium distribution where B is the normalizing constant. If c=0 then the reversible Markov process is truncated to A and the resulting Markov process is reversible with equilibrium distribution Truncation of reversible processes 10 A S\A

Time reversed process X(t) reversible Markov process  X(-t) also, but Lemma 1.11: tijdshomogeneity not inherited for non- stationary process Theorem 1.12 : If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), j  S, then the reversed process X(  - t) is a stationary Markov process with transition rates and the same equilibrium distribution Theorem 1.13: Kelly’s lemma Let X(t) be a stationary Markov processwith transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), j  S, and a collection of positive numbers  (j), j  S, summing to unity, such that then q’(j,k) are the transition rates of the time- reversed process, and  (j), j  S, is the equilibrium distribution of both processes.

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Kolmogorov’s criteria Theorem 1.8: A stationary Markov chain is reversible iff for each finite sequence of states Notice that

Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

Summary / next: Birth-death process Simple queue Reversibility, stationarity Truncation Kolmogorov’s criteria Next input / output simple queue Poisson proces PASTA Output simple queue Tandem netwerk

Exercises [R+SN] 1.3.2, 1.3.3, 1.3.5, 1.5.1, 1.5.2, 1.5.5, 1.6.2, 1.6.3, 1.6.4