1. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

2. Last time on Flows and Networks: Highlights: continuous time Markov chain stochastic process X(t) countable or finite state space S Markov property transition rates independent t irreducible: each state in S reachable from any other state in S Assume ergodic and regular global balance equations (equilibrium eqns) π is stationary distribution solution that can be normalised is equilibrium distribution if equilibrium distribution exists, then it is unique and is limiting distribution

3. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

4. Birth-death process State space Markov chain, transition rates Bounded state space: q(J,J+1)=0 then states space bounded above at J q(I,I-1)=0 then state space bounded below at I Kolmogorov forward equations Global balance equations

5. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

6. Example: pure birth process Exponential interarrival times, mean 1/  Arrival process is Poisson process Markov chain? Transition rates : let t0<t1<…<tn<t Kolmogorov forward equations for P(X(0)=0)=1 Solution for P(X(0)=0)=1

7. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

8. Example: pure death process Exponential holding times, mean 1/  P(X(0)=N)=1, S={0,1,…,N} Markov chain? Transition rates : let t0<t1<…<tn<t Kolmogorov forward equations for P(X(0)=N)=1 Solution for P(X(0)=N)=1

9. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

10. Simple queue Poisson arrival proces rate , single server exponential service times, mean 1/  Assume initially empty: P(X(0)=0)=1, S={0,1,2,…,} Markov chain? Transition rates :

11. Simple queue Poisson arrival proces rate , single server exponential service times, mean 1/  Kolmogorov forward equations, j>0 Global balance equations, j>0

12. Simple queue (ctd)    j j+1   Equilibrium distribution:  <  Stationary measure; summable  eq. distrib. Proof: Insert into global balance Detailed balance!

13. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

14. Birth-death process State space Markov chain, transition rates Definition: Detailed balance equations Theorem: A distribution that satisfied detailed balance is a stationary distribution Theorem: Assume that then is the equilibrium distrubution of the birth-death prcess X.

15. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

16. Reversibility; stationarity Stationary process: A stochastic process is stationary if for all t1,…,tn,  Theorem: If the initial distribution is a stationary distribution, then the process is stationary Reversible process: A stochastic process is reversible if for all t1,…,tn,  NOTE: labelling of states only gives suggestion of one dimensional state space; this is not required

17. Reversibility; stationarity Lemma: A reversible process is stationary. Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), j  S, summing to unity that satisfy the detailed balance equations When there exists such a collection π(j), j  S, it is the equilibrium distribution Proof

18. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

19. Lemma 1.9 / Corollary 1.10: If the transition rates of a reversible Markov process with state space S and equilibrium distribution are altered by changing q(j,k) to cq(j,k) for where c>0 then the resulting Markov process is reversible in equilibrium and has equilibrium distribution where B is the normalizing constant. If c=0 then the reversible Markov process is truncated to A and the resulting Markov process is reversible with equilibrium distribution Truncation of reversible processes 10 A S\A

20. Time reversed process X(t) reversible Markov process  X(-t) also, but Lemma 1.11: tijdshomogeneity not inherited for non- stationary process Theorem 1.12 : If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), j  S, then the reversed process X(  - t) is a stationary Markov process with transition rates and the same equilibrium distribution Theorem 1.13: Kelly’s lemma Let X(t) be a stationary Markov processwith transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), j  S, and a collection of positive numbers  (j), j  S, summing to unity, such that then q’(j,k) are the transition rates of the time- reversed process, and  (j), j  S, is the equilibrium distribution of both processes.

21. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

22. Kolmogorov’s criteria Theorem 1.8: A stationary Markov chain is reversible iff for each finite sequence of states Notice that

23. Flows and Networks Plan for today (lecture 2): Questions? Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria Summary / Next Exercises

24. Summary / next: Birth-death process Simple queue Reversibility, stationarity Truncation Kolmogorov’s criteria Next input / output simple queue Poisson proces PASTA Output simple queue Tandem netwerk

25. Exercises [R+SN] 1.3.2, 1.3.3, 1.3.5, 1.5.1, 1.5.2, 1.5.5, 1.6.2, 1.6.3, 1.6.4