1. 1 Introduction to Stochastic Models GSLM 54100

2. 2 Outline  limiting distribution  connectivity  types of states and of irreducible DTMCs  transient, recurrent, positive recurrent, null recurrent  periodicity  limiting behavior of irreducible chains

3. 3 Connectivity

4. 4 Connectivity of a DTMC  connectivity: one factor that determines the limiting behavior of a DTMC 1 2 1 0.01 0.99 A 1 2 0.01 0.99 B 0.01

5. 5 Connectivity of a DTMC 1 2 1 0.01 0.99 A

6. 6 Connectivity of a DTMC 1 2 0.99 0.01 0.99 B 0.01

7. 7 Connectivity of a DTMC 1 2 0.99 0.01 0.99 B 0.01 1 2 1 0.99 A

8. 8 Connectivity of a DTMC 1 2 0.6 0.1 0.9 C 1 3 0.4

9. 9 Connectivity of a DTMC 1 2 1 0.01 0.99 A 1 2 0.8 0.1 0.9 D 0.2

10. 10 Connectivity of a DTMC  rows of P (  ) may not be the same as in the previous examples 1 2 1 0.5 3 4 5 1

11. 11 Connectivity of a DTMC 1 2 1 F 1  limit of P (m) may not exist

12. 12 Connectivity of a DTMC 1 2 3 0.25 1 1 0.75 1 2 3 0.25 0.99 1 0.75 0.01

13. 13 Types of States and of Irreducible DTMCs

14. 14 Limiting Results for a DTMC  depending on the type of states and the chain  type: transient, positive recurrent, null recurrent  connectivity and periodicity

15. 15 Transient State 1 2 1 0.01 0.99 A 1 2 1 0.5 3 4 5 1

16. 16 Recurrent State  state i is recurrent if P(return to i|X 0 = i) = 1

17. 17 Recurrent State  two types of recurrent states  positive recurrent: E(# of transitions to return to i|X 0 = i) <   null recurrent: E(# of transitions to return to i|X 0 = i) = 

18. 18 Periodicity

19. 19 Periodicity  state i is of period d if X n can return to state i in multiples of d  states 1, 2, 3 are of period 2  state i of period d 1 2 3 0.25 1 1 0.75

20. 20 Periodicity  period of states 1, 2, 3, and 4 = ?  state 4 of period 2  state 4 can return to itself in 2 steps 1 2 3 0.25 0.99 1 0.75 4 0.01 1 

21. 21 Communicating States  communicating states are of the same type  transient, positive recurrent, null recurrent at the same time  of the same period  states in an irreducible chain are of the same type  transient, positive recurrent, null recurrent at the same time  of the same period

22. 22 Limiting Behavior of Irreducible Chains

23. 23 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j  N: a very large positive integer  # of periods at state j   j N  balance of flow   j N   i (  i N)p ij   j =  i  i p ij

24. 24 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j =  i  i p ij   1 = 0.9  1 + 0.2  2   2 = 0.1  1 + 0.8  2  linearly dependent  normalization equation:  1 +  2 = 1  solving:  1 = 2/3,  2 = 1/3 1 2 0.8 0.1 0.9 C 0.2

25. 25 Limiting Behavior of a Positive Irreducible Chain   1 = 0.75  2 + 0.01  3   3 = 0.25  2   1 +  2 +  3 = 1   1 = 301/801,  2 = 400/801,  3 = 100/801 1 2 3 0.25 0.99 1 0.75 0.01

26. 26 Limiting Behavior of a Positive Irreducible Chain  an irreducible DTMC {X n } is positive  there exists a unique nonnegative solution to    j : stationary (steady-state) distribution of {X n }

27. 27 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j = fraction of expected time at state j  average cost  c j for each visit at state j  random i.i.d. C j for each visit at state j  for aperiodic chain:

28. 28 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  profit per state: c 1 = 4, c 2 = 8, c 3 = -2  average profit 1 2 3 0.25 0.99 1 0.75 0.01

29. 29 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  C 1 ~ unif[0, 8], C 2 ~ Geo(1/8), C 3 = -4 w.p. 0.5; and = 0 w.p. 0.5  E(C 1 ) = 4, E(C 2 ) = 8, E(C 3 ) = -2  average profit 1 2 3 0.25 0.99 1 0.75 0.01

30. 30 Different Interpretations of  balance equations: balance of rates  total rate into a group of states = total rate out of a group of states  {0}:  0 = q  1  {0, 1}: p  1 = q  2  {0, 1, …, n}: p  n = q  n+1, n  1 0 1 q 2 3 p q p q p 1 … q

31. 31 Example: Condition for the Following Chain to be Positive  {0}:  0 = q  1   {0, 1}: p  1 = q  2 1   {0, 1, …, n}: p  n = q  n+1, n  1 1   positive  {  j } exists   0 +  1 +  2 + … = 1 has solution ..  p < q 0 1 q 2 3 p q p q p 1 … q

32. 32 Example 4.24 of Ross  four-state production process  states  {1, 2, 3, 4}  up states  {3, 4}, down states  {1, 2}  find E(up time) & E(down time) time 1 state 2 3 4 time down state up

33. 33 Example 4.24 of Ross   1 = 3/16,  2 = 1/4,  3 = 7/24,  4 = 13/48  how to find  E(up time)  E(down time)

34. 34 Example 4.24 of Ross  fraction of up time =  3 +  4 =  rate of turning from up to down = (p 31 +p 32 )  3 + (p 41 +p 42 )  4 = rate of turning from down to up = p 13  1 + (p 23 +p 24 )  2 =