1 Introduction to Stochastic Models GSLM 54100

2 Outline  limiting distribution  connectivity  types of states and of irreducible DTMCs  transient, recurrent, positive recurrent, null recurrent  periodicity  limiting behavior of irreducible chains

3 Connectivity

4 Connectivity of a DTMC  connectivity: one factor that determines the limiting behavior of a DTMC A B 0.01

5 Connectivity of a DTMC A

6 Connectivity of a DTMC B 0.01

7 Connectivity of a DTMC B A

8 Connectivity of a DTMC C

9 Connectivity of a DTMC A D 0.2

10 Connectivity of a DTMC  rows of P (  ) may not be the same as in the previous examples

11 Connectivity of a DTMC F 1  limit of P (m) may not exist

12 Connectivity of a DTMC

13 Types of States and of Irreducible DTMCs

14 Limiting Results for a DTMC  depending on the type of states and the chain  type: transient, positive recurrent, null recurrent  connectivity and periodicity

15 Transient State A

16 Recurrent State  state i is recurrent if P(return to i|X 0 = i) = 1

17 Recurrent State  two types of recurrent states  positive recurrent: E(# of transitions to return to i|X 0 = i) <   null recurrent: E(# of transitions to return to i|X 0 = i) = 

18 Periodicity

19 Periodicity  state i is of period d if X n can return to state i in multiples of d  states 1, 2, 3 are of period 2  state i of period d

20 Periodicity  period of states 1, 2, 3, and 4 = ?  state 4 of period 2  state 4 can return to itself in 2 steps 

21 Communicating States  communicating states are of the same type  transient, positive recurrent, null recurrent at the same time  of the same period  states in an irreducible chain are of the same type  transient, positive recurrent, null recurrent at the same time  of the same period

22 Limiting Behavior of Irreducible Chains

23 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j  N: a very large positive integer  # of periods at state j   j N  balance of flow   j N   i (  i N)p ij   j =  i  i p ij

24 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j =  i  i p ij   1 = 0.9   2   2 = 0.1   2  linearly dependent  normalization equation:  1 +  2 = 1  solving:  1 = 2/3,  2 = 1/ C 0.2

25 Limiting Behavior of a Positive Irreducible Chain   1 = 0.75   3   3 = 0.25  2   1 +  2 +  3 = 1   1 = 301/801,  2 = 400/801,  3 = 100/

26 Limiting Behavior of a Positive Irreducible Chain  an irreducible DTMC {X n } is positive  there exists a unique nonnegative solution to    j : stationary (steady-state) distribution of {X n }

27 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j = fraction of expected time at state j  average cost  c j for each visit at state j  random i.i.d. C j for each visit at state j  for aperiodic chain:

28 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  profit per state: c 1 = 4, c 2 = 8, c 3 = -2  average profit

29 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  C 1 ~ unif[0, 8], C 2 ~ Geo(1/8), C 3 = -4 w.p. 0.5; and = 0 w.p. 0.5  E(C 1 ) = 4, E(C 2 ) = 8, E(C 3 ) = -2  average profit

30 Different Interpretations of  balance equations: balance of rates  total rate into a group of states = total rate out of a group of states  {0}:  0 = q  1  {0, 1}: p  1 = q  2  {0, 1, …, n}: p  n = q  n+1, n  q 2 3 p q p q p 1 … q

31 Example: Condition for the Following Chain to be Positive  {0}:  0 = q  1   {0, 1}: p  1 = q  2 1   {0, 1, …, n}: p  n = q  n+1, n  1 1   positive  {  j } exists   0 +  1 +  2 + … = 1 has solution ..  p < q 0 1 q 2 3 p q p q p 1 … q

32 Example 4.24 of Ross  four-state production process  states  {1, 2, 3, 4}  up states  {3, 4}, down states  {1, 2}  find E(up time) & E(down time) time 1 state time down state up

33 Example 4.24 of Ross   1 = 3/16,  2 = 1/4,  3 = 7/24,  4 = 13/48  how to find  E(up time)  E(down time)

34 Example 4.24 of Ross  fraction of up time =  3 +  4 =  rate of turning from up to down = (p 31 +p 32 )  3 + (p 41 +p 42 )  4 = rate of turning from down to up = p 13  1 + (p 23 +p 24 )  2 =