1. Discrete-time markov chain (continuation)

2. The GambleR’s RUIN PROBLEM (ROSS)
Consider a gambler who at each play of the game has
probability of winning one unit: 𝑝
Probability of losing one unit: 1−𝑝
Assume that successive plays of the game are independent.
Assume state 0 (broke) and 𝑁>0 (goal is achieved) are absorbing states.

3. The GambleR’s RUIN PROBLEM (ROSS)
Question: What is the probability that, starting with 𝑖 units, the gambler’s fortune will reach 𝑁 before reaching 0?

4. The GambleR’s RUIN PROBLEM (ROSS)
Let 𝑋 𝑛 be the player’s fortune at time 𝑛. This is a Markov Chain: 𝑋 𝑛 . Possible states are 0,1,2,…𝑁. Draw the state transition diagram.

5. The GambleR’s RUIN PROBLEM (ROSS)
The transition probabilities are 𝑝 00 = 𝑝 𝑁𝑁 =1 𝑝 𝑖,𝑖+1 =𝑝=1− 𝑝 𝑖,𝑖−1 for 𝑖=1,2,…,𝑁−1

6. The GambleR’s RUIN PROBLEM (ROSS)
The states can be classified into two classes Class 1 (recurrent): {0,𝑁} After some finite time, the gambler will either attain the goal 𝑁 or go broke. Class 2 (transient): {1,2,…,𝑁−1}

7. The GambleR’s RUIN PROBLEM (ROSS)
Working equation: 𝑝 𝑖,𝑁 (𝑘) =𝑝 𝑝 𝑖+1,𝑁 (𝑘) +𝑞 𝑝 𝑖−1,𝑁 (𝑘) for 𝑖=1,2,…,𝑁−1, for some 𝑘 Since 𝑝+𝑞=1, 𝑝 𝑝 𝑖,𝑁 (𝑘) +𝑞 𝑝 𝑖,𝑁 (𝑘) =𝑝 𝑝 𝑖+1,𝑁 (𝑘) +𝑞 𝑝 𝑖−1,𝑁 (𝑘) Rearranging the equation: 𝑝 𝑖+1,𝑁 (𝑘) − 𝑝 𝑖,𝑁 𝑘 = 𝑞 𝑝 𝑝 𝑖,𝑁 (𝑘) − 𝑝 𝑖−1,𝑁 (𝑘)

8. The GambleR’s RUIN PROBLEM (ROSS)
𝑝 𝑖+1,𝑁 (𝑘) − 𝑝 𝑖,𝑁 𝑘 = 𝑞 𝑝 𝑝 𝑖,𝑁 (𝑘) − 𝑝 𝑖−1,𝑁 (𝑘)
Since 𝑝 0,𝑁 (𝑘) =0,
𝑝 2,𝑁 (𝑘) − 𝑝 1,𝑁 𝑘 = 𝑞 𝑝 𝑝 1,𝑁 𝑘 −0 = 𝑞 𝑝 𝑝 1,𝑁 (𝑘)
𝑝 3,𝑁 (𝑘) − 𝑝 2,𝑁 𝑘 = 𝑞 𝑝 𝑝 2,𝑁 𝑘 − 𝑝 1,𝑁 𝑘 = 𝑞 𝑝 𝑝 1,𝑁 (𝑘)

9. The GambleR’s RUIN PROBLEM (ROSS)
𝑝 4,𝑁 (𝑘) − 𝑝 3,𝑁 𝑘 = 𝑞 𝑝 𝑝 3,𝑁 𝑘 − 𝑝 2,𝑁 𝑘 = 𝑞 𝑝 𝑝 1,𝑁 (𝑘) ⋮
𝑝 𝑁,𝑁 (𝑘) − 𝑝 𝑁−1,𝑁 𝑘 = 𝑞 𝑝 𝑝 𝑁−1,𝑁 𝑘 − 𝑝 𝑁−2,𝑁 𝑘 = 𝑞 𝑝 𝑁−1 𝑝 1,𝑁 (𝑘)
Note: we also know that 𝑝 𝑁,𝑁 (𝑘) =1.

10. The GambleR’s RUIN PROBLEM (ROSS)
Adding all the equations in bullet: 𝑝 𝑁,𝑁 (𝑘) − 𝑝 1,𝑁 𝑘 = 𝑝 1,𝑁 𝑘 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑁−1 1= 𝑝 1,𝑁 𝑘 1+ 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑁−1 1= 𝑝 1,𝑁 𝑘 1− 𝑞 𝑝 𝑁 1− 𝑞 𝑝 if 𝑞 𝑝 ≠1

11. The GambleR’s RUIN PROBLEM (ROSS)
Hence, 𝑝 1,𝑁 𝑘 = 1− 𝑞 𝑝 1− 𝑞 𝑝 𝑁 if 𝑞 𝑝 ≠1

12. The GambleR’s RUIN PROBLEM (ROSS)
Adding only the first 𝑖−1 equations in bullet: 𝑝 𝑖,𝑁 (𝑘) − 𝑝 1,𝑁 𝑘 = 𝑝 1,𝑁 𝑘 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑖−1 𝑝 𝑖,𝑁 (𝑘) = 𝑝 1,𝑁 𝑘 1+ 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑖−1 𝑝 𝑖,𝑁 (𝑘) = 𝑝 1,𝑁 𝑘 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 if 𝑞 𝑝 ≠1

13. The GambleR’s RUIN PROBLEM (ROSS)
Combining 𝑝 1,𝑁 𝑘 = 1− 𝑞 𝑝 1− 𝑞 𝑝 𝑁 if 𝑞 𝑝 ≠1 𝑝 𝑖,𝑁 (𝑘) = 𝑝 1,𝑁 𝑘 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 if 𝑞 𝑝 ≠1

14. The GambleR’s RUIN PROBLEM (ROSS)
Results in 𝑝 𝑖,𝑁 𝑘 = 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 if 𝑞 𝑝 ≠1 Since 𝑝+𝑞=1 but 𝑞 𝑝 ≠1 then 𝑝≠ 1 2 .

15. The GambleR’s RUIN PROBLEM (ROSS)
When the gambler plays continuously without end: lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =1− 𝑞 𝑝 𝑖 if 𝑝> 1 2 lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =0 if 𝑝< 1 2

16. The GambleR’s RUIN PROBLEM (ROSS)
The case when 𝑞 𝑝 =1 or 𝑝= 1 2 :
1= 𝑝 1,𝑁 𝑘 1+ 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑁−1
means 𝑝 1,𝑁 𝑘 = 1 𝑁 .
𝑝 𝑖,𝑁 (𝑘) = 𝑝 1,𝑁 𝑘 1+ 𝑞 𝑝 + 𝑞 𝑝 2 +…+ 𝑞 𝑝 𝑖−1
means 𝑝 𝑖,𝑁 𝑘 = 𝑖 𝑁 .

17. The GambleR’s RUIN PROBLEM (ROSS)
The case when 𝑞 𝑝 =1 or 𝑝= 1 2 : lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 𝑖 𝑁 =0.

18. The GambleR’s RUIN PROBLEM (ROSS)
Therefore, when the gambler plays continuously without end: lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =1− 𝑞 𝑝 𝑖 if 𝑝> 1 2 lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =0 if 𝑝≤ 1 2

19. The GambleR’s RUIN PROBLEM (ROSS)
Therefore, when the gambler plays continuously without end: lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =1− 𝑞 𝑝 𝑖 if 𝑝> 1 2 There is a positive probability that the gambler’s fortune will increase indefinitely if 𝑝> 1 2 .

20. The GambleR’s RUIN PROBLEM (ROSS)
Therefore, when the gambler plays continuously without end: lim 𝑁→∞ 𝑝 𝑖,𝑁 𝑘 = lim 𝑁→∞ 1− 𝑞 𝑝 𝑖 1− 𝑞 𝑝 𝑁 =0 if 𝑝≤ 1 2 The gambler will surely go broke against an infinitely rich adversary if 𝑝≤ 1 2 .

21. STEADY-STATE AND PERIODICITY

22. Not all converge to steady-state
The long-run properties of a Markov chain depend greatly on the characteristics of its states and transition matrix. It may or may not converge to steady-state. Example of case not converging to steady-state: periodicity

23. CLASSES BASED ON “COMMUNICATION”
CLASSES of TYPE 1: Those that do communicate Example: An irreducible Markov Chain consists of one class, which contains all states. CLASSES of TYPE 2: Those that do not communicate

24. CLASSES BASED ON “COMMUNICATION”1
CLASSES BASED ON “COMMUNICATION”
State 0 .3 1
State 3
0.3
State 1
State 2
p=0.7
p=0.7

25. CLASSES BASED ON “COMMUNICATION”
CLASSES of TYPE 1: CLASS 1: {1,2} CLASSES of TYPE 2: CLASS 2: {0} CLASS 3: {3}

26. CLASSES BASED ON “COMMUNICATION”
THEOREMS:
Recurrence is a class property. That is, all states in a class are either recurrent or transient. (We will prove this later)
Periodicity is a class property. That is, if state i in a class has period t, then all states in that class have period t.

27. DEFINITION
The period of state 𝑖 is defined to be the integer 𝑡> 1 such that 𝑝 𝑖𝑖 (𝑛) =0 for all values of 𝑛 other than 𝑡, 2𝑡, 3𝑡,... and 𝑡 is the smallest integer with this property.
If there are two consecutive numbers 𝑠 and 𝑠+1 such that the process can be in state 𝑖 at times 𝑠 and 𝑠+1, the state is said to have period 1 and is called an aperiodic state.

28. DEFINITION
In a finite-state Markov chain, recurrent states that are aperiodic are called ergodic states.
A Markov chain is said to be ergodic if all its states are ergodic states.

29. 1
EXAMPLE 1:
State 0 .3 1
State 3
0.3
State 1
State 2
p=0.7
p=0.7

30. EXAMPLE 2: Periodic or not. If Periodic, what is the period
EXAMPLE 2: Periodic or not? If Periodic, what is the period? If aperiodic, is IT ergodic? 1
State 0 1 1
State 1
State 2
State 3 1

31. Example 3: Periodic or not. If Periodic, what is the period
Example 3: Periodic or not? If Periodic, what is the period? If aperiodic, is IT ergodic? 1
State 0
0.5
0.5
0.5 1
State 1
State 2
State 3
0.5

32. EXAMPLE 4: Periodic or not. If Periodic, what is the period
EXAMPLE 4: Periodic or not? If Periodic, what is the period? If aperiodic, is IT ergodic? 1 1
State R
State P
State S 1

33. EXAMPLE 5: Periodic or not. If Periodic, what is the period
EXAMPLE 5: Periodic or not? If Periodic, what is the period? If aperiodic, is IT ergodic?
0.5
0.5
0.5
0.5
0.5
State R
State P
State S
0.5

34. THEOREM on STEADY-STATE
The n-step transition probabilities of a Markov chain that is both irreducible and ergodic (all states are recurrent and aperiodic) will converge to steady-state probabilities as n grows large.

35. THEOREM on STEADY-STATE
Question: If I have a periodic state, does that mean the n-step transition probabilities of my Markov chain will NOT converge to steady-state probabilities as n grows large?

36. PROOF OF “Recurrence is a class property”
Restatement: If state 𝑖 is recurrent, and state 𝑖 communicates with state 𝑗, then state 𝑗 is recurrent. Proof. Since state 𝑖 communicates with state 𝑗, there exist integers 𝑘 and 𝑚 such that 𝑝 𝑖𝑗 (𝑘) >0 and 𝑝 𝑗𝑖 (𝑚) >0.

37. PROOF OF “Recurrence is a class property”
For any integer 𝑛, 𝑝 𝑗𝑗 (𝑚+𝑛+𝑘) ≥ 𝑝 𝑗𝑖 𝑚 𝑝 𝑖𝑖 𝑛 𝑝 𝑖𝑗 𝑘 . Then 𝑛=1 ∞ 𝑝 𝑗𝑗 (𝑚+𝑛+𝑘) ≥ 𝑝 𝑗𝑖 (𝑚) 𝑝 𝑖𝑗 (𝑘) 𝑛=1 ∞ 𝑝 𝑖𝑖 (𝑛) =∞

38. PROOF OF “Recurrence is a class property”
𝑛=1 ∞ 𝑝 𝑗𝑗 (𝑚+𝑛+𝑘) =∞ which means state 𝑗 is also recurrent.

39. RECALL: Transient state will only be visited a finite number of times
State 𝑖 is Recurrent if 𝑛=1 ∞ 𝑝 𝑖𝑖 (𝑛) =∞ Transient if 𝑛=1 ∞ 𝑝 𝑖𝑖 (𝑛) <∞ .