1. Discrete Time Markov Chain
Nur Aini Masruroh

2. Discrete Time Markov Chains
Consider a stochastic process {Xn, n=0, 1, 2, …} that takes on a finite or countable number of possible values
– assume set of possible values are non negative integers {0, 1, 2, …}
• Let Xn= i denote the process in state i at time n
• {Xn, n=0, 1, 2, …} describes the evolution of process over time
• Define Pij as the probability that the process will next be in state j given that it is currently in state i Pij = P{Xn+1 = j | Xn= i}
• Such a stochastic process is known as a Discrete Time Markov chain (DTMC)

3. Discrete Time Markov Chains
DTMC can be used to model a lot of real life
stochastic phenomena
– Example: Xn can be the inventory on-hand of a warehouse at the nth period
– Example: Xn can be the amount of money a taxi driver gets for his nth trip
– Example: Xn can be the status of a machine on the nth day of operation
– Example: Xn can be the weather (rain/shine) on the nth day

4. DTMC properties Markov Property:
The conditional distribution of any future state Xn+1 given the past state X0, X1, …, Xn-1 and the present state
Xn, is independent of the past states and depends only on the present state
P{Xn+1 = j | Xn=i, Xn-1=in-1, …, X1=i1, X0= i0}=P{Xn+1 = j | Xn=i}= Pij
– Pij represents the probability that the process will, when in state i, next make a transition into state j
Since probabilities are non negative and since the process must make a transition into some state, we have that
Pij≥0, i,j ≥ 0;
i = 0, 1, …

5. Short term analysis
The N2 transition probabilities can be represented by matrix P with element pij;
0 ≤ pij ≤ 1
Sum of the row should be 1

6. Demonstrate the Markov property Find the probability transition
Defining a DTMC
Specify the state
State, s
Variable to describe the present situation of the system, i.e. pressure, temperature, etc
Demonstrate the Markov property
Find the probability transition

7. Example 1: rain
Suppose the chance of rain tomorrow depends on previous weather conditions only through whether or not it rains today, and not the past weather conditions. Suppose also that if it rains today, then it will rain tomorrow with probability α; and if it does not rain today, then it will rain tomorrow with probability β. Let 0 be the state when it rains and 1 when it does not rain. Model this as a DTMC!

8. Example 2: stock movement
Consider the following model for the value of a stock:
At the end of a given day, the price is recorded
If the stock has gone up, the probability that it will go up tomorrow is 0.7
If the stock has gone down, the probability that it will go up tomorrow is only 0.5 (assume stock staying the same as a decrease)
Model the price movement (up/down) as a DTMC

9. Example 3: Mood
On any given day Gary is either cheerful (C), so-so (S), or glum (G).
If he is cheerful today, then he will be C, S or G tomorrow with respective probabilities 0.5, 0.4, 0.1.
If he is so-so today, then he will be C, S, or G tomorrow with probabilities 0.3, 0.4, 0.3.
If he is glum today, then he will be C, S, or G tomorrow with probabilities 0.2, 0.3, 0.5.
Model this as a DTMC

10. Example 4: Gambler’s ruin
My initial fortune is $1 and my opponent’s fortune is $2. I win a play with probability p, in which case I receive $1 and my opponent loses $1. We play until one of us have fortune 0.
Model this as a DTMC

11. Example 5: beer branding
A leading brewery company in Singapore (label T) has asked its IE team to analyze its market position. It is particularly concerned about its major competitor (label A).
It is believed (and somewhat verified by consumer surveys) that consumers chose their brand preference based on the brand of beer they are currently consuming.
From market survey data collected monthly,
95% of the current consumers of label T will still prefer label T in the next month, while 3% will switch to label A and the remaining to label C (all other foreign brands)
90% of consumers of label A will remain loyal to label A, while 8% will shift preferences to label T
80% of consumers of label C will prefer label C, while 10% will shift preferences to label T.
Model the brand loyalty/switching of consumers as a DTMC

12. State transition diagram
The transition matrix of the Markov chain can be represented by a state transition diagram (“bubble” diagram)
In the diagram:
circles = nodes = states: one for each element of the state space
arrows = archs = transitions: one for each non-zero probability (labeled with probabilities)
Diagram Rule: arrows out of a state have label sums that add to 1.
Try to draw state transition diagram for the Gambler’s ruin and the beer branding examples!

13. Assignment
Consider an experiment in which a rat is wandering inside a maze. The maze has six rooms labeled F, 2, 3, 4, 5 and S. If a room has k doors, the probability that the rat selects a particular door is 1/k. However, if the rat reaches room F, which contains food, or room S, which gives it an electrical shock, then it is kept there and the experiment stops.
Model the rat’s movement around the maze as a DTMC
Draw the state transition diagram!

14. Chapman-Kolmogorov Equations
The n-step transition probabilities, Pijn is the probability that a process in state i will be in state j after n additional transitions.
The Chapman-Kolmogorov equations state:

15. Proff of Chapman-Kolmogorov Equations

16. N-step transition probabilities (probability matrix)
Let P(n) denote the matrix of n-step transition probabilities Pijn
Then the Chapman-Kolmogorov equations assert
P(n+m) = P(n).P(m)
i.e. P(2) = P(1+1) = P.P = P2
P(n) = P(n-1+1) = P(n-1).P = Pn
Thus the n-step transition matrix may be obtained by multiplying the matrix P by itself n times

17. Example (Rain)
Consider the weather example in which the chance of rain tomorrow depends only on whether or not it is raining today. If it rains today, then it will rain tomorrow with probability α = 0.7; and if it does rain today, then it will rain tomorrow with probability β = 0.4
Calculate the probability that it will rain four days from today given that it is raining today

18. Unconditional distribution of states
Recall that the one or n-step transition probabilities are conditional probabilities
Pijn is the probability that the state at time n is j given that the initial state at time 0 is i
Let’s find the unconditional distribution of the state at time n (distribution of the state regardless of where it started)
Firs, specify the probability distribution of the initial state
Then all unconditional probabilities can be computed by conditioning on the initial state

19. Example (rain)
Consider the weather example in which the chance of rain tomorrow depends only on whether or not it is raining today. If it rains today, then it will rain tomorrow with probability α = 0.7; and if it does rain today, then it will rain tomorrow with probability β = 0.4
Suppose that the initial probability of rain α0 = 0.4
What is the (unconditional) probability that it will rain four days after we begin keeping the weather records?

20. Solution P{X4=0} = α0P004 + α0P104 = 0.4 P004 + 0.6 P104
= (0.4)(0.5749) + (0.6)(0.5668)
= 0.57
A more general approach
Let α be the initial probability distribution vector
Let α4 be the probability distribution vector of the weather after four days
The unconditional distribution of the weather can be computed
α4 = α.P(4)
Where α is the initial probability distribution (vector)
Probability that it will rain four days after keeping record = 0.57
Probability that it will not rain four days after keeping record = 0.43

21. Also try the following examples
Gambler’s ruin example
What is the probability distribution of my fortune after 2 plays?
Beer branding example
Suppose there are 200,000 beer consumers in Singapore in the month of October, with 75% of consumers preferring label T and 15% label A.
What is the market share of each label in November?

22. Classification of states
Accessibility and communication
Classes and irreducibility
Transient and recurrent

23. Accessibility and communication
Definition:
State j is said to be accessible from state i if Pijn > 0 for some n ≥ 0
i.e. it is possible for the system to enter state j eventually when it stars from state i
Two states i and j that are accessible to each other are said to communicate (denote as )

24. Accessibility and communication
Relation of communication satisfies 3 properties
State i communicates with state i, for all i ≥ 0
Proof: from definition Pii0 = P{X0=i|X0=i} = 1
If state i communicates with state j, then state j communicates with state i
Proof: from definition
If state i communicates with state j and state j communicates with state k, then state i communicates with state k

25. Classes and irreducibility
Two states that communicate are said to be in the same class
Any two classes are either identical or disjoint
Concept of communication divides the state space into a number of separate classes
Definition:
A Markov chain is said to be irreducible if there is only one class (i.e. all states communicate with each other)

26. Example
Consider the Markov chain consisting of the three states 0, 1, 2 and having transition probability matrix as follows.
How many classes are there?
As , this is an irreducible chain (only has 1 class)

27. Example
Consider the Markov chain consisting of the four states 0, 1, 2, 3 and having transition probability matrix as follows.
How many classes are there?
{0, 1}, {2}, {3}

28. Transient and recurrent
For any state i, let fi denote the probability that starting in state i, the process will ever reenter state i
Definition:
State i is said to be recurrent if fi = 1
State i is said to be transient if fi < 1
If state is recurrent, then starting in state i, the process will definitely return to state i again (hence reenter infinitely often)
If state is transient, each time the process enters state i, there will be positively probability (1 – fi) that it will never again enter that state

29. Transient and recurrent
Proposition:
State i is recurrent if
State i is transient if
Remark:
In a finite state Markov chain, not all states can be transient (proof by contradiction)

30. Proof by contradiction
Suppose first that we have a m=4 states Markov Chain
Suppose that each state is transient
Then
After a finite amount of time T0, state 0 will never be visited again (transient)
After a finite amount of time T1, state 1 will never be visited again (transient)
After a finite amount of time T2, state 2 will never be visited again (transient)
After a finite amount of time T3, state 3 will never be visited again (transient)
Let Tmax = max {T1, T2, T3, T4}
After Tmax, no state will ever be visited again
But the process has to be in some state …. CONTRADICTION!
Hence, at least one state has to be recurrent
We can show this for any finite m

31. Transient and recurrent
Corollary:
if state i is recurrent and state i communicates with state j, then state j is recurrent (recurrence is a class property)
Transition is class property
All states of a finite irreducible Markov chain are recurrent

32. Example
Consider the Markov chain consisting of the four states 0, 1, 2, 3 and having transition probability matrix as follows.
Determine which states are transient and which are recurrent?
As every state is accessible from any other state, so the chain is irreducible. Therefore, every state is recurrent

33. Example
Consider the Markov chain having transition probability matrix as follows.
Determine the recurrent state!

34. Solution
We observe that 3 can only be reached from 3, therefore 3 is in a class of its own.
State 1 and 2 can reach each other but no other state, so form a class together
Furthermore, 4, 5, 6 all communicate with each other.
The division of class thus is {1, 2}, {3}, and {4, 5, 6}
Clearly, f3 = 0.4, so {3} is a transient class.
On the other hand, {1, 2} and {4, 5, 6} are both closed and therefore recurrent

35. Positive recurrent
Definition:
A state i is positive recurrent if, the state i is recurrent, and starting in i, the expected time until the process returns to state i is finite
It can be shown that positive recurrent is a class property
It can be shown that in a finite Markov chain, all recurrent states are positive recurrent

36. Period of states
A state i is said to have period d, if starting from state i, the chain can only revisit it, d or a multiple d steps later
Definition:
State i has period d if Piin = 0 whenever n is not divisible by d and d is the largest integer with this property (revisit state in d, 2d, 3d, … and d is the largest integer with this property)

37. Periodicity is class property
Fact
Periodicity is class property
Is state i in class has period d, then all states in that class have period d

38. Example

39. A state with period = 1 is said to be aperiodic Example:
Definition:
A state with period = 1 is said to be aperiodic
Example:
1/2

40. Ergodic
Definition: positive recurrent, aperiodic states are called ergodic A Markov chain is said to be ergodic if all states are ergodic states

41. Long term behavior and analysis
• In designing physical systems, there are often “start-up” effects that are different from what can be expected in the “long-run”.
– A designer might be interested in the long-run behavior, or operations
• The LLN holds for iid random variables.
• Question: Do similar limiting results hold for DTMC when n is large?
– Limiting distributions
– Long term averages

42. Limiting probabilities
Consider the two states Markov chain with transition probability (rain example)
Observations:
P(4) is almost identical to P(8)
Seems like Pijn is converging to some values as n ∞, which is the same for all i
There seems to be a limiting probability that the process will be in state j after a large number of transitions and this value is independent of the initial state

43. Example: Tank warfare
One theoretical model for tank warfare expresses the firing mechanism as a two-state Markov process in which the states are 0: a hit, 1: a miss. Thus Xn is 1 or 0 depending on whether the nth shot is a miss or a hit. Suppose the probability of the tank’s hitting on a certain shot after it had hit on the previous shot is ¾, and the probability of the tank’s hitting on a certain shot after it had missed on the previous shot is ½,
– find the probability that the 11th shot fired from the tank (the 10th shot after the first) hits its target, given that the initial shot hit.
– Suppose that a tank commander on first encountering an enemy fires his first round for ranging purposes, and suppose that it is fairly unlikely that the first round hits. More specifically, suppose that the probability of a hit on the initial shot is ¼, and that the probability of a miss on the initial shot is ¾. What is the probability that the fourth shot hits?

44. Limiting probabilities
Theorem:
For an irreducible ergodic Markov chain exists and is independent of i, let
Then πj is the unique non negative solution of
Remarks:
πj also equals the long-run proportion of time that the process will be in state j
If a cost cj is incurred whenever state j is visited, then the long-run average cost is given by

45. Example: rain
Suppose the chance of rain tomorrow depends on previous weather conditions only through whether or not it rains today, and not the past weather conditions. Suppose also that if it rains today, then it will rain tomorrow with probability α; and if it does not rain today, then it will rain tomorrow with probability β. Let 0 be the state when it rains and 1 when it does not rain. Find the limiting probabilities π0, π1 .

46. Solution