1. Asst. Professor in Mathematics
MA305 Markov Chain
By: Prof. Nutan Patel
Asst. Professor in Mathematics
IT-NU
A-203
patelnutan.wordpress.com
MA305 Mathematics for ICE
Mathematics for ICE

2. Markov Chain or Markov Process
Example: Suppose a box A contains 5 red, 3 white and 8 black marbles, while box B contains 3 red and 5 white marbles. A fair die is tossed and if 2 or 5 occurs a marble is chosen from B otherwise from A. further in box A, two red, one white and 4 black marbles are defective while in box B 1 red and 2 white marbles are defective. To determine the probability that a marble drawn at random is say a defective red marble.
we have to conduct a sequence of experiments in which each experiment has a finite number of outcomes with given probabilities as shown in the tree diagram:
MA305 Mathematics for ICE

3. Defective 2 red, 1 white, 4 black
2/5
Box A: 5 red, 3 white, 8 black
Defective 2 red, 1 white, 4 black
Here probability of a defective red marbles is
Red marbles
3/5
Non Defective
Defective
Box B: 3 red, 5 white
Defective 1 red, 2 white
5/16
1/3
white marbles
Box A
3/16
2/3
Non Defective
2/3
8/16
Defective
1/2
Black marbles
Q: Find probability of a non defective white marbles?
Non Defective
1/2
Defective
1/3
1/3
Red marbles
2/3
Non Defective
Box B
3/8
Defective
The four experiments are: toss a die, choose a box, draw a marble and decide whether it is defective.
5/8
2/5
white marbles
3/5
Non Defective
MA305 Mathematics for ICE

4. Markov Chains or Markov Process: Transition Matrix Markov Chain
Steady State
Finding the Steady-state Matrix
MA305 Mathematics for ICE

5. Transition Matrix
Markov chains provide a means to analyse certain kinds of problems. The techniques use matrix operations and system of equations.
The alumni office of a university knows that the generally 80% of the alumni who contribute one year will contribute the next year. They also know that 30% of those who do not contribute one year will contribute the next. To find answer the following kinds of questions :
If 40% of a graduating class contributes the first year, how many will contribute the second year? The fifth year? The tenth years?
Before we solve this kind of problems. Let’s define some terminology and basic concepts.
MA305 Mathematics for ICE

6. A person’s emotional state may be happy, angry or sad.
State: A state is a category, situation, outcome, or position that a process can occupy at any given time. The states are disjoint and cover all possible outcomes.
For example the alumni are either in the state of contributors or in the state of non contributors.
A patient is ill or well.
A person’s emotional state may be happy, angry or sad.
In a Markov process the system may move from one state to another at any given time.
When process moves from one state to the next, we say that a transition is made the present state to next state.
An alumni can make a transition from the non-contributory state to the contributor state or vice-versa.
MA305 Mathematics for ICE

7. The information on the proportion of alumni who do or do not contribute can be represented be a transition matrix.
Let C represent those who contribute and NC represent those who do not.
Transition Matrix:
Next State
C NC
Present C
State NC
0.8 is the probability that a person passes from present state C to next state C;
0.2 is the probability that a person passes from present state C to next state NC;
And so on..
MA305 Mathematics for ICE

8. Definition (Transition Matrix)
A transition matrix is a square matrix with each entry a number between 0 and 1.
And addition of each row is 1.
MA305 Mathematics for ICE

9. Assume that all the trucks are rented daily.
Ex: A rental firm has three locations. A truck rented at one location may be returned to any of the locations. The company’s records show the probability of a truck rented at once location being returned to another. From these records the transition matrix is formed. Each location is considered to be a state.
Returned to
Rented
from 2 3
Assume that all the trucks are rented daily.
If the trucks are initialy distributed with 40% at location 1, 25% at location 2, and 35% at location 3, find the distribution on second and third days.
Ans: the initial-state matrix is [ ]
The distribution for day 2 is
MA305 Mathematics for ICE

10. The distribution for day 3 is
Thus 45.3% are at location 1, 26.1% are at location 2, and 28.6% are at location 3 on the third day.
MA305 Mathematics for ICE

11. Each entry is between 0 and 1, The entries in each row add to 1.
Markov chain is a sequence of experiments with the following properties:
1. An experiment has a finite number of discrete outcomes, called states.
2. With each additional trial the experiment can move from its present state to any state or remain in the same state.
3. The probability of going from one state to another on the next trial depends only on the present state and not on past states.
4. The probability of moving from any one state to another in one step is represented in a transition matrix.
The transition matrix is square, since all possible states are used for rows and columns.
Each entry is between 0 and 1,
The entries in each row add to 1.
5. The state matrix times the transition matrix give the next-state matrix.
MA305 Mathematics for ICE