1. Chapter 5 Markov Analysis

2. Learning Objectives Students will be able to:
Determine future states or conditions using Markov analysis.
Compute long-term or steady-state conditions by using only the matrix of transition probabilities.
Explain the use of absorbing state analysis in predicting future conditions.

3. Chapter Outlines
5.1 Introduction. 5.2 States and State Probabilities. 5.3 Matrix of Transition Probabilities. 5.4 Predicting Future Market Shares. 5.5 Markov Analysis of Machine Operations. 5.6 Equilibrium Conditions. 5.7 Absorbing States and the Fundamental Matrix: Accounts Receivable Application.

4. 1. Introduction Markov Analysis
A technique dealing with probabilities of future occurrences with currently known probabilities.
Numerous applications in:
Business (e.g., market share analysis),
Bad debt prediction,
Machine breakdown prediction,
University enrollment predictions.

5. Matrix of Transition Probabilities
Shows the likelihood that the system will change from one time period to the next.
Markov Process.
It enables the prediction of future states or conditions.

6. Assumptions of Markov Analysis
1. A finite number of possible states. 2. Probability of changing states remains the same over time. 3. Future state predictable from previous state and the matrix of transition probabilities. 4. Size and states of the makeup of the system (e.g., the total number of manufactures and customers) do not change during the analysis. - States are collectively exhaustive: (All possible states have been identified). - States are mutually exclusive: (Only one state at a time is possible).

7. 2. States and State Probabilities
States are used to identify all possible conditions of a process or system.
States are mutually exclusive (A system can exist in only one state at a time).
Examples include:
Working and broken states of a machine,
Three shops in town, with a customer able to patronize (treat) one at a time,
Courses in a student schedule with the student able to occupy only one class at a time.

8. States and State Probabilities (Cont’d.)
Identify all states.
Determine the probability that the system is in this state.
This information is placed into a vector of state probabilities.
p(i ) = vector of state probabilities for period i
p(i ) = (p1, p2, p3,…,pn) ………. (15-1)
Where:
n = number of states.
p1, p2,…,pn = P (being in state 1, 2, …, n).

9. States and State Probabilities (continued)
For example:
If dealing with only 1 machine, it may be known that it is currently functioning correctly.
The vector of states can then be shown:
p(1) = (1, 0)
Where:
p(1) = vector of states for the machine in period 1,
p1 = 1 = P (being in state 1) = P (machine working),
p2 = 0 = P (being in state 2) = P (machine broken).
Most of the time, problems deal with more than one item!

10. The Vector of State Probabilities for Three Grocery Stores Example
The vector of states for people in the small town with three grocery stores.
100,000 people could be shopping at the three stores during any given month.
40,000 people may be shopping at American Food Store, which will be called state 1.
30,000 people may be shopping at Food Mart, which will be called state 2.
30,000 people may be shopping at Atlas Food, which will be called state 3.

11. Three Grocery Stores example (continued)
The probability that a person will be shopping at one of these grocery stores is:
State 1 = store 1 (American Food Store) = 40,000/100,000 = 40%,
State 2 = store 2 (Food Marts) = 30,000/100,000 = 30%,
State 3 = store 3 (Atlas Foods) = 30,000/100,000 = 30%.
Vector of state probabilities:
p(1) = (0.4, 0.3, 0.3)
Where:
p(1) = vector of state probabilities in period 1,
p1 = 0.4 = P (of a person being in store 1),
p2 = 0.3 = P (of a person being in store 2),
p3 = 0.3 = P (of a person being in store 3).

12. Three Grocery Stores example (continued)
Probabilities in the vector of states for the stores represent market share for the first period.
In period 1, the market shares are:
Store 1: = 0.4 = 40%,
Store 2: = 0.3 = 30%,
Store 3: = 0.3 = 30%.
But, every month, customers who frequent one store have a likelihood of visiting another store.
Customers from each store have different probabilities for visiting other stores.

13. Three Grocery Stores example (continued)
Store-specific customer probabilities for visiting a store in the next month:
Store 1:
Return to Store 1 = 0.8 ,
Visit Store = 0.1,
Visit Store = 0.1 .
Store 2:
Return to Store 2 = 0.7,
Visit Store = 0.1,
Visit Store = 0.2.
Store 3:
Return to Store 3 = 0.6,
Visit Store = 0.2,
Visit Store = 0.2.

14. Three Grocery Stores example (continued)
Combining the starting market share with the customer probabilities for visiting a store next period yields the market shares in the next period:

15. The Diagram for Three Grocery Stores Example
Tree Diagram:
American Food #1
0.4
Food Mart#2
0.3
Atlas Foods#3 #1 #2 #3
0.32 = 0.4(0.8)
0.8
0.1
0.7
0.2
0.6
0.04 = 0.4(0.1)
0.03 = 0.3(0.1)
0.21 = 0.3(0.7)
0.06 = 0.3(0.2)
0.18 = 0.3(0.6)
Market share for American Food #1 next month = = 0.41 = (41%)

16. 3. Matrix of Transition Probabilities
To calculate periodic changes, it is much more convenient to use:
a matrix of transition probabilities,
a matrix of conditional probabilities of being in a future state given a current state.
Let Pij = conditional probability of being in state j in the future given the current state of i.
For example:
P12 is the probability of being in state 2 in the future given the event was in state 1 in the period before.

17. Matrix of Transition Probabilities (continued)
Let P = matrix of transition probabilities:
P P P13 …. P1n
P P P23 … P2n
P =
Pm … Pmn
Important:
Each row must sum to 1.
But, the columns do NOT necessarily sum to 1.
Row Sum 1
….. (16-2) * *

18. Transition Probabilities for the Three Grocery Stores
The previously identified transitional probabilities for each of the stores can now be put into a matrix:
P =
Row 1 interpretation:
0.8 = P11 = P (in state 1 after being in state 1),
0.1 = P12 = P (in state 2 after being in state 1),
0.1 = P13 = P (in state 3 after being in state 1).

19. Transition Probabilities for the Three Grocery Stores (continued)
P =
Row 2 interpretation:
0.1 = P21 = P (in state 1 after being in state 2),
0.7 = P22 = P (in state 2 after being in state 2),
0.2 = P23 = P (in state 3 after being in state 2).
Row 3 interpretation:
0.2 = P31 = P (in state 1 after being in state 3),
0.2 = P32 = P (in state 2 after being in state 3),
0.6 = P33 = P (in state 3 after being in state 3).

20. 4. Predicting Future Market Shares
Grocery Store example
A purpose of Markov analysis is to predict the future.
Given:
Vector of state probabilities (p) and,
Matrix of transitional probabilities (P).
It is easy to find the state probabilities in the future.
This type of analysis allows the computation of the probability that a person will be at one of the grocery stores in the future.
Since this probability is equal to market share, it is possible to determine the future market shares of the grocery store.

21. Predicting Future Market Shares (continued)
Grocery Store example When the current period is 0, finding the state probabilities for the next period (1) can be found using: p(1) = p(0)P …………… (16-3) Generally, in any period n, the state probabilities for period (n + 1) can be computed as: p(n +1) = p(n)P ……………. (16-4)

22. Predicting Future Market Shares (continued)
= state probabilities π [ ]
(0) = p 6 . 2 7 1 8 ú û ù ê ë é = P
(0) = p
(1) P 6 . 2 7 1 8 ú û ù ê ë é [ ]
(1) = p [ ] ) 1 ( = p
(.4 * * *.2),
(.4* * *.2),
(.4* * *.6)
(1)
= [ ].

23. Predicting Future Market Shares (continued)
p(2 ) = p(1) P, and p(1 ) = p(0) P,
Then: p(2 ) = p(0) P 2,
In general:
p(n) = p(0) ………… (16-5)
Therefore, the state probabilities n periods in the future can be obtained from the:
current state probabilities,
matrix of transition probabilities.

24. Lab Exercise # 5
Solve the Three Grocery Stores Example using the QM for Windows (see Appendix 16.1).
2. Solve the Three Grocery Stores Example using Excel (see Appendix 16.2).
To be continue

33. Excel Input
Current state Probabilities

34. Excel Input
Highlight the three cells B7,C7 and D7 where the resulting matrix will be.
Enter:
=MMULT(B6:D6;E6:G8)

35. Press Ctrl+Shift+Enter all at once

36. Enter remaining formulas for other results

37. =MMULT(B7:D7;E6:G8)

38. Excel Input and Formulas

39. Excel Output for the Example

40. Excel Output for the Example

41. 5. Markov Analysis of Machine Operations
Paul Tolsky has recorded the operations of his milling machine for several years.
Over the past two years, 80% of the time the milling machine functioned correctly during the current month if it had functioned correctly in the preceding month. This also means that only 20% of the time did the machine not function correctly for a given month when it was functioning correctly during the preceding month.
In addition, it has been observed that 90% of the time the machine remained incorrectly adjusted for any given month if it was incorrectly adjusted the preceding month.

42. 5. Markov Analysis of Machine Operations
Paul Tolsky has recorded the operations of his milling machine for several years. Over the past two years, 80% of the time the milling machine functioned correctly during the current month if it had functioned correctly in the preceding month. This also means that only 20% of the time did the machine not function correctly for a given month when it was functioning correctly during the preceding month. In addition, it has been observed that 90% of the time the machine remained incorrectly adjusted for any given month if it was incorrectly adjusted the preceding month.
These values can now be used to construct the vector of state probabilities and the matrix of transition probabilities. State 1 is a situation in which the machine is functioning correctly, and state 2 is a situation in which the machine is not functioning correctly. 42

43. Markov Analysis of Machine Operations Vector of State probabilities
Working Not working (this period)
Working
Not working
P =
(Last period)
Where:
P11 = 0.8 = probability of machine functioning correctly this period given it was working last period.
P12 = 0.2 = probability of machine not working this period given it was working last period.
P21 = 0.1 = probability of machine working this period if not working last period.
P22 = 0.9 = probability machine not working this period if not working last period.

44. Markov Analysis of Machine Operations (continued)
If the machine works this month, what is the probability that the machine will be functioning correctly next month?
p(1) = p(0)P
= (1, 0)
= [(1)(0.8) + (0)(0.1), (1)(0.2) + (0)(0.9)]
= (0.8, )
Then there is:
a 80% chance that it will be working next month,
a 20% chance it will be broken.

45. Markov Analysis of Machine Operations (continued)
What is the probability that the machine will be functioning correctly in two months?
p(2) = p(1)P
= (0.8, 0.2)
= [(0.8)(0.8) + (0.2)(0.1), (0.8)(0.2) + (0.2)(0.9)]
= (0.66, )
If the machine functions correctly, then in two months there is:
a 66% chance that it will be functioning correctly,
a 34% chance it will not be functioning correctly.

46. 6. Equilibrium Conditions
Equilibrium state probabilities are the long-run average probabilities for being in each state.
Equilibrium conditions exist if state probabilities do not change after a large number of periods.
At equilibrium, state probabilities for the next period equal the state probabilities for the current period.

47. Equilibrium Conditions (Cont’d.)
One way to compute the equilibrium share of the market is to use Markov analysis for a large number of periods and see if the future amounts approach stable values.

48. Equilibrium Conditions (Cont’d.)
For the machine example, the Markov analysis is repeated for 15 periods.
By the 15th period, the share of time the machine spends functioning correctly and not functioning correctly is around 66% and 34%, respectively.

49. Machine Example: Periods to Reach Equilibrium
State 1
State 2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1.0
0.8
0.66
0.562
0.4934
0.0
0.2
0.34
0.438
0.5066

50. Equilibrium Conditions
(n)  P  (n +1)
Matrix of
Transition
Probabilities
New
State
Current  
Equilibrium Conditions:
(n)  P  (n +1) = (n)
Current
State
Matrix of
Transition
Probabilities
New   =

51. p = p P Equilibrium Conditions (Cont’d.) p (n +1) = p (n) P
p (next period) = p (this period) P or
p (n +1) = p (n) P
At Equilibrium:
p (n) = p (n +1) = p (n) P *
p (n) = p (n) P *
Dropping the n term:
p = p P
……… (16-6)

52. Machine Breakdown Example
At Equilibrium:
p = p P
(p1, p2) = (p1, p2)
Applying matrix multiplication:
(p1, p2) = [(p1)(0.8) + (p2)(0.1),
(p1)(0.2) + (p2)(0.9)]
Multiplying through yields
the equilibrium equations:
p1 = 0.8 p p (a)
p2 = 0.2 p p (b)

53. p’s = 1 In this example, then: p1 + p2 = 1 (d)
Machine Breakdown example (Cont’d.)
The state probabilities must sum to 1, therefore :
p’s = 1
p1 + p2 + … + pn = (c)
In this example, then:
p1 + p2 = (d)
In a Markov analysis, there are always n state equilibrium equations and 1 equation of state probabilities summing to 1.

54. Machine Breakdown Example (Cont’d.)
Summarizing the equilibrium equations:
p1 = 0.8 p p (1)
p2 = 0.2 p p (2)
p1 + p2 = (3)
, (2) are the same ! (each is in the form:
0.2 p p2 = 0)
Solving (2) & (3)by simultaneous equations:
p1 =
p2 =
Therefore, in the long-run, the machine will be functioning 33.33% of the time and broken down 66.67% of the time.

55. Check the Correctness of the Answer.
Multiplying the equilibrium states by the original matrix of transition probabilities, the results will be the same equilibrium states. This is a way to check the correctness of the answer. =
Initial-state probability values do not influence equilibrium conditions.

56. 7. Absorbing States and the Fundamental Matrix:
Accounting Receivable Application
Any state that does not have a probability of moving to another state is called an absorbing state.
An example of such a process is accounts receivable.

57. Accounts Receivable example
The possible states for any customer are:
State 1 (p1): Paid all bills (absorbing state),
State 2 (p2): Bad debt (overdue > 3 months “absorbing state”),
State 3 (p3): Overdue Less than 1 month,
State 4 (p4): Overdue between 1 and 3 months.
A transition matrix looks similar to:
Next month
Paid Bad <
Dept
This month
Paid
Bad Dept
<

58. Accounts Receivable example (Cont’d.)
The categories are determined by the oldest unpaid bill, it is possible to pay one bill that is one to three month old and still have another bill that is one month or less old.
Any customer may have more than one outstanding bill at any point in time.

59. Markov Process Fundamental Matrix I A B
P =
…. (16-7)
Partition the transition probability matrix into 4 quadrants to make 4 new sub-matrices:
I, 0, A, and B.

60. ( )-1 = Inverse of a Matrix
Fundamental Matrix (Cont’d.)
Where: I = Identity matrix, and
0 = Null matrix (a matrix with all 0’s).
Then:
Fundamental Matrix
…. (16-8)
( )-1 = Inverse of a Matrix
Once F is found, multiply by the A matrix: FA
The FA indicates the probability that an amount in one of the non-absorbing states will end up in one of the absorbing states in the long run.

61. Fundamental Matrix (Cont’d.)

62. Inverse of a 2×2 Matrix
The inverse of a matrix is:
Where:
r = ad - bc

63. F A FA =

64. Once the FA matrix is found, multiply by the M vector, which is the starting values for the non-absorbing states, MFA,
Where:
M = ( M1 , M2 , M3 , … Mn )
n = number of nonabsorbing states,
M1 = amount in the first state or category,
M2 = amount in the second state or category, ……
Mn = amount in the nth. state or category,

65. Assume that there is $2000 in the less than one month category and $5000 in the one to three month category. Then:
M = (2000, 5000).

66. Amount paid and amount in bad debts = MFA
=(2000, 5000) FA M
= (6240, 760)
Thus, out of the total of $7000: ($2000 in the less than one month category, and $5000 in the one to three month category), $6240 will be eventually paid, and $760 will end up as bad debts.

67. Lab Exercise # 5 (Cont’d.).
3. Explain in details the use of QM for Windows and the Excel to find the Fundamental Matrix and Absorbing States for the Accounts Receivable Example.
(see Appendix 16.2).

68. QM for Windows

71. Excel Input for the Account receivable Example

72. Calculate the Matrix I-B
=D3:E4-F5:G6
Ctrl+Shift+Enter

73. 1 - <#>

74. Calculate the Fundamental Matrix F
=MINVERSE(D9:E10)
Ctrl+Shift+Enter

76. Calculate FA
=MMULT(D12:E13;D5:E6)
Ctrl+Shift+Enter

78. Input the M Values

79. Calculate MFA Values
=MMULT(D18:E18;d15:E16)
Ctrl+Shift+Enter