1. Markov Analysis
Jørn Vatn
NTNU

2. Introduction
Markov analysis is used to model systems which have many different states
These states range from “perfect function” to a total fault state
The migration between the different states may often be described by a so-called Markov-model
The possible transitions between the states may further be described by a Markov diagram

3. Purpose
Markov analysis is well suited for deciding reliability characteristics of a system
Especially the method is well suited for small systems with complicated maintenance strategies
In a Markov analysis the following topics will be of interest
Estimating the average time the system is in each state. These numbers might further form a basis for economic considerations.
Estimating how frequent the system in average “visits” the various states. This information might further be used to estimate the need for spare parts, and maintenance personnel.
Estimate the mean time until the system enters one specific state, for example a critical state.

4. Markov Analysis procedure
Make a sketch of the system
Define the system states
Draw the Markov diagram with the transition rates
Quantitative assessment
Compilation and presentation of the result from the analysis

5. Make a sketch of the system
Pump system wit active pump and a spare pump in standby

6. Definition of system states
x1 = state of active pump
x2 = state of standby pump
System state xS
Component state
Comments x1 x2 2 1
Both pumps functioning
The active pump is in a fault state, the standby pump is functioning
Both pumps in a fault state

7. State transitions
For this system we have assumed that if the active pump fails, the standby pump could always be started
Further we assume that if both pumps have failed, they will both be repaired before the system is put into service again
The following transition rates are defined
1 = failure rate of the active pump
2 = failure rate of the standby pump (while running, 2 = 0 in standby position)
1 = repair rate of the active pump (1/1 = Mean Down Time when the active pump has failed)
B = repair rate when both pumps are in a fault state. I.e. we assume that if the active pump has failed, and a repair with repair rate 1 is started, one will ”start over again” with repair rate B, if the standby pump also fails, independent of “how much” have been repaired on the active pump.

8. Markov state space diagram
The circles represent the system states, and the arrows represent the transition rates between the different system states
The Markov diagram and the description of states represent the total qualitative description of the system

9. Quantitative assessment
We want to assess the following quantities
Average time the system remain in the various system states
The visiting frequencies to each system state

10. Transition matrix
The indexing starts on 0, and moves to r, e.g. there are r +1 system states
Each cell in the matrix has two indexes, where the first (row index) represent the ”from” state, whereas the second (column index) represent the “to” state.
The cells represent transition rates from one state to another
aij is thus the transition rate from state i to state j
The diagonal elements are a kind of ”dummy”-elements, which are filled in at the end, and shall fulfil the condition that all cells in a row adds up to zero

11. Example transition matrix: (From , To )

12. State probabilities
Let Pi(t) represent the probability that the system is in state i at time t
Now introduce vector notation, i.e.
P(t) = [P0(t), P1(t),…,Pr(t)]
From the definition of the matrix diagram it might be shown that the Markov state equations are given by: P(t) A = d P(t)/d t
These equations may be used to establish both the steady state probabilities, and the time dependent solution

13. Steady state probabilities
Let the vector P = [P0, P1,…,Pr] represent the average time the system is in the various system states in the long time run
For example, P0 is average fraction of the time the system is in state 0, P1 is average fraction of the time the system is in state 1
The elements P = [P0, P1,…,Pr] are also denoted steady state probabilities to indicate that in the stationary situation Pi represents the probability that the system is in state i.

14. The steady state solution
In the long run when the system has stabilized we must have that d P(t)/d t = 0, hence
P A = 0
This system of equations is over-determined, hence we may delete one column, and replace it with the fact that P0+ P1+…+Pr = 1
Hence, we have

15. The steady state solution
P A1 = b
where
and
b = [0,0, …,0,1]

16. Example
which gives

17. Numerical solution
To solve the steady state equations P A1 = b is a tedious task
Often we therefore solve these equations by numerical methods
The Markov.xls program does this, where we have to:
Define the transition rates
Assign numerical values to the transition rates
Specify the Markov state space matrix

19. Visiting frequencies
Often we are interested in evaluating how many times the system enters the various states, i.e. the visiting frequencies
The visiting frequency for state j is denoted j, and could be obtained by:
j = -Pjajj
From our example we obtain the “system failure rate”

20. Time dependent solution
Up to now we have investigated the steady state situation
In some situations we also want to investigate the time dependent solution, i.e. the probability that the system is in e.g. state 0 at time t
We now let Pi(t) be the probability that the system is in state i at time t
The time dependent solution may be found by:
P(t) A = d P(t)/d t
Which could be solved by Laplace methods, or numerical methods
For numerical methods we apply Markov.xls

21. Spare parts – Simple model
Assume a spare part regime where there is only one spare in the stock
Upon a demand (with demand rate ) a new spare is ordered
The intensity of arrival of a new spare is  = 1/ MTA (Mean Time to Arrival)

22. Spare parts 2 spares in stock
Assume a spare part regime where there are two spares in the stock
Upon a demand (with demand rate ) a new spare is ordered
The intensity of arrival of a new spares is  = 1/ MTA (Mean Time to Arrival) independent of how many in order