1. CS6800 Advanced Theory of Computation Fall 2012 Vinay B Gavirangaswamy
Markov Models
CS6800 Advanced Theory of Computation
Fall 2012
Vinay B Gavirangaswamy

2. Markov Property Processes future values are conditionally dependent on the present state of the system. Strong Markov Property Similar as Markov Property, where values are conditionally dependent on the stopping time (Markov time) instead of present state.
Introduction

3. Markov Process Process that follows Markov Property. e. g
Markov Process Process that follows Markov Property. e.g. First order Markov Process for probability space (, F, P) P(Xt+1=Sj | Xt=Si, Xt-1=Sk ,...) = P(Xt+1=Sj | qt=Si) , for t = 1, 2, 3, … where,  = {S1, S2, S3, …, Sn }, State at time t is Xt=Si Markov Model Model’s a stochastic system which assumes Markov property. It is formally represented by triplet M = (K, , A), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. Markov model is an non deterministic finite state machine.
Introduction (Contd.)

4. Simple Markov Model of Weather
Stochastic FSM
The transition matrix:
Stochastic matrix:
Rows sum up to 1
Double stochastic matrix:
Rows and columns sum up to 1
Simple Markov Model of Weather

5. Prediction Based on State Transition Probability
If we want to know probability of the sequence SUNNY SUNNY SUNNY SUNNY SUNNY
Take initial probability of SUNNY day i.e. on a any given day probability that it will be SUNNY is
And for use to get another SUNNY day after a SUNNY day is 0.42
So, by using Markov Chain we can say probability of getting 5 consecutive SUNNY day is
Prediction Based on State Transition Probability

6. Types of Markov Model System state is fully observable
System state is partially observable
System is autonomous
Markov chain
Hidden Markov model
System is controlled
Markov decision process
Partially observable Markov decision process
Types of Markov Model

7. Types of Markov Model (Contd.)
Markov Chain
Simplest Markov model
System state at time t+1 depends on state at t.
Hidden Markov Model
Similar to Markov chain but system states are unobservable.
Markov decision process
Models a system where outcomes are partially random and partially under the control of a decision maker.
Types of Markov Model (Contd.)

8. Types of Markov Model (Contd.)
Markov decision process It is formally represented by 4-tuple M = (K, , A, R), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. R is reward given for transition from state St to Sp with transition probability given in matrix A.
Types of Markov Model (Contd.)

9. Types of Markov Model (Contd.)
Partially observable Markov decision process
Similar to Markov decision process where state of system is partially observed. It is formally represented by 4-tuple M = (K, , O , A, ), where
K is a finite set of states.
 is a set of actions.
O is a set of observations.
A is a matrix representing transition probabilities.
 is a set of conditional observation probabilities.
Types of Markov Model (Contd.)

10. System that transitions from one state to another, between a finite number of possible states.
Next state depends only on current state and not its history.
Follows Markov property.
Formally for a stochastic process { Xt }
P{ Xt+1 = j | X0 = k0, , Xt-1 = kt-1, Xt = i }
= P{ Xt+1 = j | Xt = i }
for every i, j, k0, , kt-1 and for every t.
Stationary Markov Chains
Pr{ Xt+1 = j | Xt = i } = Pr{ X1 = j | X0 = i } for all t
Markov Chain

11. If the state space S = { 0, 1, . . . , m–1} then we have
j pij = 1  i and pij  0  i, j
where pij = Pr{ X1 = j | X0 = i }
(we must go some where)
(each transition has probability  0)
Transition Matrix

12. Representation of a Markov Chain as a Digraph
Each directed edge AB is associated with the positive transition probability from A to B. A B C D
0.2
0.3
0.5
0.05
0.95
0.8 1 1
0.8
0.2
0.3
0.5
0.05
0.95 A B C D
Representation of a Markov Chain as a Digraph

13. Properties of Markov Chain states
States of Markov chains are classified by the digraph representation (omitting the actual probability values)
A, C and D are recurrent states: they are in strongly connected components which are sinks in the graph. A B C D
B is not recurrent – it is a transient state
Alternative definitions:
A state s is recurrent if it can be reached from any state reachable from s; otherwise it is transient.
Properties of Markov Chain states

14. Another example of Recurrent and Transient StatesB C D
A and B are transient states, C and D are recurrent states.
Once the process moves from B to D, it will never come back.
Another example of Recurrent and Transient States

15. Irreducible Markov Chains
A Markov Chain is irreducible if the corresponding graph is strongly connected (and thus all its states are recurrent). A B C D 1 A B C D E
Irreducible Markov Chains

16. A state s has a period k if k is the GCD of the lengths of all the cycles that pass via s. (in the shown graph the period of A is 2).
A Markov Chain is periodic if all the states in it have a period k >1. It is aperiodic otherwise. A B C D E
Periodic States

17. A B C D
A Markov chain is ergodic if :
the corresponding graph is strongly connected.
It is not peridoic
Ergodic Markov Chains are important since they guarantee the corresponding Markovian process converges to a unique distribution, in which all states have strictly positive probability.
Ergodic Markov Chains

18. Reversible Markov chain
A Markov chain is reversible if
Reversible Markov chain

19. Markov Model Applications
Physics
Chemistry
Testing
Information sciences
Queuing theory
Internet applications
Statistics
Economics and finance
Social science
Mathematical biology
Games
Music
Markov Model Applications

20. "Wkipedia-Markov model," [Online]. Available: http://en. wikipedia
"Wkipedia-Markov model," [Online]. Available: [Accessed ].
P. Xinhui Zhang, "DTMarkovchains," [Online]. Available: [Accessed ].
References