CSE 221: Probabilistic Analysis of Computer Systems Topics covered: Discrete time Markov chains (Sec )

Computation of n-step probabilities  One-step transition probability matrix:  What is the probability of going from state i to state j in n steps?

Absolute probability distribution  Definition:  Mathematical representation:

Absolute probability distribution (contd..)  How to compute:

Steady-state or limiting probabilities  Definition:  Classification of DTMCs based on limiting probabilities:

Absorbing chains  Example:  Transient states:  Absorbing states:

Absorbing chains (contd..)  Number of visits to transient state 1 before reaching state 0:  What is the “average” number of visits to each transient state before being absorbed?

Absorbing chains (contd..)  Arrange the entries of matrix P:  Partition the matrix P:

Absorbing chains (contd..)  Fundamental matrix M:  Mean number of visits:

Absorbing chains (contd..)  Example:

Irreducible chains  Example:  Definition:  Steady-state or equilibrium state:

Irreducible chains (contd..)  Computation of steady-state probabilities:

Irreducible chains (contd..)  Example: