048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion Review #3: Discrete-Time Markov Chains

Spring – Packet Switch Architectures2 Simple DTMCs  “States” can be labeled (0,)1,2,3,…  At every time slot a “jump” decision is made randomly based on current state  1 0 p q 1-q 1-p a d f c b e (Sometimes the arrow pointing back to the same state is omitted)

Spring – Packet Switch Architectures3 1-D Random Walk  Time is slotted  The walker flips a coin every time slot to decide which way to go  X(t) p 1-p

Spring – Packet Switch Architectures4 Single Server Queue  Consider a queue at a supermarket  In every time slot:  A customer arrives with probability p  The HoL customer leaves with probability q Bernoulli(p) Geom(q)

Spring – Packet Switch Architectures5 Birth-Death Chain  Can be modeled by a Birth-Death Chain (aka. Geom/Geom/1 queue)  Want to know:  Queue size distribution  Average waiting time, etc

Spring – Packet Switch Architectures6 Discrete Time Markov Chains  Markov property (memoryless): “Future” is independent of “Past” given “Present”  A sequence of random variables {X n } is called a Markov chain if it has the Markov property:  States are usually labeled {0,1,2,…}  State space can be finite or infinite

Spring – Packet Switch Architectures7 Transition Probability  Probability to jump from state i to state j  Assume stationary: independent of time  Transition probability matrix: P = (p ij )  Two state MC:

Spring – Packet Switch Architectures8 Stationary Distribution Define Then  k+1 =  k P (  is a row vector) Stationary Distribution: if the limit exists. If  exists, we can solve it by

Spring – Packet Switch Architectures9 Balance Equations  These are called balance equations  Transitions in and out of state i are balanced

Spring – Packet Switch Architectures10 In General  If we partition all the states into two sets, then transitions between the two sets must be “balanced”.  Equivalent to a bi-section in the state transition graph  This can be easily derived from the Balance Equations

Spring – Packet Switch Architectures11 Conditions for  to Exist (I)  Definitions:  State j is reachable by state i if  State i and j communicate if they are reachable by each other  The Markov chain is irreducible if all states communicate

Spring – Packet Switch Architectures12 Conditions for  to Exist (I) (cont’d)  Condition: The Markov chain is irreducible  Counter-examples: p=1 1

Spring – Packet Switch Architectures13 Conditions for  to Exist (II)  The Markov chain is aperiodic:  Counter-example:

Spring – Packet Switch Architectures14 Conditions for  to Exist (III)  The Markov chain is positive recurrent:  State i is recurrent if it will be re-entered infinitely often:  Otherwise transient  If recurrent State i is positive recurrent if E(T i )<1, where T i is time between visits to state i Otherwise null recurrent

Spring – Packet Switch Architectures15 Irreducible Ergodic Markov Chain  The Markov chain is ergodic if it is positive recurrent and aperiodic.  In an irreducible ergodic Markov chain, if  k+1 =  k P, then:   is independent of the initial conditions   (j) is the limiting probability that the process will be in state j at time n.  It is also equal to the long-run proportion of time that the process will be in state j (ergodicity).  It is called the stationary probability.

Spring – Packet Switch Architectures16 Irreducible Ergodic Markov Chain  If f is a bounded function on the state space:  Let m jj be the expected number of transitions until the Markov chain, starting in state j, returns to state j. Then m jj =1/  (j) References: books on stochastic processes (e.g., Ross)