Examples of DTMCs

Navigating the Web as a Markov Chain Ranking web pages One option is to use the number of other pages that link to a given page (easy to fool) Could weigh each page by the number of pages pointing to it (easy to fool through “cliques”) A more robust option is proceed recursively Rank of page depends on rank of pages pointing to it, which in turn may depend on your rank (and that of other pages) Solution must satisfy j = i iPij Same formulation as Markov chain Rank of page is its limiting probability

Some Real-World Challenges The web’s “Markov chain” is not irreducible… Pages that point to no other pages (absorbing state) Solution: Add (fake) links from each page to all other pages with a small overall weight (popularity tax) Final solution is sensitive to magnitude of tax Solving for the limiting probabilities The transition matrix is big but very sparse, so that matrix multiplication can be implemented efficiently

The Slotted Aloha Protocol Distributed access of shared communication resource (channel) At most one successful packet transmission per time slot m hosts that transmit with probability p in each time slot Collision if more than one host transmits in a slot Retransmission with probability q after failed transmission (this is in addition to new transmissions) Aloha Markov chain: State is number of messages being retransmitted Probability pk of k new messages if Binomial(m,p) Probability qnk of k retransmissions given n retransmission messages is also Binomial(n,q)

Aloha Transition Probabilities If in state 0 (no packet left in system) P0,0 = (1-p)m+mp(1-p)m-1 (0 or 1 packet transmission) P0,1 = 0 (at least 2 packets left after a collision) P0,j = Choose(m,j)pj(1-p)m-j, j=2,…,m P0,j = 0, j>m (cannot increase state by more than m) If in state k>0 (k packets waiting for retransmission) Pk,j = 0, jk-2 or j>k+m At most one successful transmission and no more than m new packets Pk,k-1 = (1-p)m+kp(1-p)k-1 No new transmission, one retransmission Pk,k = m(1-p)m-1(1-q)k +(1-p)m(1-q)k +(1-p)m(1-(1-q)k –kq(1-q)k-1) one new transmission, no retransmission No new transmission and no retransmission No new transmission and two or more retransmissions Pk,k+1 = m(1-p)m-1(1(1-q)k) One new transmission and one or more retransmissions Pk,k+j = Choose(m,j)pj(1-p)m-j, j=2,…,m j>1 new transmissions and any number of retransmissions

Aloha Protocol Properties Chain is readily seen to be irreducible and aperiodic However it is transient for any fixed value of q Expected number of transmissions when in state k E[N] = mp+kq (we obviously need mp < 1, but it is not enough to ensure “stability”) E[N] > 1 whenever k > 1/q Stabilizing the chain calls for making q state dependent We need mp+kqk < 1 or qk < (1-mp)/k One possible option: q = -k,  > 1 (Ethernet protocol)