1. Examples of DTMCs

2. Navigating the Web as a Markov Chain
Ranking web pages
One option is to use the number of other pages that link to a given page (easy to fool)
Could weigh each page by the number of pages pointing to it (easy to fool through “cliques”)
A more robust option is proceed recursively
Rank of page depends on rank of pages pointing to it, which in turn may depend on your rank (and that of other pages)
Solution must satisfy j = i iPij
Same formulation as Markov chain
Rank of page is its limiting probability

3. Some Real-World Challenges
The web’s “Markov chain” is not irreducible…
Pages that point to no other pages (absorbing state)
Solution: Add (fake) links from each page to all other pages with a small overall weight (popularity tax)
Final solution is sensitive to magnitude of tax
Solving for the limiting probabilities
The transition matrix is big but very sparse, so that matrix multiplication can be implemented efficiently

4. The Slotted Aloha Protocol
Distributed access of shared communication resource (channel)
At most one successful packet transmission per time slot
m hosts that transmit with probability p in each time slot
Collision if more than one host transmits in a slot
Retransmission with probability q after failed transmission (this is in addition to new transmissions)
Aloha Markov chain: State is number of messages being retransmitted
Probability pk of k new messages if Binomial(m,p)
Probability qnk of k retransmissions given n retransmission messages is also Binomial(n,q)

5. Aloha Transition Probabilities
If in state 0 (no packet left in system)
P0,0 = (1-p)m+mp(1-p)m-1 (0 or 1 packet transmission)
P0,1 = 0 (at least 2 packets left after a collision)
P0,j = Choose(m,j)pj(1-p)m-j, j=2,…,m
P0,j = 0, j>m (cannot increase state by more than m)
If in state k>0 (k packets waiting for retransmission)
Pk,j = 0, jk-2 or j>k+m
At most one successful transmission and no more than m new packets
Pk,k-1 = (1-p)m+kp(1-p)k-1
No new transmission, one retransmission
Pk,k = m(1-p)m-1(1-q)k +(1-p)m(1-q)k +(1-p)m(1-(1-q)k –kq(1-q)k-1)
one new transmission, no retransmission
No new transmission and no retransmission
No new transmission and two or more retransmissions
Pk,k+1 = m(1-p)m-1(1(1-q)k)
One new transmission and one or more retransmissions
Pk,k+j = Choose(m,j)pj(1-p)m-j, j=2,…,m
j>1 new transmissions and any number of retransmissions

6. Aloha Protocol Properties
Chain is readily seen to be irreducible and aperiodic
However it is transient for any fixed value of q
Expected number of transmissions when in state k
E[N] = mp+kq (we obviously need mp < 1, but it is not enough to ensure “stability”)
E[N] > 1 whenever k > 1/q
Stabilizing the chain calls for making q state dependent
We need mp+kqk < 1 or qk < (1-mp)/k
One possible option: q = -k,  > 1 (Ethernet protocol)