1. 15 October 2012 Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Discrete Time Markov Chains

2. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  X(n): the state/value of a process at the n -th period (time slot)  X(n) is random and takes values in a finite or countable set  The sequence X(1), X(2), …, X(n) could be a) The values of a stock each day b) The web page a user is currently browsing c) The Access Point(AP) a moving user is currently associated with  We would like a probabilistic model for X(1),X(2),…,X(n) a) X(i) are independent => not so realistic (for above examples) b) X(i+1) depends on X(i) => not a bad model to start with 2 P{X(n+1) = j | X(n) = i,X(n-1) = i n-1,…,X(1) = X 0 } = P{X(n+1) = j | X(n) = i} = p ij (Markov Property)

3. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Assume state space: 0, 1, 2, …  P ij : P{S(n+1) = j | S(n) = i}   Stationary: P{S(n+1) = j | S(n) = i} = P{S(1) = j | S(0) = i}  for any n  p ij define a transition matrix P = 3

4. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Simple weather model  X(n): weather after n days  Simple channel error model  (total) prob of error = 0.1  Case 1: q = 0.1, p=0.9  Case 2: q = 0.9  What is the transition matrix P in both cases? 4 goodbad p 1-p q 1-q (uncorrelated) (burst of errors)

5. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A simple handover model  cell -> state  need to find transition probabilities  depend on road structure, user profile, statistics 5 user on the phone Cellular Network 0.2 0.3 0.5 0.4 0.7 0.2 0.15 0.2 0.8 0.6 Markov Chain

6. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  So far we know probabilities for next step: S(n)->S(n+1)  Probabilities for S(1) -> S(n)?  If P (2) = {p ij (2) } denotes the 2-step trans. probability matrix, then P (2) = P P  P ij (2) : multiply row I with column j 6

7. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Generalizing for n steps:  Kolmogorov-Chapman equations  P (n) = PP…P = (P) n (n-step transition probabilities)  P (n) = P (m) P (l)  e.g. P (5) = P (4) P (1) = P (2) P (3) = P (2) P P (2)  Initial state vector s (0) : e.g. (0,1,0,0) or = (0.2,0,0.6,0.2) Q: Where after n steps? A: s (n) = s (0) P (n) 7

8. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Your iPhone browser can pre-fetch pages to improve speed  The current Web Page being browsed has 3 links  Study of click statistics have shown that  A user clicks link 1 next with prob 0.8  Link 2 with prob 0.1  Link 3 with prob 0.1  It seems reasonable that the browser pre-fetches link 1  Assume now a tiny subset of the Web with the following transition matrix  Start at page A Q: Which 2 web pages should the browser pre-fetch? 8 S (1) = [0.2, 0.2, 0.5, 0.1] S (2) = [0.13, 0.29, 0.24, 0.34] S (0) = [1, 0, 0, 0]

9. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 9

10. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  The weather system  Data rates supported for outdoor user  e.g. (128Kbps, 320Kbps, 1 Mbps)  What about this one? 10

11. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis (Reachability) State j is reachable by i  P ij (n) > 0 for some n  denote i  j (Communication) states i and j communicate (i j) if i  j and j  i (Recurrence/Transience) Denote f i the probability to ever return to i, starting from i  Transient) state is transient if f i < 1  number of returns is geometric (f i ): why?  Recurrent) state i is recurrent if f i = 1  Positive recurrent) expected time between visits to i is finite  Null recurrent) expected time between visits is infinite (strange!?)  Periodicity) state i is periodic, if exists d such that p ii (n) = 0, if n ≠ kd  Largest d with above property is called period 11

12. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Definition: A subset of states S: {for all i,j in S => i j} is called a class Lemma 1: recurrence is a class property  if i is recurrent and i j, then j is recurrent  Proof (by contradiction): if j transient => can never return to j after some time => cannot be at i either (since there is always a chance to go from i to j) Lemma 2: transience is a class property  Similar argument  Periodicity and positive/null-recurrence are also class properties Irreducible) A Markov Chain is irreducible if it has only 1 class  A finite MC which is irreducible is always positive-recurrent 12 Theorem: If a DTMC is aperiodic, irreducible and positive recurrent (“ergodic”) then it has a stationary distribution π = {π 1,…,π N }

13. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Search for “Network Modeling”  1000s of pages (courses, companies, etc.) contain these keywords  Goal: Make the page I’m interested appear in at least the top 10 pages shown Q: How should pages be ranked? A1: A page is important if many links to this page Q: What is wrong with this metric? A: - link from Yahoo more important than link from /~spyropou - can easily “fool” this! (create many dummy pages pointing to mine) Q: How to fix this? A: weigh a link from x to page p with the number of links into x Q: Can this system be fooled? A: Yes! Create 1000 dummy pages pointing to each other and mine 13

14. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A page p has high rank (is important) if the pages pointing to it also have high rank Q: How is this different from before? A: All the dummy pages pointing to each other would still be low rank, if no “external” link from a high rank page Q: Is a link from a page with 1000 other links as important as a link from a page with 5 other links? A: No! If page i has a link to j and k total outgoing links  P ij = 1/k Q: Where does this lead? How do we calculate the rank? A: rank of page j = Σ i { rank of page i * Prob of link i  j} i.e. r j = Σ i r i P ij  (basis of) Page Rank Algorithm: rank of pages = stationary prob. of an MC with transitions P ij 14

15. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: What about this tiny WWW? A: π A = π N = 2/5, π M = 1/5 Q: But what about these two? A: These two chains are reducible and do not have a stationary distr. 15  Google’s heuristic: introduce additional arrows to avoid such loops and dead ends (not necessarily optimal!) Q: How does Google solve the huge system of eq (million pages)?

16. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 16 [Network] m nodes on a common wire  Or wireless channel  Time is slotted [New Packets] Each node transmits a new packet with probability p (p < 1/m) [Success] If exactly 1 packet is transmitted [Collision] If k>1 (re-)transmissions during a slot  Every collided message is stored, to be resent later [Retransmission] Each collided(back-logged) message is retransmitted with prob q new packet old packet p p p √ X q Does Aloha work??

17. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Q: What should we take as the state of the chain X n ?  A: The number of backlogged messages (at slot n)  Transition probabilities from state 0 (no backlogged msg) P 00 = P 01 = P 0k = 17 (0 or 1 node transmits) 0 (not possible) if 1< k ≤ m (any k of the m nodes transmit) 0 if k > m

18. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis P k,k-1 = (better) P k,k = P k,k+1 =(worse) P k,k+r = (1< r ≤ m)(worse) P k,k+r = 0(r > m)  Assume we are now at state k (i.e. k messages backlogged) 18 (0 new, 1 old) (0 new, 0 old)(1 new, 0 old) (r new, any old) (0 new, ≥2 old) (1 new) (>0 old)

19. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : :  How can we tell if “Aloha works”?  Assume 10 nodes and transmission probability p = 0.05  Load = 0.5 Capacity  Intuition: (necessary) q should be small (re-Tx)  To ensure retransmissions don’t overload medium  Let’s assume q = 0.005 (10 times smaller than p)  Q: Is Aloha Stable? If backlog becomes infinite => delay goes to infinity! 19

20. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  When at state k:  P back (k) : reduce backlog  P fwd (k) : increase backlog (MHB, Ch.10) (Why?)  So??? 20 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

21. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  For large enough k => states k+1, k+2, …, are transient  Markov Chain is transient => Aloha protocol is unstable!  Q: would Aloha work if we make q really (REALLY) small?  A: No!  Intuition: Let E[N] the expected Tx at state k  If E[N] ≥ 1 then situation either stays the same or worse  But E[N] = mp + kq --- what happens if k  ∞? 21 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

22. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: How can we fix the problem and make the chain ergodic (and the system stable)? A1: E[N] = mp + kq q < (1-mp)/k i.e. q = f(backlog) A2: or be more aggressive  geometric backoff q = a/k n  a < (1-mp) A3: or even exponential q = β -k  β > 1 Exponential backoff is the basis behind Ethernet Q: Why should q not be too small? A: Retransmission delay goes to infinity 22

23. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Time Averages  N i (t) = number of times in state i by time t  p i = (percentage of time in state i) Ensemble Averages  m ij : expected time to reach j (for 1 st time), starting from i  m ii = expected time between successive visits to i  π i = (prob. of being at state i after many steps Theorem: for an ergodic DTMC (proof based on Renewal Theory) 23

24. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  π i p ij : rate of transitions from state i to state j  π i = percentage of time in state i  p ij = percentage of these times the chain moves next to j  Q: What is Σ j π j p ji ?  A: rate into state i 24 π0π0 p 00 p 01 π1π1 π2π2 p 10 p 21 p 12  From stationary equation: π i =Σ j π j p ji  π i : rate into i  But also π i =Σ j π i p ij (why?)  Theorem: Σ j π j p ji = Σ j π i p ij (rate in = rate out)  Q: why is this reasonable?  A: Cannot make a transition from i, without a transition into i before it (difference in number is at most 1)  Assume a subset of states S: rate into S = rate out of S rate into 1 π 2 p 21 rate out of 1 π 1 p 12 +π 1 p 10

25. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Assume there exist π i : π i p ij = π j p ji (for all i and j)  and Σ i π i = 1  Then:  π i is the stationary distribution for the chain  The above equations are called local balance equations  The Markov chain is called time-reversible  Solving for the Stationary Distribution of a DTMC 25 Stationary Eq. π i = Σ j π j p ji (not always easy) Global Balance Σ j ≠i π i p ij = Σ j ≠i π j p ji (a bit easier) Local Balance π i p ij = π j p ji (easiest! try first!)