15 October 2012 Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / Discrete Time Markov Chains

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  X(n): the state/value of a process at the n -th period (time slot)  X(n) is random and takes values in a finite or countable set  The sequence X(1), X(2), …, X(n) could be a) The values of a stock each day b) The web page a user is currently browsing c) The Access Point(AP) a moving user is currently associated with  We would like a probabilistic model for X(1),X(2),…,X(n) a) X(i) are independent => not so realistic (for above examples) b) X(i+1) depends on X(i) => not a bad model to start with 2 P{X(n+1) = j | X(n) = i,X(n-1) = i n-1,…,X(1) = X 0 } = P{X(n+1) = j | X(n) = i} = p ij (Markov Property)

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Assume state space: 0, 1, 2, …  P ij : P{S(n+1) = j | S(n) = i}   Stationary: P{S(n+1) = j | S(n) = i} = P{S(1) = j | S(0) = i}  for any n  p ij define a transition matrix P = 3

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Simple weather model  X(n): weather after n days  Simple channel error model  Prob of error = 0.1  Case 1: q = 0.1  Case 2: q = 0.9  What is the transition matrix P in both cases? 4 goodbad p 1-p q 1-q (uncorrelated) (burst of errors)

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  A simple handover model  cell -> state  need to find transition probabilities  depend on road structure, user profile, statistics 5 user on the phone Cellular Network Markov Chain

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  So far we know probabilities for next step: S(n)->S(n+1)  Probabilities for S(1) -> S(n)?  If P (2) = {p ij (2) } denotes the 2-step trans. probability matrix, then P (2) = P P  P ij (2) : multiply row I with column j 6

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Generalizing for n steps:  Kolmogorov-Chapman equations  P (n) = PP…P = (P) n (n-step transition probabilities)  P (n) = P (m) P (l)  e.g. P (5) = P (4) P (1) = P (2) P (3) = P (2) P P (2) 7

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Your iPhone browser can pre-fetch pages to improve speed  The current Web Page being browsed has 3 links  Study of click statistics have shown that  A user clicks link 1 next with prob 0.8  Link 2 with prob 0.1  Link 3 with prob 0.1  It seems reasonable that the browser pre-fetches link 1  Assume now a tiny subset of the Web with the following transition matrix  Start at page A Q: Which 2 web pages should the browser pre-fetch? 8 S (1) = [0.2, 0.2, 0.5, 0.1] S (2) = [0.13, 0.29, 0.24, 0.34] S (0) = [1, 0, 0, 0]

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  What happens to p ij (n) as n goes to infinity?  What is the lim P (n) as n->∞?  Q: Does it always converge?? (we’ll see this later)  If limit exists, then for any initial state i  π = {π 0, π 1,…, π m } is called the stationary distribution IF π P = π and Σ i π i = 1  The above equation can be used to find π 9

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  The weather system  Data rates supported for outdoor user  e.g. (128Kbps, 320Kbps, 1 Mbps)  What about this one? 10

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis (Reachability) State j is reachable by i  P ij (n) > 0 for some n  denote i  j (Communication) states i and j communicate (i j) if i  j and j  i (Recurrence/Transience) Denote f i the probability to ever return to i, starting from i  Transient) state is transient if f i < 1  number of returns is geometric (f i ): why?  Recurrent) state i is recurrent if f i = 1  Positive recurrent) expected time between visits to i is finite  Null recurrent) expected time between visits is infinite (strange!?)  Periodicity) state i is periodic, if exists d such that p ii (n) = 0, if n ≠ kd  Largest d with above property is called period 11

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Definition: A subset of states S: {for all i,j in S => i j} is called a class Lemma 1: recurrence is a class property  if i is recurrent and i j, then j is recurrent  Proof (by contradiction): if j transient => can never return to j after some time => cannot be at i either (since there is always a chance to go from i to j) Lemma 2: transience is a class property  Similar argument  Periodicity and positive/null-recurrence are also class properties Irreducible) A Markov Chain is irreducible if it has only 1 class  A finite MC which is irreducible is always positive-recurrent 12 Theorem: If a DTMC is aperiodic, irreducible and positive recurrent (“ergodic”) then it has a stationary distribution π = {π 1,…,π N }

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Search for “Network Modeling”  1000s of pages (courses, companies, etc.) contain these keywords  Goal: Make the page I’m interested appear in at least the top 10 pages shown Q: How should pages be ranked? A1: A page is important if many links to this page Q: What is wrong with this metric? A: - link from Yahoo more important than link from /~spyropou - can easily “fool” this! (create many dummy pages pointing to mine) Q: How to fix this? A: weigh a link from x to page p with the number of links into x Q: Can this system be fooled? A: Yes! Create 1000 dummy pages pointing to each other and mine 13

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  A page p has high rank (is important) if the pages pointing to it also have high rank Q: How is this different from before? A: All the dummy pages pointing to each other would still be low rank, if no “external” link from a high rank page Q: Is a link from a page with 1000 other links as important as a link from a page with 5 other links? A: No! If page i has a link to j and k total outgoing links  P ij = 1/k Q: Where does this lead? How do we calculate the rank? A: rank of page j = Σ i { rank of page i * Prob of link i  j} i.e. r j = Σ i r i P ij  (basis of) Page Rank Algorithm: rank of pages = stationary prob. of an MC with transitions P ij 14

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Q: What about this tiny WWW? A: π A = π N = 2/5, π M = 1/5 Q: But what about these two? A: These two chains are reducible and do not have a stationary distr. 15  Google’s heuristic: introduce additional arrows to avoid such loops and dead ends (not necessarily optimal!) Q: How does Google solve the huge system of eq (million pages)?

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis 16 [Network] m nodes on a common wire  Or wireless channel  Time is slotted [New Packets] Each node transmits a new packet with probability p (p < 1/m) [Success] If exactly 1 packet is transmitted [Collision] If k>1 (re-)transmissions during a slot  Every collided message is stored, to be resent later [Retransmission] Each collided(back-logged) message is retransmitted with prob q new packet old packet p p p √ X q Does Aloha work??

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Q: What should we take as the state of the chain X n ?  A: The number of backlogged messages (at slot n)  Transition probabilities from state 0 (no backlogged msg) P 00 = P 01 = P 0k = 17 (0 or 1 node transmits) 0 (not possible) if k ≤ m (any k of the m nodes transmit) 0 if k > m

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis P k,k-1 = (better) P k,k = P k,k+1 =(worse) P k,k+r = (1< r ≤ m)(worse) P k,k+r = 0(r > m)  Assume we are now at state k (i.e. k messages backlogged) 18 (0 new, 1 old) (0 new, 0 old)(1 new, 0 old) (r new, any old) (0 new, ≥2 old) (1 new) (>0 old)

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis 0 p 00 p p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : :  How can we tell if “Aloha works”?  Assume 10 nodes and transmission probability p = 0.05  Load = 0.5 Capacity  Intuition: (necessary) q should be small (re-Tx)  To ensure retransmissions don’t overload medium  Let’s assume q = (10 times smaller than p)  Q: Is Aloha Stable? If backlog becomes infinite => delay goes to infinity! 19

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  When at state k:  P back (k) : reduce backlog  P fwd (k) : increase backlog (MHB, Ch.10) (Why?)  So??? 20 0 p 00 p p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  For large enough k => states k+1, k+2, …, are transient  Markov Chain is transient => Aloha protocol is unstable!  Q: would Aloha work if we make q really (REALLY) small?  A: No!  Intuition: Let E[N] the expected Tx at state k  If E[N] ≥ 1 then situation either stays the same or worse  But E[N] = mp + kq --- what happens if k  ∞? 21 0 p 00 p p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Q: How can we fix the problem and make the chain ergodic (and the system stable)? A1: E[N] = mp + kq q < (1-mp)/k i.e. q = f(backlog) A2: or be more aggressive  geometric backoff q = a/k n  a < (1-mp) A3: or even exponential q = β -k  β > 1 Exponential backoff is the basis behind Ethernet Q: Why should q not be too small? A: Retransmission delay goes to infinity 22

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Time Averages  N i (t) = number of times in state i by time t  p i = (percentage of time in state i) Ensemble Averages  m ij : expected time to reach j (for 1 st time), starting from i  m ii = expected time between successive visits to i  π i = (prob. of being at state i after many steps Theorem: for an ergodic DTMC (proof based on Renewal Theory) 23

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  π i p ij : rate of transitions from state i to state j  π i = percentage of time in state i  p ij = percentage of these times the chain moves next to j  Q: What is Σ j π j p ji ?  A: rate into state i 24 π0π0 p 00 p 01 π1π1 π2π2 p 10 p 21 p 12  From stationary equation: π i =Σ j π j p ji  π i : rate into i  But also π i =Σ j π i p ij (why?)  Theorem: Σ j π j p ji = Σ j π i p ij (rate in = rate out)  Q: why is this reasonable?  A: Cannot make a transition from i, without a transition into i before it (difference in number is at most 1)  Assume a subset of states S: rate into S = rate out of S rate into 1 π 2 p 21 rate out of 1 π 1 p 12 +π 1 p 10

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Assume there exist π i : π i p ij = π j p ji (for all i and j)  and Σ i π i = 1  Then:  π i is the stationary distribution for the chain  The above equations are called local balance equations  The Markov chain is called time-reversible  Solving for the Stationary Distribution of a DTMC 25 Stationary Eq. π i = Σ j π j p ji (not always easy) Global Balance Σ j ≠i π i p ij = Σ j ≠i π j p ji (a bit easier) Local Balance π i p ij = π j p ji (easiest! try first!)

15 October 2012 Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / Absorbing Markov Chains 26

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  A mouse is trapped in the above maze with 3 rooms and 1 exit  When inside a room with x doors, it chooses any of them with equal probability (1/x) Q: How long will it take it on average to exit the maze, if it starts at room i? Q: How long if it starts from a random room? exit

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Def: T i = expected time to leave maze, starting from room I  T 2 = 1/3*1 + 1/3*(1+T 1 )+1/3*(1+T 3 ) = 1 + 1/3*(T 1 +T 3 )  T 1 = 1 + T 2  T 3 = 1 + T 2  T 2 = 5, T 3 = 6, T 1 = 6 Q: Could you have guessed it directly? A: times room 2 is visited before exiting is geometric(1/3)  on average, the wrong exit will be taken twice (each time costing two steps) and the 3 rd time the mouse exits exit

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  A packet must be routed towards the destination over the above network  “Hot Potato Routing” works as follows: when a router receives a packet, it picks any of its outgoing links randomly (including the incoming link) and send the packet immediately. Q: How long does it take to deliver the packet? 29 destination

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  First Step Analysis: We can still apply it!  But it’s a bit more complicated: 9x9 system of linear equations  Not easy to guess solution either!  We’ll try to model this with a Markov Chain 30 destination

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  9 transient states: 1-9  1 absorbing state: A Q: Is this chain irreducible? A: No! Q: Hot Potato Routing Delay  expected time to asborption? A 1 1/4 1/2 1/3 1/2 1/4 1/3 1/2 1/3 1/2 1/4 1 1/3

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  We can define transition matrix P (10x10) Q: What is P (n) as n  ∞? A: every row converges to [0,0,…,1] Q: How can we get ET iA ?  (expected time to absorption starting from i) Q: How about ? A: No, the sum goes to infinity! A 1 1/4 1/2 1/3 1/2 1/4 1/3 1/2 1/3 1/2 1/4 1 1/3

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Transition matrix can be written in canonical form  Transient states written first, followed by absorbing ones  Calculate P (n) using canonical form Q: Q n as n  ∞? A: it goes to O Q: where does the (*) part of the matrix converge to if only one absorbing state? A: to a vector of all 1s 33

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Theorem: The matrix (I-Q) has an inverse  N = (I-Q) -1 is called the fundamental matrix  N = I + Q + Q 2 + …  n ik : the expected number of times the chain is in state k, starting from state i, before being absorbed Proof: 34

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Theorem:  Let T i be the expected number of steps before the chain is absorbed, given that the chain starts in state i,  let T be the column vector whose i th entry is T i. then T = Nc,  where c is a column vector all of whose entries are 1 Proof:  Σ k n ik :add all entries in the i th row of N  expected number of times in any of the transient states for a given starting state i  the expected time required before being absorbed.  T i = Σ k n ik. 35

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis Theorem:  b ij :probability that an absorbing chain will be absorbed in (absorbing) state j, if it starts in (transient) state i.  B: (t-by-r) matrix with entries b ij. then B = NR,  R as in the canonical form. Proof: 36

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  Use Matlab to get matrices  Matrix N =  Vector T =

Thrasyvoulos Spyropoulos / Eurecom, Sophia-Antipolis  A wireless path consisting of H hops (links)  link success probability p  A packet is (re-)transmitted up to M times on each link  If it fails, it gets retransmitted from the source (end-to-end) Q: How many transmissions until end-to-end success? 38