1. Matrix Analytic methods in Markov Modelling

2. Continous Time Markov Models X: R -> X µ Z (integers) X(t): state at time t X: state space (discrete – countable) R: real numbers (continuous time line)

3. Continous Time Markov Models X is piecewise constant X is typically cadlag ("continue à droite, limite à gauche") = RCLL (“right continuous with left limits”)

4. Transition probabilities P(X(t+h)=j|X(t)=i) ¼ h ¸ ij for i  j P(X(t+h)=j|X(t)=j) = 1-  i  j P(X(t+h)=i|X(t)=j) ¼ 1- h  i  j ¸ ji P(X(t+h)=j)=  i P(X(t+h)=j and X(h)=i) =  i P(X(t+h)=j | X(h)=i) P(X(h)=i) =  i  j ¸ ij h P(X(h)=i) + h(1-  i  j ¸ ji ) P(X(h)=j)

5. Transition probabilities P(X(t+h)=j)=  i P(X(t+h)=j and X(h)=i) + (1-  k  j ¸ ik ) P(X(h)=j) P(t)=[P(X(t)=0) P(X(t)=1) P(X(t)=2)..] P(t+h) ¼ P(t)H H=I+hQ Q ij = ¸ ij Q jj = 1-  i  j ¸ ji

6. Taking limits P(t+h) ¼ P(t)H H=I+hQ P(t+h) ¼ P(t)(I+hQ)=P(t)+hP(t)Q (P(t+h)-P(t))/h ¼ P(t)Q d/dt P(t) = P(t)Q P(t)=P(0)exp(Qt)

7. Irreducibility X is irreducible when states are mutually reachable. X is irreducible iff for every i,j 2 X there is a sequence {i(1),i(2),..,I(N) 2 X} such that i(1)=i i(N)=j and ¸ i(k),i(k+1) > 0 for every k 2 1,..,N-1

8. Recurrence Assume X(t n - )=j and X(t n )=i  j then t n is a transition time Let {t n } be the sequence of consequetive transition times. {Xn=X(t n )} is called the embedded chain Let t 0 =0, X 0 =i then k(i)=inf{k>0: X(k)=i}=inf{k>1: X(k)=i} T i =t k(i) T i is the time to next visit at i X is recurrent if P(T i < 1 )=1 for all i (almost certain return to all states) X is positive recurrent if E(T i ) · 1 for all i

9. Stationary probability d/dt P(t) = P(t)Q P(t)=P(0)exp(Qt) When X is irreducible and positive recurrent there is a unique probability vector ¦ such that P(t) -> ¦ ¦ solves ¦ Q=0

10. Stationary probability Ergodicity: ¦ i = E(D i )/E(T i ) ¦ i is the fraction of the time in state i Statistically intuitively appealing X(t) ==i TiTi DiDi time

11. Example Poisson Counting Process Q i,i+1 = ¸ ¸¸¸¸ 1023 Counts Poisson events Birth Chain Not irreducible Not recurrent

12. Example Birth/Death(BD)-chain Q i,i+1 = ¸ Q i+1,i = ¹ ¸¸¸¸ 1023 Models a queueing system with Poisson arrival process and independent exponentially distributed service times ¸ is arrival rate ¹ is service rate Irreducible Positive recurrent for ¸ < ¹ ¹¹¹¹

13. Example Birth/Death(BD)-chain ¸¸¸¸ 1023 ½ = ¸ / ¹ ¦ n = ½¦ n-1 ¦ n = ½ n ¦ 0 P 0 =1/  n=0 1 ½ n =1- ½ ¦ n = ½ n (1- ½ ) E(X) =  n=0 1 n ¦ n = ½ /(1- ½ ) ¹¹¹¹

14. Example Birth/Death(BD)-chain ¸1¸2¸3¸4 1023 ½ n = ¸ n / ¹ n ¦ n = ½ n ¦ n-1 ¦ n = ¦ i=1 n ½ i ¦ 0 P 0 =1/  n=0 1 ¦ i=1 n ½ i ¹1¹2¹3¹4

15. Markov Modulated Poisson Process Has two modes: Modes={ON,OFF} M(t) 2 Modes is a two state CTMC Transmits with rate ¸ in ON mode. Counting proces combines state spaces, i.e. X = Modes £ {0,1,2,..} Q (ON,i),(ON,i+1) = ¸ Q (ON,i),(OFF,i) = ¯ Q (OFF,i),(ON,i) = ® Q i,j =0 otherwise

16. Markov Modulated Poisson Process Q (ON,i),(ON,i+1) = ¸ Q (ON,i),(OFF,i) = ¯ Q (OFF,i),(ON,i) = ® Q i,j =0 otherwise 1OFF23 ¸¸¸¸ 1023 0 ON ® ¯®®®¯¯¯

17. MMPP with exponential service Q (ON,i),(ON,i+1) = ¸ Q (ON,i),(OFF,i) = ¯ Q (OFF,i),(ON,i) = ® Q (ON,i),(ON,i-1) = ¹ Q (OFF,i),(OFF,i-1) = ¹ Q i,j =0 otherwise 1OFF23 ¸¸¸¸ 1023 0 ON ® ¯®®®¯¯¯ ¹¹¹¹ ¹¹¹¹

18. State ordering For a state (i,M) we denote i the level of the state We order states so that equal levels are gathered (i,OFF),(i,ON)(i+1,OFF)(i+1,ON)(i+2,OFF)(i+2,ON)

19. Generator matrix

20. Sub matrices

21. Generator matrix by submatrices Balance equations: P 0 A 0 + P 1 B = 0 eq(0) P 0 C + P 1 A + P 2 B = 0 eq(1) P i C + P i+1 A + P i+2 B = 0 eq(i+1) We look for a matrix geometric solution, i.e. P 1 =P 0 R P i+1 = P i R Inserting in eq(0): P 0 A 0 + P 0 R B = 0 and eq(i+1) P i (C+R A + R 2 B)=0 for all P i

22. Solving for R P i (C+R A + R 2 B)=0 for all P i Sufficient that C+R A + R 2 B=0 (Ricatti equation) Iterative solution R 0 =0 repeat R n+1 =-(C+R n 2 B) A -1 Converges for irreducible positive recurrent Q

23. MM - service Q (ON,i),(ON,i+1) = ¸ Q (OFF,i),(OFF,i+1) = ¸ Q (ON,i),(OFF,i) = ¯ Q (OFF,i),(ON,i) = ® Q (ON,i),(ON,i-1) = ¹ Q (OFF,i),(OFF,i-1) =0 Q i,j =0 otherwise 1OFF23 ¸¸¸¸ 1023 0 ON ® ¯®®®¯¯¯ ¹¹¹¹ ¸ ¸¸¸

24. Generator matrix

25. Sub matrices

26. Generally We still look for a matrix geometric solution: Now:  i=0 1 R i A i = 0  A 0 + R A 1  i=2 1 R i A i Iteration: R n = -A 1 -1 (A 0 +  i=2 1 R n-1 i A i ) Solving for P 0 : P 0  i=0 1 R i B i = 0 Conditions for solution: Irreducibility and pos. recurrence

27. Miniproject (i) Let traffic be generated by an on/off Markov process with on rate: ¸ =1, mean rate 0.1 ¸, average on time: T=1 Let service be exponential with rate ¹ = 0.2 ¸ Construct the generator matrix for the data given above. Use the iterative algoritme to solve for the load matrix R Solve for P 0 and P i Compute the mean queue length Compare with M/M/1 results

28. Miniproject (ii) Collect file size or web session duration data Check for power tails and estimate tail power Find appropriate parameters for a hyperexponential approximation of the reliability estimated reliability Construct the generator matrix of an equivalent ME/M/1 queue