1. ST3236: Stochastic Process Tutorial 3 TA: Mar Choong Hock Email: g0301492@nus.edu.sg Exercises: 4

2. Question 1 A markov chain X 0,X 1,… on state 0, 1, 2 has the transition probability matrix and initial distributions, p 0 = P(X 0 = 0) = 0.3, p 1 = P(X 0 = 1) = 0.4 and p 2 = P(X 0 = 2) = 0.3. Determine P(X 0 = 0, X 1 = 1, X 2 = 2) and draw the state- diagram with transition probability.

3. Question 1 P(X 0 = 0,X 1 = 1,X 2 = 2) = P(X 0 = 0)P(X 1 = 1 | X 0 = 0)P(X 2 = 2 | X 0 = 0,X 1 = 1) = P(X 0 = 0)P(X 1 = 1 | X 0 = 0)P(X 2 = 2 | X 1 = 1) = p 0 x p 01 x p 12 = 0.3 x 0.2 x 0 = 0.

5. Question 2 A markov chain X 0,X 1,… on state 0, 1, 2 has the transition probability matrix Determine the conditional probabilities P(X 2 = 1,X 3 = 1|X 1 = 0) and P(X 1 = 1,X 2 = 1|X 0 = 0).

6. Question 2 P(X 2 = 1, X 3 = 1 | X 1 = 0) = P(X 2 = 1 | X 1 = 1)P(X 3 = 1 | X 1 = 0, X 2 = 1) = P(X 2 = 1 | X 1 = 0)P(X 3 = 1 | X 2 = 1) = p 01 x p 11 = 0.2 x 0.6 = 0.12 Similarly (or by stationarity), P(X 1 = 1, X 2 = 1 | X 0 = 0) = 0.12 In general, P(X n+1 = 1, X n+2 = 1 | X n = 0) = 0.12 for any n. That is, it doesn’t matter when you start.

7. Question 3 A markov chain X 0,X 1,… on state 0, 1, 2 has the transition probability matrix If we know that the process starts in state X 0 = 1, determine probability P(X 0 = 1,X 1 = 0,X 2 = 2).

8. Question 3 P(X 0 = 1,X 1 = 0,X 2 = 2) = P(X 0 = 1)P(X 1 = 0| X 0 = 1)P(X 2 = 2 | X 0 = 1,X1 = 0) = P(X 0 = 1)P(X 1 = 0| X 0 = 1)P(X 2 = 2 | X 1 = 0) = p 1 x p 10 x p 02 = 1 x 0.3 x 0.1 = 0.03

9. Question 4 A markov chain X 0,X 1,… on state 0, 1, 2 has the transition probability matrix

10. Question 4a Compute the two-step transition matrix P(2). Note: Observe that the rows must always sum to one for all transition matrices.

11. Question 4b What is P(X 3 = 1|X 1 = 0)? P(X 3 = 1|X 1 = 0) = 0.13 In general, P(X n+2 = 1 | X n = 0) = 0.13 for any n.

12. Question 4c What is P(X 3 = 1|X 0 = 0)? Note that: Thus, P(X 3 = 1|X 0 = 0) = 0.16 In general, P(X n+3 = 1 | X n = 0) = 0.16 for any n.

13. Question 5 A markov chain X 0,X 1,… on state 0, 1, 2 has the transition probability matrix It is known that the process starts in state X 0 = 1, determine probability P(X 2 = 2).

14. Question 5 Note that: P(X 2 = 2) = P(X 0 = 0) x P(X 2 = 2 | X 0 = 0) +P(X 0 = 1) x P(X 2 = 2 | X 0 = 1) +P(X 0 = 2) x P(X 2 = 2 | X 0 = 2) = p 0 p 02 + p 1 p 12 + p 2 p 22 = 1 x p 12 = 0.35

15. Question 6 Consider a sequence of items from a production process, with each item being graded as good or defective. Suppose that a good item is followed by another good item with probability  and by a defective item with probability 1- . Similarly, a defective item is followed by another defective item with probability  and by a good item with probability 1- . Specify the transition probability matrix. If the first item is good, what is the probability that the first defective item to appear is the fifth item?

16. Question 6 Let X n be the grade of the nth product. P(X n+1 = g | X n = g) = , P(X n+1 = d | X n = g) = 1 -  P(X n+1 = d | X n = d) = , P(X n+1 = g | X n = d) = 1 -  Thus, the transition probability matrix is

17. Question 6 The probability is, P(X 5 = d,X 4 = g,X 3 = g,X 2 = g | X 1 = g) = P(X 2 = g | X 1 = g) x P(X 3 = g | X 2 = g) x P(X 4 = g | X 3 = g) x P(X 5 = d | X 4 = g) (why?) = p gg x p gg x p gg x p gd =  3 (1 -  )

18. Question 7 The random variables  1,  2,... are independent and with common probability mass function Set X 0 = 0 and let X n = max {  1,  2,... }. Determine the transition probability matrix for the MC {X n }. Draw the state-diagram associated with transition probability

19. Question 7 Observe: X 0 = 0, X 1 = max {X 0,  1 }, X 2 = max {X 1,  2 }, … X n = max {X n-1,  n } Hence, X n recursively compares the previous maximum and the current input to obtain the new maximum.

20. Question 7 The state space is S = {0, 1, 2, 3} P(X n+1 = 0 | X n = 0) = P(  n+1 = 0)= 0.1 P(X n+1 = 1 | X n = 0) = P(  n+1 = 1)= 0.3 P(X n+1 = 2 | X n = 0) = 0.2 P(X n+1 = 3 | X n = 0) = 0.4 P(X n+1 = 1 | X n = 1) = P(  n+1 = 0) + P(  n+1 = 1) = 0.1 + 0.3 = 0.4 P(X n+1 = 2 | X n = 1) = 0.2 … P(X n+1 = 2 | X n = 2) = 0.1 + 0.3 + 0.2 = 0.6 … P(X n+1 = 3 | X n = 3) = 0.1 + 0.3 + 0.2 + 0.4 = 1 … P(X n+1 = j | X n = i) = 0 if j < i. (Cannot Happen!)

21. Question 7 The transition probability matrix is