ST3236: Stochastic Process Tutorial 4 TA: Mar Choong Hock Exercises: 5

Question 1 An urn initially contains a single red and single green ball. A ball is drawn at random, removed and replaced by a ball of the opposite color and this procedure repeats so that there are always two balls in the urn. Let X n be the number of red balls in the urn after n draws, with X 0 = 1. Specify the transitions probabilities for MC {X}.

Question 1 Case (X n =0): Both balls are green. One ball will certainly be replaced with red after a ball is drawn. P(X n+1 = 1 | X n = 0) = 1, P(X n+1 = 0 | X n = 0) = 0, P(X n+1 = 2 | X n = 0) = 0 Case (X n =2): Both balls are red. One ball will certainly be replaced with green after a ball is drawn. P(X n+1 = 1 | X n = 2) = 1, P(X n+1 = 0 | X n = 2) = 0, P(X n+1 = 2 | X n = 2) = 0

Question 1 Case (X n = 1): One ball is red. Outcome dependent on the colour of the ball drawn. Note: P(Green is drawn|X n =1) = P (Red is drawn|X n =1) = 0.5 P(X n+1 = 0 | X n = 1) = P(Red is drawn|X n = 1) = 0.5 P(X n+1 = 2 | X n = 1) = P(Green is drawn|X n = 1) = 0.5 P(X n+1 = 1 | X n = 1) = 0

Question 1 The transition matrix is given as follows:

Question 2 Find the mean time to reach state 3 starting from state 0 for the MC whose transition probability matrix is

Question 2 Let T = min{n : X n = 3} and v i = E(T | X 0 = i). The mean time to reach state 3 starting from state 0 is v 0. We apply first step analysis.

Question 2 Therefore, v 0 = v v v v 3 v 1 = 1 + 0v v v v 3 v 2 = 1 + 0v 0 + 0v v v 3 v 3 = 0 Solving the equations, we have v 0 = 10.

Question 3 Consider the MC with transition probability matrix (a) Starting in state 1, determine the probability that the MC ends in state 0 (b) Determine the mean time to absorption.

Question 3a Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). u 0 = 1 u 1 = 0.1u u u 2 u 2 = 0 we have u 1 = 0.25.

Question 3b Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). v 0 = 0 v 1 = v v v 2 v 2 = 0 we have v 1 = 2.5.

Question 4 Consider the MC with transition probability matrix (a) Starting in state 1, determine the probability that the MC ends in state 0 (b) Determine the mean time to absorption.

Question 4a Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). u 0 = 1 u 1 = 0.1u u u u 3 u 2 = 0.2u u u u 3 u 3 = 0 we have, u 1 = , u 2 = Starting in state 1, the probability that the MC ends in state 0 is u 1 =

Question 4b Let T = min {n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). v 0 = 0 v 1 = v v v v 3 v 2 = v v v v 3 v 3 = 0 we have v 1 = v 2 = 3.33.

Question 5 A coin is tossed repeatedly until two successive heads appear. Find the mean number of tosses required. [Hint: Let X n be the cumulative number of successive heads. Then the state space is 0,1,2 before stop]

Question 5 – Method 1 Let Y n be the outcome {H, T} of each toss and (Y n-1, Y n ) denotes the sample point for the sucessive tosses. There are 4 possible sample points.

Question 5 – Method 1 The transition probability matrix is

Question 5 – Method 1 Let v i denote the mean time to each state 3 starting from state i. By the first step analysis, we have the following equations: v 0 = v v 1 v 1 = v v 3 v 2 = v v 1 v 3 = 0 Therefore, v 0 = 6 (v 1 = 4, v 2 = 6)

Question 5 – Method 2 Let X n be the cumulative number of successive heads. The 3-state state space now is {0,1,2}. Example: n01234 YnYn THTHH XnXn 01012

Question 5 – Method 2 Case (X n = 0) :Previous two tosses are tails Or a head followed by tail. Note: P(Head) = P(Tail) = 0.5 P(X n+1 = 1 | X n = 0) = P(Head) = 0.5, P(X n+1 = 0 | X n = 0) = P(Tail) = 0.5, P(X n+1 = 2 | X n = 0) = 0 Case (X n = 1): A tail is followed by a head P(X n+1 = 2 | X n = 1) = P(Head) = 0.5 P(X n+1 = 0 | X n = 1) = P(Tail) = 0.5 P(X n+1 = 1 | X n = 1) = 0

Question 5 – Method 2 Case (X n = 2): Previous two tosses are heads. We make this state absorbing. P(X n+1 = 1 | X n = 2) = 0 P(X n+1 = 2 | X n = 2) = 1, P(X n+1 = 0 | X n = 2) = 0

Question 5 – Method 2 The transition probability matrix is

Question 5 – Method 2 Let v i denote the mean time to each state 2 starting from state i. By the first step analysis, we have the following equations: v 0 = v v 1 v 1 = v v 2 v 2 = 0 Thus, we have, v 0 = 6, v 1 = 4