Stochastic Models Lecture 3 Continuous-Time Markov Processes Nan Chen MSc Program in Financial Engineering The Chinese University of Hong Kong (ShenZhen) Oct 14, 2015

Outline Introduction of Continuous-Time Processes Limiting Probabilities

3.1 Introduction Use a section header for each of the topics, so there is a clear transition to the audience.

A New Perspective on Poisson Process A Poisson process can be constructed as follows: At each state it will stay for an exponential time with mean Then, it proceeds from to

Continuous-Time Markov Chains We may follow a similar way to construct a continuous-time Markov chains on a state space : Each time when it enters a state , we select a random sojourn time independently of its history; After a time it will exit the state . It will enter state with probability (Remark: We have a discrete-time Markov chain embedded in the process. )

Transition Probability Function Let record the state of the above Markov chain over time. The following quantity is known as the transition probability function of the process. Obviously, we have

Kolmogorov Backward/Forward Equations Theorem (Kolmogorov Backward Equation) For states and times where

Kolmogorov Backward/Forward Equations (Continued) Theorem (Kolmogorov Forward Equation) For states and times

Example I: Two-State Chain Consider a machine that works for an exponential amount of time having mean before breaking down; and suppose that it takes an exponential amount of time having mean to repair the machine. If the machine is in working condition at time 0, then what is the probability that it will be working at time Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

Example I: Two-State Chain From the backward equations, we have From them, we can solve for Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

Computing Transition Probabilities For any pair of states and let Then, we can rewrite the backward/forward equations as follows (backward) (forward)

Computing Transition Probabilities (Continued) Introduce matrices by letting the element in row , column of these matrices be, respectively, The two equations can be written as (backward) (forward) The solution should be

3.2 Limiting Probabilities Use a section header for each of the topics, so there is a clear transition to the audience.

Limiting Probabilities In analogy with a basic result in discrete-time Markov chains, the probability that a continuous-time Markov chain will be in state at time often converges to a limiting value that is independent of the initial state. We intend to study

Limiting Probabilities (Continued) To derive a set of equations for the , we may let approach in the forward equation. Then, In addition,

Limiting Probabilities (Continued) It is easy to see that if the embedded Markov chain has a limiting stationary distribution i.e., then The solution to the equations on the last slide should be

Example II: A Shoe Shine Shop Consider a shoe shine establishment consisting two chairs --- chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done, the customer moves on to chair 2 where the polish is buffed. The service times at the two chairs are independent and exponentially distributed with respective rates and

Example II: A Shoe Shine Shop (Continued) Suppose that potential customers arrive in accordance with a Poisson process having rate , and that a potential customer will enter the system only if both chairs are empty. Let us define the state of the system as follows: State 0: empty system State 1: chair 1 is taken State 2: chair 2 is taken Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

Example II: A Shoe Shine Shop (Continued) What is the limiting probability for this continuous-time Markov chain? Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

Homework Assignments Read Ross Chapter 6.1, 6.2, 6.4, 6.5, and 6.9. Answer Questions: Exercises 8 (Page 399, Ross) Exercises 10, 13 (Page 400, Ross) Exercises 14, 17 (Page 401, Ross) Exercise 20 (Page 402, Ross) (Optional, Extra Bonus) Exercise 48 (Page 407). Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.