1. Stochastic Models Lecture 3 Continuous-Time Markov Processes
Nan Chen
MSc Program in Financial Engineering
The Chinese University of Hong Kong (ShenZhen)
Oct 14, 2015

2. Outline Introduction of Continuous-Time Processes
Limiting Probabilities

3. 3.1 Introduction
Use a section header for each of the topics, so there is a clear transition to the audience.

4. A New Perspective on Poisson Process
A Poisson process can be constructed as follows:
At each state it will stay for an exponential time with mean
Then, it proceeds from to

5. Continuous-Time Markov Chains
We may follow a similar way to construct a continuous-time Markov chains on a state space :
Each time when it enters a state , we select a random sojourn time independently of its history;
After a time it will exit the state . It will enter state with probability
(Remark: We have a discrete-time Markov chain embedded in the process. )

6. Transition Probability Function
Let record the state of the above Markov chain over time. The following quantity
is known as the transition probability function of the process.
Obviously, we have

7. Kolmogorov Backward/Forward Equations
Theorem (Kolmogorov Backward Equation)
For states and times
where

8. Kolmogorov Backward/Forward Equations (Continued)
Theorem (Kolmogorov Forward Equation)
For states and times

9. Example I: Two-State Chain
Consider a machine that works for an exponential amount of time having mean before breaking down; and suppose that it takes an exponential amount of time having mean to repair the machine.
If the machine is in working condition at time 0, then what is the probability that it will be working at time
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.
Introduce each of the major topics.
To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

10. Example I: Two-State Chain
From the backward equations, we have
From them, we can solve for
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.
Introduce each of the major topics.
To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

11. Computing Transition Probabilities
For any pair of states and let
Then, we can rewrite the backward/forward equations as follows
(backward)
(forward)

12. Computing Transition Probabilities (Continued)
Introduce matrices by letting the element in row , column of these matrices be, respectively,
The two equations can be written as
(backward)
(forward)
The solution should be

13. 3.2 Limiting Probabilities
Use a section header for each of the topics, so there is a clear transition to the audience.

14. Limiting Probabilities
In analogy with a basic result in discrete-time Markov chains, the probability that a continuous-time Markov chain will be in state at time often converges to a limiting value that is independent of the initial state.
We intend to study

15. Limiting Probabilities (Continued)
To derive a set of equations for the , we may let approach in the forward equation. Then,
In addition,

16. Limiting Probabilities (Continued)
It is easy to see that if the embedded Markov chain has a limiting stationary distribution
i.e.,
then
The solution to the equations on the last slide should be

17. Example II: A Shoe Shine Shop
Consider a shoe shine establishment consisting two chairs --- chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done, the customer moves on to chair 2 where the polish is buffed.
The service times at the two chairs are independent and exponentially distributed with respective rates and

18. Example II: A Shoe Shine Shop (Continued)
Suppose that potential customers arrive in accordance with a Poisson process having rate , and that a potential customer will enter the system only if both chairs are empty.
Let us define the state of the system as follows:
State 0: empty system
State 1: chair 1 is taken
State 2: chair 2 is taken
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.
Introduce each of the major topics.
To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

19. Example II: A Shoe Shine Shop (Continued)
What is the limiting probability for this continuous-time Markov chain?
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.
Introduce each of the major topics.
To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.

20. Homework Assignments Read Ross Chapter 6.1, 6.2, 6.4, 6.5, and 6.9.
Answer Questions:
Exercises 8 (Page 399, Ross)
Exercises 10, 13 (Page 400, Ross)
Exercises 14, 17 (Page 401, Ross)
Exercise 20 (Page 402, Ross)
(Optional, Extra Bonus) Exercise 48 (Page 407).
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.
Introduce each of the major topics.
To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.