1. 8/14/04J. Bard and J. W. Barnes Operations Research Models and Methods Copyright 2004 - All rights reserved Lecture 11 – Stochastic Processes Topics Definitions Review of probability Realization of a stochastic process Continuous vs. discrete systems Examples Classification scheme

2. 2 Basic Definitions Stochastic process: System that changes over time in an uncertain manner State: Snapshot of the system at some fixed point in time Transition: Movement from one state to another Examples Automated teller machine (ATM) Printed circuit board assembly operation Runway activity at airport

3. 3 Elements of Probability Theory Experiment: Any situation where the outcome is uncertain. Sample Space, S: All possible outcomes of an experiment (we will call it “state space”). Event: Any collection of outcomes (points) in the sample space. A collection of events E 1, E 2,…,E n is said to be mutually exclusive if E i  E j =  for all i ≠ j = 1,…,n. Random Variable: Function or procedure that assigns a real number to each outcome in the sample space. Cumulative Distribution Function (CDF), F(·): Probability distribution function for the random variable X such that F(a) = Pr{X ≤ a}.

4. 4 Model Components (continued) Time: Either continuous or discrete parameter. State:Describes the attributes of a system at some point in time. s = (s 1, s 2,..., s v ); for ATM example s = (n) Convenient to assign a unique nonnegative integer index to each possible value of the state vector. We call this X and require that for each s  X. For ATM example, X = n. In general, X t is a random variable.

5. 5 Activity:Takes some amount of time – duration. Culminates in an event. For ATM example  service completion. Transition:Caused by an event and results in movement from one state to another. For ATM example, Stochastic Process:A collection of random variables {X t }, where t  T = {0, 1, 2,...}.

6. 6 Markovian Property Given that the present state is known, the conditional probability of the next state is independent of the states prior to the present state. Present state at time t is i: X t = i Next state at time t + 1 is j: X t+1 = j Conditional Probability Statement of Markovian Property: Pr{X t+1 = j | X 0 = k 0, X 1 = k 1,…,X t = i} = Pr{X t+1 = j | X t = i} for t = 0, 1,…, and all possible sequences i, j, k 0, k 1,..., k t –1.

7. 7 Realization of the Process Deterministic Process Time between arrivals Pr{ t a   } = 0,  < 1 min = 1,   1 min Arrivals occur every minute. Time for servicing customer Pr{ t s   } = 0,  < 0.75 min = 1,   0.75 min Processing takes exactly 0.75 minutes. Number in system, n (no transient response)

8. 8 Realization of the Process (continued) Stochastic Process Time for servicing customer Pr{ t s   } = 0,  < 0.75 min = 0.6, 0.75    1.5 min = 1,   1.5 min Number in system, n

9. 9 Birth and Death Processes Pure Birth Process; e.g., Hurricanes Pure Death Process; e.g., Delivery of a truckload of parcels Birth-Death Process; e.g., Repair shop for taxi company

10. 10 Queueing Systems Queue Discipline:Order in which customers are served; FIFO, LIFO, Random, Priority Five Field Notation: Arrival distribution / Service distribution / Number of servers / Maximum number in the system / Number in the calling population

11. 11 Queueing Notation Distributions (interarrival and service times) M = Exponential D = Constant time E k = Erlang GI = General independent (arrivals only) G = General Parameters s = number of servers K = Maximum number in system N = Size of calling population

12. 12 Characteristics of Queues Infinite queue: e.g., Mail order company (GI/G/s) Finite queue: e.g., Airline reservation system (M/M/s/K) a. Customer arrives but then leaves b. No more arrivals after K

13. 13 Characteristics of Queues (continued) Finite input source: e.g., Repair shop for trucking firm (N vehicles) with s service bays and limited capacity parking lot (K – s spaces). Each repair takes 1 day (GI/D/s/K/N). In this diagram N = K so we have GI/D/s/K/K system.

14. 14 Examples of Stochastic Processes Service Completion Triggers an Arrival: e.g., multistage assembly process with single worker, no queue. state = 0, worker is idle state = k, worker is performing operation k = 1,..., 5

15. 15 Examples (continued) Multistage assembly process with single worker with queue. (Assume 3 stages only) s = (s 1, s 2 ) where { s 1 = number of parts in system s 2 = current operation being performed 3 d 2,1 0,0 1,13,1 a 1 d a a 2,21,23,2 a a 2,31,33,3 aa 2 d 1 d 1 d 2 d 2 d 3 d 3 d … … … Assume k = 1, 2, 3

16. 16 Queueing Model with Two Servers, One Operation s = (s 1, s 2, s 3 ) wheres i = { s 3 = number in queue 0 if server i is idle i = 1, 2 1 if server i is busy State- transition network

17. 17 Series System with No Queues ComponentNotationDefinition State s = (s 1, s 2, s 3 ) State space S = { (0,0,0), (1,0,0),..., (0,1,1), (1,1,1) } The state space consists of all possible binary vectors of 3 components. Activities Y = {a, d 1, d 2, d 3 }a =arrival at operation 1 d j =completion of operation j for j = 1, 2, 3 0 if server i is idle 1 if server i is busy for i = 1, 2, 3 s i = {

18. 18 Transitions for Markov Processes Exponential interarrival and service times (M/M/s) State space: S = {1, 2,...} Probability of going from state i to state j in one move: p ij Theoretical requirements: 0  p ij  1,  j p ij = 1, i = 1,…,m State-transition matrix P =

19. 19 Single Channel Queue – Two Kinds of Service Bank teller: normal service (d), travelers checks (c), idle (i) Letp = portion of customers who buy checks after normal service s 1 = number in system s 2 = status of teller, where s 2  {i, d, c} State- transition network

20. 20 Part Processing with Rework Consider a machining operation in which there is a 0.4 probability that upon completion, a processed part will not be within tolerance. Machine is in one of three states: 0 = idle, 1 = working on part for first time, 2 = reworking part. State-transition network a =arrival s 1 =service completion from state 1 s 2 =service completion from state 2

21. 21 Markov Chains A discrete state space Markovian property for transitions One-step transition probabilities, p ij, remain constant over time (stationary) Example: Game of Craps Roll 2 dice:Win = 7 or 11; Loose = 2, 3, 12; otherwise 4, 5, 6, 8, 9, 10 (called point) and roll again  win if point  loose if 7 otherwise roll again, and so on. (There are other possible bets not include here.)

22. 22 State-Transition Network for Craps

23. 23 Transition Matrix for Game of Craps

24. 24 State-Transition Network for Simple Markov Chain

25. 25 Classification of States Accessible:Possible to go from state i to state j (path exists in the network from i to j). Two states communicate if both are accessible from each other. A system is irreducible if all states communicate. State i is recurrent if the system will return to it after leaving some time in the future. If a state is not recurrent, it is transient.

26. 26 Classification of States (continued) A state is periodic if it can only return to itself after a fixed number of transitions greater than 1 (or multiple of a fixed number). A state that is not periodic is aperiodic. a.Each state visited every 3 iterations b.Each state visited in multiples of 3 iterations

27. 27 Classification of States (continued) An absorbing state is one that locks in the system once it enters. This diagram might represent the wealth of a gambler who begins with $2 and makes a series of wagers for $1 each. Let a i be the event of winning in state i and d i the event of losing in state i. There are two absorbing states: 0 and 4.

28. 28 Classification of States (continued) Class: set of states that communicate with each other. A class is either all recurrent or all transient and may be either all periodic or aperiodic. States in a transient class communicate only with each other so no arcs enter any of the corresponding nodes in the network diagram from outside the class. Arcs may leave, though, passing from a node in the class to one outside.

29. 29 Illustration of Concepts Example 1 Every pair of states communicates, forming a single recurrent class; however, the states are not periodic. Thus the stochastic process is aperiodic and irreducible.

30. 30 Illustration of Concepts Example 2 States 0 and 1 communicate and for a recurrent class. States 3 and 4 form separate transient classes. State 2 is an absorbing state and forms a recurrent class.

31. 31 Illustration of Concepts Example 3 Every state communicates with every other state, so we have irreducible stochastic process. Periodic?Yes, so Markov chain is irreducible and periodic.

32. 32 What you Should know about Stochastic Processes What a state is What a realization is (stationary vs. transient) What the difference is between a continuous and discrete-time system What the common applications are What a state-transition matrix is How systems are classified