1. Previously Optimization Probability Review Inventory Models Markov Decision Processes

2. Agenda Hwk Projects Markov Decision Processes Queues

3. Markov Decision Processes (MDP) States i=1,…,n Possible actions in each state Reward R(i,k) of doing action k in state i Law of motion: P(j | i,k) probability of moving i  j after doing action k

4. MDP f(i) = largest expected current + future profit if currently in state i f(i,k) = largest expected current+future profit if currently in state i, will do action k f(i) = max k f(i,k) f(i,k) = R(i,k) + ∑ j P(j|i,k) f(j) f(i) = max k [R(i,k) + ∑ j P(j|i,k) f(j)]

5. MDP as LP f(i) = max k [R(i,k) + ∑ j P(j|i,k) f(j)] Idea: f(i) decision variables piecewise linear function min ∑ j f(i) s.t. f(i) ≥ R(i,k) + ∑ j P(j|i,k) f(j) for all i,k

6. MDP Examples Breast cancer screening Stock options Airline ticket pricing Elevator scheduling Reservoir management

7. Queues (Ch 14) Queue = waiting line the “system” image from http://staff.um.edu.mt/jskl1/simweb/intro.htm

8. Examples Airport security Customer service line Checkout Doctor’s office ER Canada: scheduling operations Elevators

9. Performance Measures T time in system T q waiting time (time in queue) N #customers in system N q #customers in queue system arrivals departures queue servers W = E[T] W q = E[T q ] L = E[N] L q = E[N q ]  fraction of time servers are busy (utilization)

10. Randomness is Key arrivals every 15 min (not random) processing times random with mean of 13 min (exponential random variable) waiting time customer #