Adaptive Ambulance Redeployment via Multi-armed Bandits Ümitcan Şahin, Veysel Yücesoy, Aykut Koç, Cem Tekin Aselsan Research Center 30th June, 2018

Ambulance Redeployment Setup Proposed Method: via Multi-armed Bandits Overview Motivation Problem Definition Related Work Ambulance Redeployment Setup Proposed Method: via Multi-armed Bandits ARP Formulation Risk Aversion Results

Motivation In Turkey, seperate telephone numbers for emergency services 112 -> Ambulance 155 -> Police 110 -> Fire department etc. A need for a common emergency number system such as 911 A project launced by the Ministry of Health to combine all emergency services under one roof ASELSAN provides the setup for the emergency medical services in Turkey.

Problem Definition Accident occurs Closest idle ambulance is sent Ambulance arrives at accident scene Patient may need transportation How to position ambulances in order to: minimize the arrival times, increase the coverage of demand points. Methods in the literature: Static allocation Dynamic redeployment Learning based

Related Work Static allocation: Each ambulance is sent back to its predetermined base whenever it becomes idle + Computational efficiency and model simplicity - Does not accommodate to dynamically changing parameters such as call distributions, geographics (e.g., road conditions, traffic level, accidents, etc.) Static allocation models: Deterministic: Location set covering model (LSCM) [1] Maximal covering location problem (MCLP) [2] Stochastic: Maximum expected covering location problem (MEXCLP) [3]

Related Work Dynamic deployment: Redeploy ambulances periodically by considering current realizations (e.g., locations of accidents, number of idle ambulances, etc.) + Practical and applicable advanced models - Hard to find the optimal solution space in most cases Dynamic Redeployment Models: Approximate dynamic programming (e.g. Markov Decision Processes) Dynamic adaptations of MEXCLP: Dynamic MEXCLP ADVANTAGES: 24% performance increase in EMS system of Alberta, Canada [4] 33% more calls responded under 10 min. from 2015 to 2016, in Utrecht, the Netherlands [5]

Related Work Dynamic MEXCLP [5]: Given: Optimize: Location of bases (i.e. ambulance waiting locations) Number of available ambulances Expected demand at every location Driving times between locations Optimize: Distribution of idle ambulances over the bases in order to minimize fraction arrivals later than a certain threshold time (patient friendly) Ambulances relocate ONCE it becomes idle (crew friendly)

Related Work Dynamic MEXCLP [5]: Given: Optimize: Location of bases (i.e. ambulance waiting locations) Number of available ambulances Expected demand at every location Driving times between locations Optimize: Distribution of idle ambulances over the bases in order to minimize fraction arrivals later than a certain threshold time (patient friendly) Ambulances relocate ONCE it becomes idle (crew friendly)

Ambulance Redeployment Setup 𝑑 𝑖,𝑗 : Distance between from node i to node j 𝑥 𝑖, 𝑗 𝑡 : Context from node i to node j (context can be any information: traffic states, weather, accidents, etc.)

Proposed Method: Multi-armed Bandits Gambler plays one of the arms of 𝐾 slot machines sequentially. In round 𝑡 he bases its decision to play the arm 𝑎 𝑡 on his past observations. After playing 𝑎 𝑡 , he receives a reward from an unknown distribution 𝑟 𝑡 ~𝐹 𝑎 𝑡 . He can only observe the rewards of the arms he chooses to play. OBJECTIVE: Maximizing the long-term expected total reward max 𝐸 𝑡=1 𝑇 𝑟 𝑡

Proposed Method: Multi-armed Bandits Trade-off between exploration and exploitation: Exploration Exploitation Random Action Best Action Learn more about the best action Take the best action learned so far

Proposed Method: Multi-armed Bandits Challenges in ARP 1- Unknown expected demand at every location Learning-based approach rather than optimization Problem of partial observations Ex: Upper confidence bounds, Thompson sampling, 𝜖-greedy 2- Unknown and stochastic driving times Model traffic states on the roads as a Markov process Time-dependent travel times

ARP Formulation Regret for single ambulance redeployment in T rounds: 𝜇 ∗ : arrival times to calls from the best possible base location (known only by an oracle) 𝜋 𝑡 : location selected by MAB algorithm 𝜋 at round 𝑡 𝑟 𝑡, 𝜋 𝑡 : arrival time to call at round 𝑡 from 𝜋 𝑡 ORACLE: Computable and optimal OBJECTIVE: min 𝜋∈Π 𝑅(𝑇)

ARP Formulation Regret for multiple ambulance redeployment in T rounds: 𝑅 𝑚 𝑇 =E 𝑡=1 𝑇 𝑟 𝑡, 𝜋 𝑡,𝑛(𝑡) −E 𝑡=1 𝑇 𝑟 𝑡, 𝜋 𝑡,𝑛(𝑡) ∗ 𝑛 𝑡 : number of available ambulances at round t 𝜋 𝑡,𝑛(𝑡) ∗ : closest ambulance to call dispatched by an oracle (MEXCLP) that knows the true call distributions and traffic states on the roads 𝜋 𝑡,𝑛(𝑡) : closest ambulance to call dispatched by MAB algorithm at round 𝑡 𝑟 𝑡, 𝜋 𝑡,𝑛(𝑡) : arrival time of the closest ambulance to call at round 𝑡 from 𝜋 𝑡,𝑛(𝑡) ORACLE: NP-hard (combinatorial problem) => regret can only be defined against an oracle which might not be optimal for ARP OBJECTIVE: min 𝜋∈Π 𝑅 𝑚 (𝑇)

Challenges in ARP (Continued) Risk Aversion in ARP Challenges in ARP (Continued) 3- Is minimizing only the expected value of the arrival times enough? Guarantee that each call is to be responded as quickly as possible Minimize the arrival times under worst-case scenarios What happens to the expected arrival times under these guarantees then?

Mean-variance (MV) metric: [6] 𝑀 𝑉 𝑖 = 𝜎 𝑖 2 −𝜌 𝜇 𝑖 Risk Aversion in ARP Mean-variance (MV) metric: [6] 𝑀 𝑉 𝑖 = 𝜎 𝑖 2 −𝜌 𝜇 𝑖 𝜌: risk coefficient 𝜎: standard deviation of reward 𝜇: expected (average) value of reward Modification made for ARP: 𝑀 𝑉 𝑖 = 𝜎 𝑖 2 +𝜌 𝜇 𝑖 Rewards are simply the negative of the arrival times Objective: (1) Minimizing both the variance and (2) the expected value of the arrival times Is there a trade-off between (1) and (2)?

Challenges in ARP (Continued) Risk Aversion in ARP Challenges in ARP (Continued) 4 – Unknown expected demand at every location Unknown 𝜎 and 𝜇 parameters in the MV metric 5 – Only arrival time of the closest ambulance can be observed (i.e., the problem of partial feedback) Distinct 𝜎 𝑖 and 𝜇 𝑖 must be estimated for each bandit arm 𝑖

Risk Aversion in ARP Flow of the algorithm: Beginning: Place ambulance in each location to initiate 𝑛 𝑖 (number of times an ambulance is placed at location 𝑖), 𝑟 𝑖 (list of arrival times from location 𝑖 to calls) While there are idle ambulances: In each round 𝑡, compute the estimations 𝑢 𝑖 , 𝜎 𝑖 using 𝑛 𝑖 and 𝑟 𝑖 up to round 𝑡 Compute 𝑀 𝑉 𝑖 terms using 𝑢 𝑖 and 𝜎 𝑖 for each base location 𝑖, then compute the LCB terms: 𝐿𝐶 𝐵 𝑖 =𝑀 𝑉 𝑖 − 2 𝑙𝑜𝑔 𝑡 𝑛 𝑖 Select the location 𝑖 ∗ =𝑎𝑟𝑔𝑚𝑖 𝑛 𝑖 𝐿𝐶 𝐵 𝑖 for ambulance redeployment Exclude 𝑖 ∗ from the possible base locations, update 𝑛 𝑖 ∗ Dispatch closest ambulances to calls after round 𝑡 and update 𝑟 𝑖 for each dispacthed ambulance until next bandit update (exploration term)

Results – Setup Parameters 625 nodes 20-40 ambulance base locations 4 different equal time intervals in a day 5000 call capacity in a week (Poisson process with 𝜆=2) Time-dependent travel times w.r.t. Markov processes

Results – Traffic Consideration Non-existing traffic on the roads (deterministic driving times) Existing traffic on the roads (stochastic driving times) Response times confidence intervals of the algorithms when 𝑁=20

Results – Risk Aversion True call distributions Taking less risk Standard UCB1 distributions Mean-variance LCB distributions

Results – Risk Aversion

Results – Ambulance Redeployments Algorithms: ARP-UCB1, ARP-TS, ARP-𝜖-greedy

Future Work on ARP via Multi-armed Bandits Driving lower and upper bounds on the regret of multiple ambulance redeployment Combinatorial bandit optimization Trying to find the best arm combination to play (i.e. best ambulance redeployment in a given round -> optimal oracle) Submodular optimization Determining how many ambulances are needed for an approximately (1− 1 3 𝜖)% coverage (trade-off between ambulance redeployment cost and coverage)

References C. Toregas, R. Swain, C. ReVelle, and L. Bergman, “The location of emergency service facilities,” Op. Res., vol. 19, no. 6, pp. 1363–1373, 1971. R. Church and C. R. Velle, “The maximal covering location problem,” Papers Reg. Sci., vol. 32, no. 1, pp. 101–118, 1974. R. Batta, J. M. Dolan, and N. N. Krishnamurthy, “The maximal expected covering location problem: Revisited,” Transportation Sci., vol. 23, no. 4, pp. 277–287, 1989. M. S. Maxwell, S. G. Henderson, and H. Topaloglu, “Ambulance redeployment: An approximate dynamic programming approach,” Winter Sim. Conf., pp. 1850–1860, 2009. C. J. Jagtenberg, S. Bhulai, and R. D. van der Mei, “An efficient heuristic for real-time ambulance redeployment,” Op. Res. Health Care, vol. 4, pp. 27–35, 2015. A. Sani, A. Lazaric, and R. Munos, "Risk-aversion in multi-armed bandits," NIPS. 2012.

Thank you for your attention. Questions?