UAV Route Planning in Delay Tolerant Networks Daniel Henkel, Timothy X Brown University of Colorado, Boulder Infotech @ Aerospace ‘07 May 8, 2007 Actual title: Route Design for UA-based Data Ferries in Delay Tolerant Wireless Networks TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

Familiar: Dial-A-Ride Dial-A-Ride: curb-to-curb, shared ride transportation service The Bus Receive calls Pick up and drop off passengers Minimize overall transit time Path Planning Problem Motivation Problems with TSP Solutions Queueing Theoretical Approach Formulation as MDP and Solution Methods Optimal route not trivial !

In context: Dial-A-UAV Complication: infinite data at sensors; potentially two-way traffic Delay tolerant traffic! Talk tomorrow – 8am: Sensor Data Collection Sensor-1 Sensor-3 Sensor-5 Monitoring Station Sensor-2 Sensor-6 Sensor-4 Have sensors all on left side Highlight ferrying to the right side SMS locations Sparsely distributed sensors, limited radios TSP solution not optimal Our approach: Queueing and MDP theory

TSP’s Problem Traveling Salesman Solution One cycle visits every node UAV pA hub pB One cycle visits every node Problem: far-away nodes with little data to send Visit them less often dA dB fA fB B TSP has no option but alternately visiting A and B. New: cycle defined by visit frequencies pi B

Minimize average delay Queueing Approach Goal Minimize average delay Idea: express delay in terms of pi, then minimize over set {pi} pi as probability distribution Expected service time of any packet Inter-service time: exponential distribution with mean Ti/pi Weighted delay: A B UAV fB fA pA pB dA dB pC * Since the inter-service time is exp. distributed and we are picking up ALL waiting packets when visiting a node, the average delay for a node is the mean of the exponential distribution. * f_i/F is fractional visit probability for node i. C hub pD dC dD D fC fD

Solution and Algorithm Probability of choosing node i for next visit: Implementation: deterministic algorithm 1. Set ci = 0 2. ci = ci + pi while max{ci} < 1 3. k = argmax {ci} 4. Visit node k; ck = ck-1 5. Go to 2. Performance improvement over TSP!

Unknown Environment What is RL? Distinguishing Features: Learning what to do without prior training Given: high-level goal; NOT: how to reach it Improving actions on the go Distinguishing Features: Interaction with environment Trial & Error Search Concept of Rewards & Punishments Example: training dog Learns model of environment.

The Framework Agent Environment Performs Actions Gives rise to Rewards Puts Agent in situations called States

Elements of RL Policy: what to do (depending on state) Reward Value Model of Environment Policy: what to do (depending on state) Reward: what is good Value: what is good because it predicts reward Model: what follows what MDP: action does not depend on previous states or actions; just on current state. Source: Sutton, Barto, Reinforcement Learning – An Introduction, MIT Press, 1998

UA Path Planning - Simple Goal Minimize average delay -> Find pA and pB A B UAV pA hub pB dA dB fA fB Service traffic from A and B to hub H Goal: minimize average packet delay State: traffic waiting at nodes: (tA, tB) Actions: fly to A; fly to B Reward: # packets delivered Optimal policy: # visits to A and B; depend on flow rates, distances

MDP If a reinforcement learning task has the Markov Property, it is basically a Markov Decision Process (MDP). If state and action sets are finite, it is a finite MDP. To define a finite MDP, you need to give: state and action sets one-step “dynamics” defined by transition probabilities: reward expectation:

RL approach to solving MDPs Policy: Mapping from set of States to set of Actions π : S → A Sum of Rewards (:=return): from this time onwards Value function (of a state): Expected return when starting with s and following policy π. For an MDP,

Bellman Equation for Policy π Evaluating E{.}; assuming deterministic policy; π solution: Action-Value Function: Value of taking action a in state s. For an MDP,

Optimality V and Q, both have a partial ordering on them since they are real valued. π also ordered: Concept of V* and Q*: Concept of π*: The policy π which maximizes Qπ(s,a) for all states s.

Reinforcement Learning - Methods To find π*, all methods try to evaluate V/Q value functions Different Approaches: Dynamic Programming Approach Policy evaluation, improvement, iteration Monte-Carlo Methods Decisions are taken based on averaging sample returns Temporal Difference Methods (!!)