David Shim Omid Shakernia A Hierarchical Approach to Probabilistic Pursuit-Evasion Games with Unmanned Ground and Aerial Vehicles Jin Kim René Vidal David Shim Omid Shakernia Shankar Sastry UC Berkeley

Outline Pursuit Evasion Game Scenario Previous Work Hierarchical Control Architecture Implementation on Ground/Air Vehicles Experiment/Simulation Platform Evaluation of Game Strategies Speed, Sensing, Intelligence Experimental & Simulation Results Conclusions and Current Research I will begin the talk by describing the pursuit-evasion scenario, and the previous work that has led up to our current contribution. Next I’ll describe a hierarchical control architecture that we proposed to implement the PEG on real UAV/UGVs. I’ll briefly described our fleet of robots, and the control architecture on these robots. Also, I’ll explain a novel experiment/simulation platform that allows us to perform PEG experiments with real robots, simulations with pure software as well as hardware in the loop simulations of PEG Our simulation/experiment platform enabled us to evaluate the performance of different pursuit policies, and how the performance of high level strategies vary with the speed, intelligence and sensing capabilies of the players in the game. Finaly present conclusions and directions for research ---------------------------- When we implemented the original pursuit strategies on real robots that have nontrivial sensing (vision) and complex dynamics (helicopter, nonholonomic robot, etc), we found many theoritical issues that were not considered with the original theoretical formulation. That formulation had a simplified discrete jump model of robots, and simplified sensing model.

Scenario Evade! This is scenario: There is an open field, out doors, with an unknown terrain and unknown number of trees and other obstacles. There are a group of unmanned ground vehicles, as well as a group of unmanned aerial vehicles. These ground and air vehicles can communicate with each other and form a TEAM of PURSUERS. The mission of the pursuers is to build a map of this environment and to capture another team of EVADERS The evaders can be actively trying to avoid being captured, (for example, by hiding behind obstacles, trees, etc. This is way we solve the problem: Divide the game arena into disjoint number of cells or an OCCUPANCY GRID Compute a probabilistic map of the arena, such that each cell has an associated probability of containing an evader or obstacle Given this probabilistic map, compute a PURSUIT STRATEGY which guides the pursers to be locations in the arena to maximize the probability of capturing an evader Evade!

Probabilistic Map Building Measurements Step sensor model Prediction step: evader motion model Hespanha, et. al. [CDC ’99, CDC ‘00] Optimal pursuit policies computationally infeasible Greedy Pursuit / random evader A few more details on probabilistic map building on the occupancy grid It is a recursive Bayesian approach, which builds up probabilities of an evader or obstacle occupying each cell in the grid, based on possibly noisy measurements of the pursuers. Given a probabilistic map, a the PREVIOUS time instant Use the measurements and sensor model to compute probabilities of evaders occupying cells at the current time instant Next, based on a evader motion model, predict the locations of evaders at the next time instant In Hespanha [cdc99]: First to proposed the approach of combining probabilistic map building and pursuit-policies This was an abstract theoretical work, which considered discrete world, with discrete-jump dynamics of robots, and highly simplified sensing model It was shown that the optimal pursuit policy, which minimizes the expected capture time of the evaders is in general infeasible to compute in real time. Proposed a greedy pursuit policy, where each pursuer maximized probability of capturing evader at next time instant Very nice theoretical results for this greedy policy Probability of having a finite capture time is 1 Expected value of capture time is finite Hespanha [cdc00] One step nash equilibium game theoretic approach

PEG on UAVs and UGVs Vidal et. al. [ICRA ‘01] Hierarchical architecture Implement regulation layer control Kim et. al. [CDC ’01] Implement high level strategies Global-max pursuit Intelligent evader Evaluate Pursuit Policy All this theory is nice, but is at a very abstract level, Many issues remain when trying to implement on real mobile robots Motion of evader is discrete Dynamics of agents not included Oversimplifies sensor model (false positives/negatives) When you implement in real systems, new theoretical issues appear: Modeling different cameras Continuous dynamics of agents, dynamics Communication issues Sensing model in the original theory was greatly simplified So, for the last year and a half our research thrust has been to implement PEGs on real UAVs and UGVS In Vidal et. Al [icra01] We began to implement the PEG on mobile robots, etc Implementation of test-bed on mobile robots: UAVs and UGVs Proposed a Hierarchical approach Implemented Low level regulation on UAVs and UGVs Implemented sensing elements, INS, GPS, integration, Computer vision, etc In Kim et. Al [CDC01] Implement high level strategy planner Perform full probabilistic PEG on real UAVs and UGVs Evaluate performance of pursuit policies in terms of: Dynamics of mobile robots (speed, maneuverability, etc) Sensing capabilities of pursuers (type of vision sensor, range, field of view, etc) In particular, we got some interesting results on the performance of pursuit policies with respect to different types vision systems, that agree with the vision systems we see in predators and pray in the animal kingdom Predators tend to have narrow field of view, forward looking eyes Prey have wide field of view omni-directional vision

Hierarchical Architecture map builder terrain evader control signals [4n] evaders detected obstacles pursuers positions Desired pursuers state of helicopter & height over terrain obstacles detected tactical planner & regulation actuator lin. accel. & ang. vel. [6n] inertial [3n] height over terrain [n] evaders detected vehicle-level sensor fusion communications network tactical planner trajectory regulation position of evader(s) position of obstacles strategy planner position of pursuers agent dynamics encoders INS GPS ultrasonic altimeter vision Exogenous disturbances The proposed hierarchical architecture for PEG comes from the theory of hierarchical hybrid systems. This architecture has been successfully applied to controlling platoons of cars in automated highway systems, as well as for conflict resolution in air traffic management systems, and flight vehicle management systems for controlling UAVs. The idea is to partition is large and complex control problem into various layers of abstraction. Strategic planner / Map Building: pursuit policy computation Communication layer tactical planner, and sensor fusion path planning, obstacle avoidance, position estimation of evader, Regulation layer real-time control, GPS, vision system

Experiment/Simulation Platform UAV MATLAB/ Simulink Vision, Communication, Path Planning Real-time Control Tactical Planner Navigation Computer Strategic Planner Map Builder UGV Tactical Planner Robot Controller Now I’ll described the unified Experiment/Simulation platform which we have built to perform the pursuit-evasion game REAL-TIME CONTROL At the lowest level of the hierarchy, is the navigation and real-time control for our fleet of UAVs and UGVs. The real-time control of the UAV was part of the PhD work of David Shim.l The UAV regulation layer provides services such as hover, lateral motion, pirouette, etc Also manages the Inertial navigation system and GPS to know exactly the position of UAV The UGV is a Pioneer2AT robot, which comes with software for simple motion control. We have also integrated a GPS system and compass similar to the UAV TACTICAL and SENSOR FUSION The UAV and UGV have integrated Vision systems which they use for sensing the position of obstacles and evaders Currently the vision-based evader position estimation is based on COLOR TRACKING Simple sonar-based obstacle avoidance behaviors for UGVs STRATEGY and MAP BUILDING This is implemented in MATLAB and SIMULINK, and they communicate with the UAVs and UGVs through TCP sockets Strategy is sheltered from details of dynamics of robots: we can apply a GREEDY policy that was designed in that abstract theory directly to any number of UAVs and UGVs

Experiment/Simulation Platform UAV model MATLAB/ Simulink System ID model, Camera model, INS/GPS model UAV Simulator Strategic Planner Map Builder UGV model Robot model, Camera model, Dead reckoning Pioneer Simulator A further benefit is that we can use the same STRATEGY PLANNER and MAP BUILDER in a SIMULATION by just replacing the actual robots with software simulation models of the robots We have a UAV model that was obtained by system ID (same model one used build the real-time controller) The pioneers come with a simulation model that performs dead reckoning We made a simple camera model, where we compute each pursuers region of visibility based on its position and field of view of its camera. In the simulation, if an evader or obstacle is within the region of visibility, then we a probability of false positives and false negatives of detection. Further, this is very flexible platform where we can purform full hardware experiments, full software simulations, and any mixture of hardware in the loop simulations.

PEG Experiment PEG with four UGVs Global-Max pursuit policy Simulated camera view (radius 7.5m with 50o FOV) Pursuer=0.3m/s Evader=0.1m/s

Pursuit Policy: Sensing, Intelligence, Speed Greedy Global-max Visibility Region Forward View Omni-directional View Evasion Policy Random Global-min Evader speed Evaluated policies against different vision capabilities Trapezoidal (narrow FOV) vs. OMNI-directional (wide FOV) Both vision systems covered same number of cells Narrow field of view Can see farther into distance Can sweep a larger area simply by rotation Omni-directional (wide angle) Can see in all angles Rotation does not help see more

Pursuit Policy vs. Vision System Why global max outperforms greedy: Before we did real implementation, both policies were performing about the same Now, with all the dynamics and sensing, global-max 3 times better Reason Greedy policy: more often gives change of direction, pursuers have to spend more time rotating, which effectively reduces the translational speed (in practice you cannot rotate and move at full speed at same time) Global max: changes less frequently: pursuers don’t spend as much time changing directions, effectively faster Why trapezoidal better than omni-directional? Omni camera and trapezoidal camera cover same number of cells By rotating in place pursuers can effectively see a larger area Pursuers see further into distance Agrees with predator/prey situations we find in nature

Evader Speed vs. Policy Next we took the best performing pursuit policy, global-max, and the best vision system (forward view), and studying how changing the speed of the evader, and the intelligence of the evader affected capture time. The the case of an intelligent evader, the evader also builds a probabilistic map of pursuers and evaders, and follows a GLOBAL-MIN policy, where it tries to go to the location in the map with minim probability of being captured. Also, we kept the pursuers speed constant at 0.3m/s and did experiments where evaders were slow (0.1m/s) or faster (0.5m/s). It is intuitive to see that a FAST INTELLIGENT evader takes longer to be captured than a SLOW INTELLIGENT evader What is interesting to notice is a FAST RANDOM takes LESS time to capture than a SLOW RANDOM evader This was actually predicted in the original work of Hespanha, where in the extreme case that the evader is staying in place, the probability of capturing in finite time is less that 1 One important point is that the total capture time is some combination of exploration time, where the pursuers are building a map of the environment, as well as pursuit time ------------------------------------ * There must be some U shaped curve with the optimal speed for each pursuer

PEG: 4 UGVs and 1 UAV

Conclusions Conclusions Current Research Hierarchical architecture applied to control multiple agents for pursuit evasion scenario Evaluated strategies vs. speed, sensing and intelligence Global-max outperforms greedy in a real scenario Forward view outperforms Omni-view Vision Agrees with biological predator/prey vision systems Current Research Multi-Body Structure from Motion for Pursuit-Evasion Games [submitted IFAC ’02] Collision Avoidance and UAV Path Planning Monte Carlo based learning of Pursuit Policies In practice, color tracking is not feasible, Need vision system that is able to identify and track each evader using features different than color Whole body of computer vision literature for one moving object: currently generalizing that theory for multiple moving objects

THE END