1. CoMotion Computational Methods for Collaborative Motion Pursuit Evasion Games for Networks of UUVs November 2004 Mike Eklund, Jonathan Sprinkle, Shankar Sastry

2. Pursuit Evasion Games for Multiple UUVs UUVs have the potential to provide an effective defense against submersible threats to military and civilian assets The strategies and protocols for their operation are at least as big a challenge as the design and construction of the vehicles themselves This project address the strategies and coordination protocols necessary to enable this technology Defense against enemy subs is the number one FNC that is called for in every briefing since 2000.

3. Bear UUV Being Developed Lt Tulio Celano III, USN 8 ft X 10 in., 400 lbs. displacement, upto 100 ft. depth, Top speed 12 knots, effective cruising speed 5 knots, endurance at 5 knots is 45 hours, Modular design

4. Prior PEG Experience and Lessons Learned Pursuit evasion games (PEGs) have been demonstrated within our group using unmanned aerial and ground vehicle (UAVs and UGVs) in both two and three dimensions. Additionally, they have also been performed in both symmetrical and asymmetrical games

5. Previous UC Berkeley PEG Experiments Berkeley Aerobot Project (BEAR) –Goal: to build a coordinated, intelligent network with multiple heterogeneous agents 11 Rotorcraft-based unmanned aerial vehicles (UAVs) 5 Unmanned ground vehicles (UGVs) Shipdeck simulator (landing platform) Stochastic Pursuit-Evasion Games (PEG) –Self-localization –Target detection –Map building –Pursuit policy –Trajectory generation –Control / Action

6. Previously: PEGs with 4 UGVs and 1 UAV Sub-problems for Pursuit Evasion Games –Sensing Navigation sensors -> Self-localization Detection of objects of interest –Framework for communication and data flow –Map building of environments and evaders How to incorporate sensed data into agents’ belief states  probability distribution over the state space of the world (I.e. possible configuration of locations of agents and obstacles) How to update belief states –Strategy planning Computation of pursuit policy  mapping from the belief state to the action space – Control / Action

7. PEG Experiment with UAV/UGVs PEG with four UGVs Global-Max pursuit policy Simulated camera view (radius 7.5m with 50degree conic view) Pursuer=0.3m/s Evader=0.5m/s MAX

8. Evaluation of Policies for different visibility Global max policy performs better than greedy, since the greedy policy selects movements based only on local considerations. Both policies perform better with the trapezoidal view, since the camera rotates fast enough to compensate the narrow field of view. Capture time of greedy and glo-max for the different region of visibility of pursuers 3 Pursuers with trapezoidal or omni-directional view Randomly moving evader

9. Evader’s Speed vs. Intelligence Having a more intelligent evader increases the capture time Harder to capture an intelligent evader at a higher speed The capture time of a fast random evader is shorter than that of a slower random evader, when the speed of evader is only slightly higher than that of pursuers. Capture time for different speeds and levels of intelligence of the evader 3 Pursuers with trapezoidal view & global maximum policy Max speed of pursuers: 0.3 m/s

10. SEC Capstone Demo: Fixed-Wing PEGs Capstone Demonstrations were proposed to highlight and test the technologies developed in the SEC program One was a fixed-wing UAV flight test –6 participant technology developers (TDs) Honeywell, Northrop Grumman, U Minnesota, MIT, Stanford, and UCB/U Colorado/CalTech –System Integrator was Boeing –OCP would be software framework –A T-33 trainer as UAV surrogate –An F-15 as wingman/opponent 13 month schedule May 03 – June 04 UCB Contribution: Fixed-Wing PEGs

11. Nonlinear Model Predictive Trajectory Control Explicitly addresses nonlinear systems with constraints on operation and performance A cost minimization problem in the presence of state and input constraints –Control resulting in the minimum cost is determined over a model predicted horizon Previously demonstrated in rotary wing UAVs [1] [1] H.J. Kim, D.H. Shim and S. Sastry, ACC, 2002

12. Demo: UCB PEG Development 20 – 60 min. games confirm NMPTC feasibility at real-time –Evader goal: get to final waypoint or avoid evader –Pursuer goal: ‘target’ evader Pursuer and evader restricted to same performance limits Planes on the same logical plane, but separated by 6000ft altitude at all times Evader and pursuer have a few scenarios –UAV as evader –UAV can become pursuer OCP Experiment Controller Snapshot T-33: Evader (yellow) F-15: Pursuer (blue)

13. Demo: Controller Coverage Final waypoint is the major component Travel to the waypoint can be interrupted by the actions of the pursuer –Relative positions –Angle-off-tail –Distance from each other –Proximity to boundaries of the testing area –Physical limitations of the aircraft Velocity, climb/turn rate Use a simpler model for predictive behavior under certain conditions –Still use the Boeing-supplied (DemoSim) model for behavioral simulations Approx. same timePursuer goes for target cone Evader turns away (regardless of endpoint) Endpoint Target cone definition (θ=10˚,d=3 nm) Left: F15 not behind UAV, middle: F15 not pointed at UAV, right: F15 behind AND pointed at UAV

14. UCB PEA Experiment #1 Test Plan: UAV as Evader UAV attempts to cross Scenario Area (SA) from East to West without being targeted by the F15 UAV “wins” by: –Reaching the RVPT –Not being targeted for 20 minutes F15 “wins” by targeting the UAV Note: F15 performance is restricted

15. UCB PEA Experiment #2 Test Plan: UAV as Evader and Pursuer UAV attempts to cross Scenario Area (SA) from East to West without being targeted by the F15, however, UAV will attempt to target F15 if suitable conditions arise UAV “wins” by: –Reaching the END ZONE –Not being targeted for 20 minutes –Targeting the F15 F15 “wins” by targeting the UAV Note: F15 performance is restricted

16. Flight Test 1 (UAV as evader)

17. Flight Test 2 (UAV as evader/pursuer)

18. UUV PEGs: Multiplayer Games In littoral waters the pursuit evasion game consists of an enemy submarine attempting to cross a line of UUVs which are protecting an asset The enemy submarine has a speed advantage over the blue force UUVs UUVs play a role in between a sensor web and a group of pursuers Research aimed at determining new approaches to teaming for multi-player games. Current literature focuses exclusively on either Nash or Stackleberg solutions

19. Multiplayer PEG Challenges The research challenge includes extending the strategies to: –Large multi-player teams –Asymmetric platform characteristics –Limited communications –High level of uncertainly

20. UUV PEG: Approach The UUV PEG involves two distinct phases: –Detection phase Maximize chances of detection constrained by: –Area to cover –Number of UUVs available –Possible evader strategies –Capabilities of UUVs and evader (sensors and noise signatures) –Response phase Maximize chances of catching the pursuer constrained by: –Capabilities of UUVs and evader (speed, manueverability and communications) –Number of UUVs available These two phases also depend on each other as both must succeed –How to share resources to maximize overall chance of success –How to overlap the strategies: detectors are responders as well

21. Multiplayer PEGs: Proposed Solution A close analogy is football or other team games: –Multi player –Initial (global) strategies well defined –Limited (local) coordination after the snap What can we learn? How can we apply this? How far does the analogy go?

22. Multiplayer PEGs Preseason (Off-line precomputed strategy) –Play book: Evaluate strategies and configurations that will maximize chance of success based on best estimate of other team’s tactics –Practice and preseason games: Test playbook and find problems Game time (On-line adaptive strategy) –Choose play based on best knowledge and experience Line up (in best detection configuration, not necessarily static) –Execute the play Active and reactive actions (respond to detected evader) Local communication Adapt to evolving behavior Learn from experience, repeat as necessary (Learning by Doing)

23. Pre-game: Strategies for Detection Maximize the chance of detecting the evader Tradeoffs –Movement of the pursuer: Moving quickly covers more area, but This makes it easier for evader to see the pursuer and avoid the pursuer –Sensors: Using passive sonar reduces the range of detection Using active sonar reveals the pursuers location –Number of pursuers in detector role: Increases chance of detection

24. Basic principle: Defense in depth May not be the optimal with limited resources, –for instance if there are not enough UUVs to ensure detection of the evader by the front line.

25. Options: Zone Defense Would this leave seams for an evader to exploit if they have superior sensors, for instance? Is communication necessary to make such a zone defense effective? Is there an alternative?

26. Options: Channeling the evader A coordinated, heterogenous detection strategy. For instance, some pursuers could use a very active strategy that exposed them to intentional detection by the evader, with the intension of “channeling” the evader towards other more passive pursuers.

27. Pursuer Strategies: Capture Speed disadvantage means that simply optimizing the detection probability is not sufficient Reachability of UUVs must be known Communication and coordination will be necessary to overcome speed disadvantage

28. Defining the Problem: Basic UUV PEG Scenarios Scenario 1 –Single evader infiltration –Objectives Red: pass through game area undetected Blue: detect red team only Scenario 2 –Single evader attack –Objectives Red: get within weapons range of some objective Blue: prevent red attack on objective –Capabilities Red: limited number of torpedoes available to attack target or blue team UUVs Red: suicide attacks only, must get within effective range Scenario 3 –Multiple evader attack –Objectives & Capabilities Same as Scenario 2

29. UUV Characteristics in General Performance –Speed and acceleration –Maximum rate of turn –Maximum rate of ascent/descent (not symmetric in general) –Maximum depth Sensors –Effective range in passive detection mode –Effective range in active detection mode –Deployable sonar buoys Communications –Effective communication range (variable in general)

30. UUV Characteristics, cont. Method of attack, including: –Self detonation, effective range and perhaps effectiveness as a function of range –Missile (i.e. torpedo) capabilities Counter measures, including: –Sonar buoys –Noise canisters Noise signature as a function of: –Speed –Acceleration –Rate of turn –Rate of ascent/descent –Sensor mode –Communication mode And others

31. The First Problem Definition Based on “Scenario 2”: –Single evader attack, many defenders –Objectives Red: get within weapons range of some objective Blue: prevent red attack on objective –Capabilities Red team: –Limited number of torpedoes available to attack target or blue team UUVs –3 times speed advantage over the Blue (pursuer) team Blue team: –Suicide attacks only, must get within effective range –3 times manuever (turn rate ) advantage over the Red team –Limited communication range –Passive and active sensors available

32. The First Problem Definition, cont Characteristics –Speed 3X advantage to Red Team –Maneuver advantage 3X to Blue Team –Detection a function only of speed, communication use Distance –Sonar Active & passive available –Communications available for each team: range a function of power detectability also a function of power

33. Detection: Strategy Comparison

34. Detection: Monte Carlo results Goals: –Statistical model as a function of configuration, spacing, etc. –Test strategies Line abreastStaggered (Recall: detection function)

35. Capture: Strategies based on reachability (Mitchell’s Level Set Toolbox) From Airplane exampleStill even, turn rate reduced to 1/3 Pursuer speed reduced to 1/3Pursuer speed reduced to 1/3, turn rate increased by 3

36. Combined the Strategies: Advances in Game Theory These PEGs fit Game Theory descriptions as: –mixed strategy –simultaneous move –multiplayer –coordinated games –games with incomplete information Specific tactics can be evaluated to find the equilibria and optimal strategies in certain situations, e.g.: –the best search patterns within a single zone for one UUV, or for two adjacent zones/UUVs Statistical methods or Monte Carlo methods could be used to determine the changes of success for each player

37. Communication Between UUVs The issue of the short range of underwater communications with multiple players is very similar to the problem of ad hoc wireless networks of motes devices This experience is directly adaptation to the multiple, coordinated UUV scenarios and used to disseminate information within the team regarding: –Detection of an evader –Likely detection by the other team –Coordination instructions –Strategic commands –etc. This is integral into the simulation environment

38. Future Research We are building a group of UUVs at the USNA in Anapolis. These are also being shared with NUWC, Newport We will develop a theory of multi-player pursuit evasion games with off-line strategies (using robust optimization, the level set toolbox), on-line modifications of the strategy using Model Predictive Control, and an outerloop of learning Expect to have simulation results by June 05, experiments for single UUV by June 05, multiple UUV experiments by June 06