1. 1 An Experimental System for the Collaborative Control of Unmanned Air Vehicles Raja Sengupta, CEE Systems, UC Berkeley Joint work with Karl Hedrick, ME UC Berkeley  Graduate Students- Tim McGee, Elaine Shaw, Xiao Xiao, Jack Tisdale, Dan Coatta, David Nguyen, Allison Ryan, Mark Godwin, Sivakumar Rathinam, Marco Zennaro, John Cason, and Dan Prull  Post Docs- Derek Caveney, Zu Kim, Stephen Spry  Engineers- Aram Soghikian, Susan Dickey, Dave Nelson

2. 2 Collaboration Research Goals  Study distributed mechanisms for the collaboration of Unmanned Vehicles Air to air communication used  Generalize a large number of missions under one framework Surveillance/Mapping Border Patrol Search & Rescue Convoy Protection, etc.  Create a system that: Can accommodate a large number of agents (UVs) Displays tolerance to communcation, hardware failure faults Capable of running in real time

3. 3 In Action

4. Mission Control

5. 5 Commander View

6. 6 Current UAV Platform Configuration  Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance  Ground-to-Air UHF Antenna for ground operator interface  GPS Antenna for navigation  802.11b Antenna for A-2-A comm.  Payload Tray for on-board computations and devices  Payload Switch Access Door for enabling / disabling on-board devices

7. 7 Current Payload Configuration  Off-the-shelf PC-104 with custom Vibration Isolation  Orinoco 802.11b Card and Amplifier for A-2-A comm.  Analog Video Transmitter for surveillance purposes  Printed Circuit Board for Power and Signal Distribution among devices.  Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft.  Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground.

8. 8 MLB Bat IVAircraft Improved payload weight (25lbs) and volume Improved logistics: 7.5 hour duration, onboard generator

9. 9 Autopilots and Sensors  Piccolo II autopilots 4Hz GPS updates (compared to 1 Hz in the old system) Improved gyros, leading to much better attitude estimation Analog I/O ports, allowing integration of user-specified sensor inputs into the core autopilot structure Satellite communication capability  Sensors several new types: IR camera, radar, IMU, gimbals will allow expansion of efforts in mapping, vision-based tracking, and control based on other sensor types. will allow testing and comparison of the effectiveness of various sensors for particular tasks will allow exploration of how sensor types in a heterogeneous UAV team can be used together in a complementary way.

10. 10 Communications and Video Link  New air-to-air communications system amplified 802.11b testing of collaborative team control concepts using short-range air-air comm. will retain long-range, low-bandwidth, air-ground links  New video downlink system better monitoring of aircraft video streams will allow ground-based testing of image processing algorithms and human-machine interface systems. Ground Station

11. 11 Future Experimental System  DURIP funded multi-aircraft testbed  Six Primary Components A. Five new aircraft with improved payload capacity and configuration B. Upgraded autopilots with improved autopilot functions C. New sensors-(bullet cameras, fisheye lenses, IR camera, radar, IMU, gimbals) D. New air-to-air communications system E. New video downlink system F. Trailer/operations center

12. 12 System Architecture Task allocation/ Conflict Resolution Mission to task decomposer UAV Team Level Mission Control missions UAV

13. 13 BLCC- Berkeley Language for Collaborative Control  Define the mission and communicate it to team members  Define the “state” of each agent  Define the mission “state”  Allow for faults  Allow for conflict resolution  Define the information to be communicated between agents.

14. 14 Current Collaborative Architecture Task 1 Task 2 Allocated Task 1 Allocated Task 1 Tasking Conflict Border Patrol Location Visit Reallocated To Task 2 Conflict Resolution

15. 15 Example Scenario fault subtasks start obstacles UAV i indicates comm.

16. 16 Mission State  Each agent communicates primarily through a list of tasks that is shared between UVs.  A task is often described by a location, such as a GPS position. 1 2 3 4

17. 17 Current Collaborative Architecture  Mission statements are decomposed into tasks and relayed from the ground to all aircraft.  Each plane without a task picks the closest available task for itself. Each plane allocates tasks only for itself.  Conflicts in task allocation are resolved using Euclidean distance.  Each aircraft broadcasts its current state and its knowledge of other vehicles’ states. It only has overwriting permissions for its own state.  Current objective If the airplanes communicate sufficiently often each task will eventually be done  More involved communication, tasking, and conflict resolution protocols are currently under development for future system integration

18. 18 Simulation Results

19. 19 Database PiccoloPayload Orbit ControlWaypt ControlVision Control Switchboard Vision Process. Orinoco Comm. Camera A-2-A Ground Commands Aircraft Avionics Vision Control For Turn-Rate Based Path Following Waypoint Control For Single-Point Visits Orbit Control For Closed-Loop Multi-Point Paths Switchboard Task Allocation, Conflict Resolution, and Controller Switching Orinoco Inter-vehicular communication protocol Database Permits inter-process communication Vision Processing Frame Grabbing Capabilities Payload Responsible for Relaying Commands between the PC-104 and Mission Control Piccolo Responsible for Relaying Commands between the PC-104 and Aircraft Avionics Aircraft Level Architecture

20. 20 Generalization: Vision Based Following of Locally Linear Structures (Closed Loop on the California Aqueduct, June 2005)

21. 21 Results – Canal Following  The road detection algorithm runs at 5 Hz (takes < 200 ms) or faster on the PC104 (700 MHz, Intel Pentium III).  No visible error was found from video sequences of over 100 frames containing the canal

22. 22 Cal Road Detection on MLB Video (No Control) Generic corridor detection by one- dimensional learning Roads Aqueducts Perimeters Pipelines Power Lines

23. 23 Vision Based Obstacle Avoidance System

24. 24 Conclusion  Built an unmanned air vehicle system for experimental work on collaboration Currently four airplanes Five more planned  Current missions Visit location and send picture Border patrol  GPS based  Vision based  Collaboration Mission to task decomposition  Each mission should have its own semantics of decomposition Autonomous in-air task division and conflict resolution Currently limited to static tasks

25. 25 Geographic Data Management

26. 26 Scalable Information Management: Target Map and Risk Map  Target distribution map P(A, N, t); probability of N targets of type t in area A  Target distribution update Fuses measurements from different kinds of sensors (SAR and EO) Bayesian update  Risk map computation Integral of threat model with respect to the measure P(A, N, t) Generates the value function for navigation Example: Target Map Risk Map UCB Rathinam 2003

27. 27 Scalable Information Management: Distributing the Publisher Service  Geographic Data Management Network Euclidean Space Voronoi tessellation Data objects Publishing Servers Sengupta AINS 2003

28. 28 Scalable Information Management: Distributing the Publisher Service Metric Space desired data server User delivery

29. 29 Movie of Implementation  4 laptops over wireless  One publisher per laptop  Start with one publisher  Three others come up  Some die  Data redistributes as publishers join and leave Total data is this map

30. 30 Movie of Implementation  4 laptops over wireless  One publisher per laptop  Start with one publisher  Three others come up  Some die  Data redistributes as publishers join and leave Total data made of many data objects

31. 31 Movie of Implementation  4 laptops over wireless  One publisher per laptop  Start with one publisher  Three others come up  Some die  Data redistributes as publishers join and leave Voronoi tessellation

32. 32 Movie of Implementation  4 laptops over wireless  One publisher per laptop  Start with one publisher  Three others come up  Some die  Data redistributes as publishers join and leave

33. 33 Movie of Implementation  4 laptops over wireless  One publisher per laptop  Start with one publisher  Three others come up  Some die  Data redistributes as publishers join and leave

34. 34 Scalable Information Management: Distributing the Publisher Service Movie of our Implementation 4 servers on 4 laptops over wireless

35. 35 Data Consistency in the Publisher: Inconsistent copies are detected whp Data Location Wrong location copy 1 Wrong location copy 2

36. 36 Data Consistency in the Publisher: Drift in a 2-D Markov Process

37. 37 Geographic Data Management Network: Survivable Information for UAV Swarms  The server backbone dynamically tracks the client agent organization  Servers move in and out while the information survives

38. 38 Tracking the Agent Organization: Dynamic GDMN backbone Control  Design a distributed control algorithm for the servers to partition the data and the clients to minimize the total bit-meters (Kumar etal.) of work done in the system and balance the load on the servers.  Let the load generated in each client be b i. If the locations of the points are denoted by p i and the location of the servers are denoted by c j, then the total cost is:  b i ( min dist(p i, c j ) )  i  j  The control algorithm updates server positions to reduce this cost

39. 39 Simulation  This example involves 100 clients and 6 servers

40. 40 Control algorithm  In each sampling interval, each server Measures the positions and the traffic generated by its clients  GDML routing protocols make the client set the Voronoi cell Calculates the weighted centroid of all the clients it serves Moves towards its weighted centroid  Works well if the servers travel faster than the clients  The algorithm is based on the k-means algorithm (MacQueen,1967 )

41. 41 Publications: 2005

42. 42 Publications: 2005 cont’d

43. 43 Publications: 2005 cont’d

44. 44 Publications: 2004

45. 45 The End

46. 46 Publications: 2004 cont’d

47. 47 Publications: 2003

48. 48 Theses and Dissertations

49. 49 The End