1. GRASP University of Pennsylvania Adaptive Autonomous Robot TEAMS for Situational Awareness GRASP Laboratory University of Pennsylvania PI:Vijay Kumar Senior Personnel: Camillo Jose Taylor, Jim Ostrowski Research Associates James Keller, John Spletzer, Aveek Das, Guilherme Pereira, Luiz Chaimowicz, Jong-Woo Kim, Anthony Cowley Co-PI: Ron Arkin Senior Personnel: Tucker Balch, Robert Burridge Research Associates Keith O’Hara, Patrick Ulam, Alan Wagner Co-PI: Gaurav Sukhatme Senior Personnel: Maja Mataric, Andrew Howard, Ashley Tews Research Associates: Srikanth Saripalli, Boyoon Jung, Brian Gerkey, Helen Yan Co-PI: Jason Redi Senior Personnel: Josh Bers, Keith Manning Mobile Robotics Laboratory Georgia Institute of Technology Robotics Research Laboratory University of Southern California BBN Technologies

2. GRASP University of Pennsylvania Future Combat Systems The Future Combat System (FCS) concept revolves around the creation of a network-centric force of heterogeneous platforms that is strategically responsive, lethal, survivable and sustainable l communication in active mobile nodes during network-centric warfare; l integration of multiple, heterogeneous views of the target area

3. GRASP University of Pennsylvania Key FCS Considerations l Adapt to variations in communication performance and strive to maximize suitably defined network-centric measures for perception, control and communication l Provide situational awareness for remotely- located war fighters in a wide range of conditions l Integrate heterogeneous air-ground assets in support of continuous operations over varying terrain

4. GRASP University of Pennsylvania Context Communication Network l 400 MHz (100Kbs), 2.4 GHz (10Mbs), 38 GHz (100 Mbs) l Affected by foliage, buildings, terrain features, indoor/outdoor l Directionality Small Team of Heterogeneous Robots l UGVs with vision, range finders l UAVs (blimp, helicopter)

5. GRASP University of Pennsylvania GOALS l A comprehensive model and framework integrating communications, perception, and execution l Automated acquisition of perceptual information for situational awareness l Reactive group behaviors for a team of air and ground based robots that are communications sensitive l A new framework for mobile networking in which robots use sensory information and relative position information to adapt network topology to the constraints of the task.

6. GRASP University of Pennsylvania GRASP Laboratory University of Pennsylvania Mobile Robotics Laboratory Georgia Institute of Technology Robotics Research Laboratory University of Southern California BBN Technologies Control, Vision Behaviors, Architecture Comms, Networking Sensing, Mapping MARS TEAMS

7. GRASP University of Pennsylvania Thrusts 1.Ad Hoc Networks for Control, Perception and Communication 2.Software framework for distributed computation, sensing, control, and human-robot interface 3.Communications-sensitive operations 4.Network-centric approach to situational awareness 5.Mission-specific planning and control for a team of heterogeneous robots 6.Adaptation of behaviors and networks to changing conditions

8. GRASP University of Pennsylvania Thrusts and Tasks

9. GRASP University of Pennsylvania 1. Ad Hoc Networks for Control, Perception and Communication Physical Network (R, E S ) Communication Network (R, E C ) Computational Network (R, H ) e ij ={i, j, b m, b v, d m, d v } q i = {  m,  v }

10. GRASP University of Pennsylvania Models of Communication Modeling l Effect of foliage l Buildings l Dependence on frequency, directionality l Statistical models of delays and “hot spots” from experimental data u Neighbors, path costs (delays, power) u Time of last communication QoS metrics l Control/perception tasks l Individual robots vs. end-to-end l Move to improve reliability and network performance Interface between network and robot software

11. GRASP University of Pennsylvania Self-Awareness and Cooperative Localization (Penn) l Discovery – robots can organize themselves into a team l Localization – establish relative pose information R1R1 R2R2 R3R3 R4R4 R5R5 R1R1 R2R2 R3R3 R4R4 R5R5

12. GRASP University of Pennsylvania Self-Awareness and Cooperative Localization l Network of UGVs and Surrogate UAV l Reactive controllers that maintain, exploit network

13. GRASP University of Pennsylvania Cooperative Control (Penn) l Reactive controllers that maintain and exploit network l Controllers and estimators are represented by graphs l Fundamental connection between graph structure and performance (stability, convergence)

14. GRASP University of Pennsylvania 2. Software framework for distributed computation, sensing, control, and human- robot interface Player/Stage (USC) l Robots l Sensors u Sonar u IR u Scanning LRF, cameras (color blob detection) l Integration

15. GRASP University of Pennsylvania 2. Software framework for distributed computation, sensing, control, and human- robot interface (continued) ROCI (Penn) l Discover other processes l Communicate with other processes l Monitor other processes l Control other processes

16. GRASP University of Pennsylvania 3. Communications-sensitive behaviors and operations Networking l Models (BBN) l Diagnostics (BBN) Control of Mobility l Behaviors (GT) l Verification and Analysis (Penn) Operations (Thrust 5) l Mission specification (GT) l Mission Planning (GT)

17. GRASP University of Pennsylvania 4. Network-centric approach to situational awareness Cooperative Localization l Vision (Penn) l Range sensors, GPS, and IMU (USC) l Unreliable communication Acquisition of 3-D information (Penn) Cooperative behaviors (USC, Penn) Cooperative Mapping (USC) Semantic Markup of Maps (USC)

18. GRASP University of Pennsylvania 5. Mission-specific planning and control for a team of heterogeneous robots FCS scenarios (BBN, GT) MissionLab integration (GT)

19. GRASP University of Pennsylvania 6. Adaptation of behaviors and networks to changing conditions l Adaptation of control modes (Penn) l Reinforcement learning to adapt mode switching (sequential composition of behaviors) (USC, Penn) l Path referenced perception and selection of behaviors (USC) l Variable autonomy (USC) l Operation under stealth (USC)

20. GRASP University of Pennsylvania Technology Integration l Air Ground Coordination l Command and Control Vehicle l Software u Mission planning u Control for communications u Active perception u Infrastructure for distributed computing

21. GRASP University of Pennsylvania Georgia Institute of Techology GT Personnel l Faculty u Prof. Ron Arkin u Prof. Tucker Balch u Dr. Robert Burridge l GRAs u Keith O’Hara u Patrick Ulam u Alan Wagner Mobile Intelligence Inc. l Dr. Doug MacKenzie

22. GRASP University of Pennsylvania Impact - GT l Provide communication-sensitive planning and behavioral control algorithms in support of network- centric warfare, that employ valid communications models provided by BBN l Provide an integrated mission specification system (MissionLab) spanning heterogeneous teams of UAVs and UGVs l Demonstrate warfighter-oriented tools in three contexts: simulation, laboratory robots, and in the field

23. GRASP University of Pennsylvania Task 1: Communication-sensitive Mission Specification MissionLab is a usability-tested Mission-specification software developed under extensive DARPA funding (RTPC / UGV Demo II / TMR / UGCV / MARS / FCS-C programs) l Adapt to incorporate air-ground communication-sensitive command and control mechanisms l Extend to support physical and simulated experiments for objective air and ground platforms l Incorporate new communication tasks and triggers

24. GRASP University of Pennsylvania Task 2: Communication Sensitive Planning l Add support for terrain models and other communications relevant topographic features to MissionLab l Use plans-as-resources as a basis for multiagent robotic communication control (spatial, behavioral, formations, etc.) and integrate within MissionLab

25. GRASP University of Pennsylvania Task 3: Communication-Sensitive Team Behaviors l Generation and testing of a new set of reactive communications preserving and recovery behaviors l Creation of behaviors sensitive to QoS l Expansion of Behaviors in support of line-of-sight and subterranean operations

26. GRASP University of Pennsylvania Task 4: Communication Models and Fidelity Work with BBN to incorporate suitable communication models into MissionLab in support of both simulation and field tests

27. GRASP University of Pennsylvania Task 5: Technology Integration l Conduct Early-on Demonstrations on Ground Robots at Georgia Tech l Provide our Hummer Command and Control Vehicle for Team support at Objective Demonstration u Currently being used for FCS- C Program u Fully actuated – capable of teleautonomous control

28. GRASP University of Pennsylvania University of Southern California Faculty: l Prof. Gaurav Sukhatme l Prof. Maja Mataric Research Associates: l Dr. Andrew Howard l Dr. Ashley Tews Graduate Students: l Srikanth Saripalli, Boyoon Jung, Brian Gerkey, Helen Yan

29. GRASP University of Pennsylvania USC Task Summary Outdoor simulation Cooperative outdoor localization Semantic representations Stealthy behaviors Path-referenced perception HRI Integration

30. GRASP University of Pennsylvania Task 1: Stage Simulation Current l Multi-robot 2D simulation, models differential and omni-drive robots, sonar, IR, scanning LRF, cameras (color blob detection), pan- tilt-zoom heads, and simple 2 DOF grippers l Language independent, architecture neutral Extensions l 3D simulation for outdoor terrain. l Incorporate USC helicopter and UPenn blimp

31. GRASP University of Pennsylvania Task 2: Cooperative Outdoor Localization l Extend existing localization algorithms to outdoor environments. l Implement outdoor localization in the presence of partial GPS. l Validate through outdoor experiments with small teams (4 ground robots).

32. GRASP University of Pennsylvania Task 3: Semantic Representation and Activity Recognition Semantic mark-up of maps with following attributes: l elevation, terrain type and traversability, foliage and coverage type, and impact on communications. Integrate activity/motion detection algorithms to locate people in the environment. Demonstrate semantic markup using ground robots at USC.

33. GRASP University of Pennsylvania Task 4: Variable Autonomy and Stealth l Develop and implement behaviors for variable autonomy incorporating operator feedback using gestures l Develop and implement a new “stealthy patrolling” behavior by integrating visibility constraints into current patrolling algorithms l Adapt and tune above behaviors using reinforcement learning to improve performance

34. GRASP University of Pennsylvania Task 5: Path-referenced Perception and Behaviors l Develop path-referenced perception and behaviors, which allow recall of behavioral strategy relative to priors paths taken in the mission l Integrate path-referencing which allows robots to query each other for relative locations of semantic mark-ups

35. GRASP University of Pennsylvania Task 6: Human Robot Interface l Extend Stage to serve as a simple visual display for war fighter. Overlay visual information with laser information in Stage. l Provide simple auditory feedback to the operator about current behavioral state of robots.

36. GRASP University of Pennsylvania Technology Integration l Demonstrations at USC of cooperative localization (laser based with IMU and GPS) using ground robots and USC helicopter. l Demonstration at USC of activity detection, semantic markup of terrain and stealthy traverses. l Support joint demonstration with ground robots.

37. GRASP University of Pennsylvania Faculty l Vijay Kumar l Camillo Jose Taylor l Jim Ostrowski Research Associates l James Keller l Luiz Chaimowicz Students l John Spletzer, l Aveek Das l Guilherme Pereira l Jong-Woo Kim l Vito Sabella

38. GRASP University of Pennsylvania Task 1: Model of Ad Hoc Network 1.Develop a comprehensive model for control, perception and communication for situational awareness 2.Integrate models of interference, bandwidth, latency and QoS of the communication network with models of control, sensing and communication. Performance measure Implications for mobility  (R, H )   (R, H )

39. GRASP University of Pennsylvania Task 2: Control Of Mobility 1. Design controllers and behaviors in support for communications, establishing or sustaining links 2. Design controllers and behaviors in support for situational awareness 3. Formal analysis of controllers and behaviors to predict team performance

40. GRASP University of Pennsylvania Task 3: Adaptation 1.Performance functions for the ad hoc network and adaptation using reinforcement learning 2.Reconfiguration of network to enable integration and fusion of sensory data in support of human interaction and situational awareness

41. GRASP University of Pennsylvania Task 4: Human Robot Team Interface 1.Synthesis and integration for perception enabling multiple views at different spatio-temporal resolution 2.Interface for human-robot interaction l ROCI l Macroscope

42. GRASP University of Pennsylvania Task 5: Performance Metrics: Verification and Validation 1. Metrics for control, communication, and perception technologies, and performance measures for system performance. l Existing measures do not incorporate the dependence of control, communication and perception 2. Designing and conducting experiments to measure performance

43. GRASP University of Pennsylvania Task 6: Technology Integration 1.Coordinated motion of four UGVs and one blimp optimizing end-to-end network performance 2.Team control, realization of situational awareness using ROCI.

44. GRASP University of Pennsylvania Summary of Tasks Penn GRASP l Integrated model for control, perception and communication for situational awareness l Synthesis and integration for perception enabling multiple views at different spatio- temporal resolution Georgia Tech MRL l Communication-sensitive planning and behavioral control algorithms in support of network-centric warfare l Integrated mission specification system (MissionLab) spanning heterogeneous teams of UAVs and UGVs BBN l Models of QoS and metrics of performance for network-centric warfare l Interface design between network and robot modules l Formulation of FCS needs, capabilities, and design of demonstrations USC RRL l Cooperative outdoor localization for small teams of robots l Semantic mark-up of maps with environmental attributes and recognition of activity. l Behaviors for path-referenced perception and for clandestine operations

45. GRASP University of Pennsylvania MARS TEAMS Impact l New paradigm and novel algorithms for network-centric operations l Mobile nodes that reconfigure to maintain and enhance connectivity l Air-Ground coordination will directly impact FCS capabilities