1. Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee Dynamic Human-Robot Teams Engaged in Complex Adversarial Tasks Using Language- Based Communication Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project

2. 2 Carnegie Mellon School of Computer Science Project Goals  Treasure hunt with two or more teams of humans and robots competing to locate target objects as they explore an unknown space  Research Goals/Questions:  Specify team member’s roles and capabilities to perform tasks  Rapidly form ad-hoc heterogeneous teams of robots and humans  Robots and humans executing synchronized action as a team, while communicating via speech  Improve team performance with experience

3. Boeing Human-Robot Teams Project 3 Carnegie Mellon School of Computer Science Alignment with Boeing’s Objectives Our project develops component technologies relevant to:  FCS – Force multiplication for human-robot teams  NASA Code T – Emphasis on robots assisting and augmenting humans in complex tasks  Space activities (Other than Code T)  Automating launch sites  Automating escape systems  Automating maintenance inspections/repair  Ice inspections  Aircraft maintenance  Automating maintenance inspections/repair  Ice inspections  Automated baggage handlers In collaboration with Phillip Koons, Boeing

4. Boeing Human-Robot Teams Project 4 Carnegie Mellon School of Computer Science STP : Overview  “Skills” for low-level control, “Tactics” for single robot, “Plays” for teamwork APPLICABLE offense DONE aborted !offense ROLE 1 pass 3 mark best_opponent ROLE 2 block ROLE 3 pos_for_pass R B 1000 0 receive_pass shoot A ROLE 4 defend_lane

5. Boeing Human-Robot Teams Project 5 Carnegie Mellon School of Computer Science STP: Implementation  Play selection from playbook  Dynamic role assignment  Active tactic determines transition  Execution monitoring  Reward, and adaptation Play 1 Play 2 Play 3 Role 1 Search Retrieve Selection Execution Monitoring, Adaptation Robot 1 Tactic Robot 1 Tactic Robot N Tactic

6. Boeing Human-Robot Teams Project 6 Carnegie Mellon School of Computer Science Relevant Research Interests 1.Methods for pickup teams 2.Extend play-based coordination to distributed execution 3.Effective human-robot teamwork with pickup teams

7. Boeing Human-Robot Teams Project 7 Carnegie Mellon School of Computer Science TraderBots : Overview  Robots are organized as an economy  Team mission is to maximize production and minimize costs  Robots exchange virtual money for tasks to maximize individual profit  System is designed to align local and global profit maximization $ $ $ $ $

8. Boeing Human-Robot Teams Project 8 Carnegie Mellon School of Computer Science TraderBots : Implementation Operator OpTrader Robot 1 Robot 2 Robot 3 X X X X Task 1 Task 2 Task 3 Task 4 Bids RoboTrader Auction Announce and clear auction OpTrader Greedy Auction Announce and clear auction

9. Boeing Human-Robot Teams Project 9 Carnegie Mellon School of Computer Science Relevant Research Interests 1.Human-multi-robot interaction 2.Role assignment for highly heterogeneous teams 3.Improving robustness and adaptivity

10. Boeing Human-Robot Teams Project 10 Carnegie Mellon School of Computer Science Speech : Overview  Communication in mobile environments  Natural spoken language- based interaction  Supporting high semantic content  Communication in multi- participant domains  Human(s) and robots(s) interacting on the same channel

11. Boeing Human-Robot Teams Project 11 Carnegie Mellon School of Computer Science Speech : Implementation  Sphinx ASR engine  Speaker-independent recognition  Speaker-adaptive capability  Phoenix semantic parser  Concept extraction  RavenClaw dialogue engine  Task-based representations  Full mixed-initiative dialogue  Festival/Theta synthesis engine  Rosetta generation engine  Portable/Wearable platforms ROBOT (A) RavenClaw ROBOT (B) RavenClaw Comms Channel... Sphinx Phoenix Theta Rosetta

12. Boeing Human-Robot Teams Project 12 Carnegie Mellon School of Computer Science Relevant Research Interests 1.Developing conversational algorithms for human-robot polylogue 2.Communicating navigational information between humans and robots in unstructured environments 3.Cooperative grounding of objects, locations, and tasks in novel environments

13. Boeing Human-Robot Teams Project 13 Carnegie Mellon School of Computer Science Current Status  Wireless network-based (UDP) communication protocol for human-robot interaction designed and implemented  Closed loop integration between speech-based command system, a Segway robot, and a Pioneer robot accomplished  2-D kinematic simulation tool for testing interface to Pioneer robots  Video demonstration of speech-based command of a Segway robot and a Pioneer robot  Plan for “year 1 scenario” and relevant technology development and integration to accomplish this scenario

14. 14 Treasure Hunt Scenario First Year Objectives

15. Boeing Human-Robot Teams Project 15 Carnegie Mellon School of Computer Science Basic Scenario  2 Segways, 2 Pioneers, and 1 human  Discover and return to base as much treasure as possible within a 20 minute period  Treasure will be identified by landmarks currently used by the Segway soccer team Mapper (M)Seeker (S)Handler (H)Deliverer (D)Leader (L) HumansXXX SegwaysXXX PioneersXXX GatorsXXX AIBOsX

16. Boeing Human-Robot Teams Project 16 Carnegie Mellon School of Computer Science Detailed Scenario  Goal  To combine TraderBots for negotiation and role assignment with plays for synchronization  Low-level software remains independent  Challenges  How do we decide which Play to adopt?  How do we handle sub-teams?  How do we generate “leaders”?

17. Boeing Human-Robot Teams Project 17 Carnegie Mellon School of Computer Science Step 1: Form Sub-Teams Lets form a sub-team Fixed sub-teams for May 2005

18. Boeing Human-Robot Teams Project 18 Carnegie Mellon School of Computer Science Step 2: Command to Search Team A, search area 1 1.Build map

19. Boeing Human-Robot Teams Project 19 Carnegie Mellon School of Computer Science Step 3: Explore and Search 1.Follow1.Go to search area 2.Build map

20. Boeing Human-Robot Teams Project 20 Carnegie Mellon School of Computer Science Step 4: Search 1.Follow and search for treasure 1.Execute search pattern 2.Build map

21. Boeing Human-Robot Teams Project 21 Carnegie Mellon School of Computer Science Step 5: Found! We found it! We are at

22. Boeing Human-Robot Teams Project 22 Carnegie Mellon School of Computer Science Step 6: Recovery Human aided loading

23. Boeing Human-Robot Teams Project 23 Carnegie Mellon School of Computer Science Step 7: Return 1.Follow 2.Unload 1.Return home 2.Unload Human aided unloading

24. Boeing Human-Robot Teams Project 24 Carnegie Mellon School of Computer Science Post Year 1 Goals  Failure detection and recovery  Outdoor environments  Adversaries  Larger more diverse teams  Adaptation and learning