Integrating Intelligent Assistants into Human Teams Katia Sycara The Robotics Institute Carnegie Mellon University Pittsburgh, PA (412) Michael Lewis School of Information Sciences University of Pittsburgh Pittsburgh, PA (412)

Team Members CMU Liren Chen Somesh Jha Rande Shern Dajun Zeng Keith Decker Anadeep Pannu Vandana Verma Prasad Chalasani Kostya Domashnev Onn Shehory

Team Members U. of Pittsburgh Michael Lewis (PI) Terry Lenox Emily Roth

Talk Outline Goals Potential Impact for the Navy Approach Research Issues Progress Plan for Next Year

Overall Research Goal Increase the effectiveness of joint Command and Control Teams through the incorporation of Agent Technology in environments that are: distributed time stressed uncertain open (information sources, communication links and agents dynamically appear and disappear) Team members are distributed in terms of: time and space expertise

Impacts for Navy Reduce time for a C2 team to arrive at a decision Allow C2 teams to consider a broader range of alternatives Enable C2 teams to flexibly manage contingencies (replan, repair) Reduce time for a C2 team to form a shared model of the situation Reduce individual and team errors Support team cohesion and team work skills Increase overall team performance

Transition Opportunities Maritime Crisis planning Target identification training Air campaign planning Strike planning Aircraft maintenance

Overall Approach develop an adaptive, self-organizing collection of Intelligent Agents (the RETSINA infrastructure) that interact with the humans and each other. –integrate multimedia information management and decision support –anticipate and satisfy human information processing and problem solving needs –perform real-time synchronization of human actions –notify about significant changes in the environment –adapt to user, task and situation develop model libraries of individual and team tasks develop verifiable useful human-agent interaction techniques

Overall Research Issues Agents and Agent Interactions Human Agent Interaction Information Filtering and Integration

Overall Research Issues: Agents and Agent Interactions interleaving planning, replanning, execution monitoring and information gathering in a multiagent setting single agent architecture and self-awareness agent coordination scheme finding appropriate agents agent interoperability agent-to-agent task delegation protocols learning through agent interactions

Overall Research Issues: Human Agent Interaction agent-based team aiding functional allocation between humans and agents (insert agents into military simulations and perform controlled experiments with human subjects to assess utility) human-agent trust development of task models (graphical task editor) user-guided instantiation of agents (agent editor)

Insert TeamAiding.ppt

Overall Research Issues: Information Filtering and Integration learning and tracking multiple interests of users increase relevance of retrieved information (refinement key words, relevance feedback, summary of most important information in documents) detecting ``interesting'' patterns from multiple data sources information integration and conflict resolution

Retsina Functional Organization

Characteristics of RETSINA Agents Agents act autonomously to accomplish objectives –Goal-directed –Taskable –Running unassisted for long periods –Proactive & Reactive

Characteristics of RETSINA Agents (Contd.) Agents engage in peer-to-peer interactions –Agents are taskable, i.e. users or other agents can delegate tasks to them, user acceptability and trust an important issue –Can interact as cooperative teams or self-interested individuals –Interaction protocols –Coordination Strategies –Negotiation Protocols Agents adapt to their environment, user, task and each other –Adapt both at the individual level and at the societal level –Employ Alternate Methods –Learn from (and about) users and each other

Progress RETSINA system infrastructure development –Java implementation RETSINA agent architecture –increased planning sophistication in individual agents Middle agents Agent interaction protocols

Middle Agent Types Service Parameters Initially Known By Service providers have capabilities and service parameters Service requesters have service request and preferences

Retsina Agent Architecture

RETSINA Planning Mechanisms hierarchical task network-based formalism library of task reduction schemas –alternative task reductions –contingent plans, loops incremental task reduction, interleaved with execution –information gathered during execution directs future planning resource and temporal constraints

A task Structure (Advertisement Task Structure)

Progress (Contd.) Agent interoperability –language for capability advertisement (Aardvark) –agent name server and distributed matchmakingª Human Agent Interaction –Task Editor –Agent Editor –Human Agent Trust –Team TANDEM experiments ________________________ ª

Insert Aardvark.ppt: language for capability advertisement

Insert Interact.ppt: Agent Editor

Progress (Contd.) Applications –Information filtering: Webmateª, DVINA –Agents in team aiding: ModSAF, multiagent air patrol, agent- aided aircraft maintenance  ___________________________ ª  This application is done in collaboration with the CMU wearable computer project.

ModSAF Vision

Insert AirMain.ppt: Aircraft Maintenance Task

Overview of the WebMate System Use the multiple TF-IDF vectors to keep track of user interests in different domains which are automatically learned Use the trigger pair model to automatically extract relevant words for refining search The user can provide multiple pages as relevance guidance for information search

Insert WebMate.ppt (more detailed description)

Insert WebMateDemo.ppt (detailed description of WebMate demo)

Overview of Informedia One of the six Digital Libraries Initiative projects funded by the NSF, DARPA, NASA and others in collaboration with WQED A multimedia library that will consist of over one thousand hours of digital video, audio, images, text and other related materials Uses combined speech, language and image understanding technology to transcribe, segment and index the linear video.

Plans for Next Year Continue enhancing the functionality of individual agents (e.g., more sophisticated planning mechanisms) Improve the robustness of the RETSINA infrastructure Finish the implementation of the agent advertisement language (Aardvark) Refine agent task delegation framework, particularly contingent task delegation Investigate situation-dependent agent coordination strategies Investigate information- and action-based conflict resolution Expand the ModSAF team-aiding scenarios by introducing agents of additional types and functionalities

Plans for Next Year (Contd.) Develop explicit agent tasking mechanisms Identify appropriate indexing mechanisms for task structure cases Expand the functionalities of agent editor Automatically learn individual and team coordination patterns from team activity traces

Plan for Integrating the Parts of CMU MURI Work with U. of Pittsburgh to identify additional agent requirements needed for agent-based team aiding U. of Pittsburgh will test the effectiveness of agent-based team aiding in ModSAF scenarios with human subjects Incorporate multimedia information from Informedia into agent- based team aiding Use the wearable computers as the platform for running the collaborative aircraft maintenance agents