L C SL C S Supporting Technology for Group Interaction Howard Shrobe MIT AI Lab Oxygen Workshop, January, 2002

L C SL C S Three Motivating Questions Why do organizations repeat their past mistakes? Why do organizations fail to notice impending opportunities and crises? Why do projects never complete on time? Because they fail to capture, organize and disseminate the information which groups exchange in their daily work lives. Because capturing such information is unnatural –Interrupts your actual task –Has value only in the future –Has value to somebody else. Because capturing it costs more than it’s worth! This is ironic because designers want to discuss their creations.

L C SL C S A Three Pronged Attack on Collaborative Knowledge Management Ubiquitous human-centered, perceptually enabled environments. An adaptive infrastructure that understands the context and content of problem solving discourse (at least a little) –It accurately indexes all information. –It distributes information to those who can use it. –It can find resources (information and people) to help a collaboration. Tools informed by an understanding of the domain and the organization. –An “almost expert” (apprentice) system *Which knows when to ask questions: *What it doesn’t understand is often what’s most important *What isn’t obvious is what needs to be documented and distributed Documentation almost for free by embedding a helpful computational system in our normal work flow

L C SL C S Team Based Collaboration Is oriented around shared goals Is opportunistic and information driven Requires information to be distributed to those who need it –And not to those who don’t Involves forming, deliberating about and executing shared plans –Each step of the plan sets the context for how people (should) interact –Plans are specific to the task and the organization

L C SL C S An Example of Group Interaction and Shared Plans Clarify Interface Propose Code Document Brainstor m Critique The side-tracker interface would be much clearer if it worked this way! I think I can make it work that way, but does it affect anything else! That should work, as long as it doesn’t break the special hack for managers. I’ll look into that, you all can start the coding and doc updates Plan

L C SL C S A Meeting Web Video/Audio Transcript (quicktime movie) Position 1 Start time; End time: Position 2 Start time; End time: Discourse Structure Issue 1 Start time; End time: Issue 2 Start time; End time: Supporting Argument Start time; End time: Refuting Argument Start time; End time: Agenda Item Start time; End time: Agenda Item Start time; End time: Agenda Item Start time; End time: Meeting Structure Commitment Start time; End time: Who: Deadline Discourse Commitment Start time; End time: Who: Deadline Discourse Action Items Commitment Start time; End time: Who: Deadline Discourse People

L C SL C S Information nodes: Goals of the project Proposed methods for the goals Arguments in favor and against methods Documents supporting these arguments Node format: Much of the content is opaque to the system (e.g. Multi-media fragments) Some slots are understood by the system And, short natural language annotations can be attached, parsed and understood by START Formal representations are also possible Relational information: Links showing the relationships between the information nodes Link types are meaningful to the system Organizational descriptions: Resource descriptions including capabilities, roles and interests (for personnel) Process goals and plans Meeting Manager Plan KB Goal Plan Knowledge Base Haystacks Knowledge acquisition Facilitator Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery web The Oxygen Collaboration Infrastructure: Knowledge Based Collaboration Webs

L C SL C S Plan KB Goal Plan Knowledge Base Haystacks Knowledge acquisition Facilitator Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery Facilitator Agent A Knowledge-Based Web Helps People Collaborate By acting as an assistant to meeting facilitators it can help to capture important information such as: –Issue, positions, arguments for and against positions –Commitments, action items –Video and audio transcripts of meetings By understanding the role of individuals within the organization it can help to decide who should see what information. By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”.

L C SL C S A Collaboration Web Helps by Making Simple Inferences Oxygen deduces new information using: –The background knowledge base –The types of the links –The slots of node structures that are understood –The semantic content of the node annotations –Descriptions of the resources and people in the organization Oxygen routes information and discussion topics to those whose: –Organizational role requires it –Interests suggest it –Capabilities and skills might be useful Oxygen posts new goals and initiates new discussion processes to address these goals. Oxygen supports human interactions in a manner relevant to the organization’s problem solving context

L C SL C S Oxygen Makes This Much Easier by Bringing Computers Into Our World Microphone Array Tracking Cameras Video Displays Pointing Camera Pointing Camera

Summarizing The Challenges: A Unified Infrastructure For Group Interactions

L C SL C S Metaglue: A Platform for Multi-modal HCI Agents Are the Basic Building Block of Meta-glue An Agent Is a Software Object That Acts on Behalf of Some Entity in the Real World –Agents Are Distributed Objects That Run in Multiple Computers and Virtual Machines and That Communicate by Direct Message Passing –Agents Find One Another Through a Catalog That Provides Resource Discovery Capabilities –Many Basic Agents Control and Represent the State of Devices Agents may communicate directly Agents Are Clustered Into Societies –A Society Is a Collection of Agents –All Agents in a Society Act on Behalf of a Common Real-world Entity Such As a Space or a Person. Events Are Qualitative Changes in Significant Properties of the World Events Are Shared Through a Publish & Subscribe Broadcast Medium

L C SL C S Structuring of Agent Societies Projector agent Resource manager VCR agent Secretary Browser agentSociety agent agent Secretary Resource manager Society agent Conference room societyMy society Catalog

L C SL C S 5 Keys Challenges for Adaptive Interfaces Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time Providing services in a multi-user environment while making optimal use of the currently available resources Recovering from equipment failures, information attacks, etc. Coordinating and fusing information from many sensors and modalities Capitalizing and recognizing context Maintaining Security and Privacy and trading these off against other goals

L C SL C S Challenge 1: Grounding in Real-World Semantics We want to build applications that service many individuals and groups of individuals These people will move among many physical spaces The devices and resources they use change as time progresses The context shifts during interactions The relevant information base evolves as well.

L C SL C S Challenge 1: Grounding in Real-World Semantics We want to build applications that service many individuals and groups of individuals These people will move among many physical spaces The devices and resources they use change as time progresses The context shifts during interactions The relevant information base evolves as well. The system is required to respond dynamically

L C SL C S Research Agenda: Knowledge Representations People –Interests, skills, responsibilities, organizational role Organizations –Members, structure Spaces –Location –Subspaces –Devices and resources Resources Information nodes –Topic area, place in ontology, format Services –Methods, parameter bindings, resource requirements Agents –Capabilities, society, acting on behalf of whom Events –E.g. Person identification, motion into a new region of space, gestures –Qualitative Changes in any of the properties in the KR

L C SL C S Challenge 2: Adaptive Resource Management In most systems, applications are written in terms of specific resources –(e.g. The left projector in Michael’s office, or worse yet, a physical address). This is in conflict with –Portability across physical contexts –Changes in equipment availability across time –Multiple applications demanding similar resources –Need to take advantage of new resources –Need to integrate mobile devices as they migrate into a space –Need to link two or more spaces What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.

L C SL C S Abstract Service Control Parameters User’s Utility Function The binding of parameters has a value to the user Resource 1,1 Resource 1,2 Resource 1,j Each method requires different resources The system selects the method which maximizes net benefit User requests A service with certain parameters Resource Cost Function The resources Used by the method Have a cost Net benefit Each method binds the settings of The control parameters in a different way Method 1 Method 2 Method n Each service can be Provided by several Methods The system adapts by having many plans for each service

L C SL C S Security and Privacy Issues Are Addressed by Factoring in the Cost of Violating a Security or Privacy Policy Resource Cost Function Resource 1,1 Resource 1,2 Resource 1,j Abstract Service User’s Utility Function Net Benefit Method 1 Method 2 Method n Security Policies Challenge 6: Security and Privacy Is Addressed by This Infrastructure

L C SL C S Challenge 3: Robustness and Recovery From Failures The intelligent room renders services by translating them into plans involving physical resources –Physical resources have known failure modes Each plan step accomplishes sub-goals needed by succeeding steps –Each sub-goal has some way of monitoring whether it has been accomplished –These monitoring steps are also inserted into the plan If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure A recovery is chosen based on the diagnosis –It might be as simple as “try it again”, we had a network glitch –It might be “try it again, but with a different selection of resources” –It might be as complex as “clean up and try a different plan”

L C SL C S I need to ask a question of a systems wizard Plan 1: Locate A systems wizard in the E21 Monitor: check that person is still there Turn on the selected projector Monitor: check that projector turned on Project the message Done Monitor: check that the person noticed the message I don’t see light on the screen I see sally by the screen Projector-1 must be broken. We’ll try again, but using Projector-3. Plan Breakdown Recovering From Failures

L C SL C S Frobulate the sidetracker White Board Place Manager Sally is near The Whiteboard If A Person approaches A device ?x and Grammar ?y is relevant to ?x Then Activate Grammar ?y Activate the Drawing Grammar The Drawing Grammar Is Relevant to the Whiteboard Challenge 4: Response within Context The attention of the system should react to what people in the facility are doing. The system should choose its reactions to events based on the context –E.g. if the door opens, turn the lights on, but not if we’re watching a movie Perceptual interpretation should be biased by context –E.g. a person near the White Board, is likely to start drawing

L C SL C S Challenge 5: Perceptual Integration Separate the implementation of perceptual tasks from the uses to which perception is put Some modules advertise the class of “behavioral events” they are capable of recognizing and signaling –These events are organized into a taxonomy –The same event can be signaled by quite different perceptual modules (e.g. both face and voice recognition can localize a person). Other modules register their interest in certain classes of events –Requests at a higher level in the taxonomy subsume lower level events Modules which receive low-level events may register for and collate many different classes of events –They combine these and signal higher-level events Modules may request perceptual services when they are uncertain of their conclusions

L C SL C S Research Agenda: A Dynamic Event Bus For Perceptual Integration Visual Tracker Signal body motion Voice Identification Interested in body motion Signal the location of individuals Face Recognition Interested in face location I signal the location of individuals White Board Context Manager Interested in the location of individuals Signal people approaching the whiteboard Face Spotter Interested in body motion I signal face location Similar to “Blackboard” systems Publishers & Subscribers are “Knowledge Sources” Events are the blackboard data items But Highly distributed Use of Bayesian and Markov Techniques

L C SL C S Integration of Reactive and Goal Directed Processing Service Mapper Goals Plans Resource Allocator Actions Resource Pool Sensory Systems Events Blackboard Context Reactions Access Policies Plan Monitor Diagnosis & Recovery

L C SL C S Plan KB Goal Plan Knowledge Base Haystack s Knowledge acquisition Facilitato r Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery Facilitator Agent Oxygen’s Knowledge-Based Infrastructure Helps People Collaborate: By acting as an assistant to meeting facilitators it can help to capture important information such as rationale By understanding the role of individuals within the organization it can help to decide who should see what information. By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”. By understanding the context within which people act, it can act and react in appropriate ways. By interacting naturally and adaptively it can make knowledge capture “cost less than it’s worth”.