L C SL C S Supporting Technology for Group Interaction Howard Shrobe MIT AI Lab Oxygen Workshop, January, 2002

L C SL C S Three Motivating Questions Why do organizations repeat their past mistakes? Why do organizations fail to notice impending opportunities and crises? Why do projects never complete on time? Because they fail to capture, organize and disseminate the information which groups exchange in their daily work lives. Because capturing such information is unnatural –Interrupts your actual task –Has value only in the future –Has value to somebody else. Because capturing it costs more than it’s worth! This is ironic because designers want to discuss their creations.

L C SL C S A Three Pronged Attack on Collaborative Knowledge Management Ubiquitous human-centered, perceptually enabled environments. An adaptive infrastructure that understands the context and content of problem solving discourse (at least a little) –It accurately indexes all information. –It distributes information to those who can use it. –It can find resources (information and people) to help a collaboration. Tools informed by an understanding of the domain and the organization. –An “almost expert” (apprentice) system *Which knows when to ask questions: *What it doesn’t understand is often what’s most important *What isn’t obvious is what needs to be documented and distributed Documentation almost for free by embedding a helpful computational system in our normal work flow

L C SL C S Team Based Collaboration Is oriented around shared goals Is opportunistic and information driven Requires information to be distributed to those who need it –And not to those who don’t Involves forming, deliberating about and executing shared plans –Each step of the plan sets the context for how people (should) interact

L C SL C S An Example of Group Interaction and Shared Plans Clarify Interface Propose Code Document Brainstor m Critique The side-tracker interface would be much clearer if it worked this way! I think I can make it work that way, but does it affect anything else! That should work, as long as it doesn’t break the special hack for managers. I’ll look into that, you all can start the coding and doc updates Plan

L C SL C S KB Goal Plan Knowledge Base Haystacks Knowledge acquisition Facilitator Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery Facilitator Agent Knowledge-Based Infrastructure Helps People Collaborate: By acting as an assistant to meeting facilitators it can help to capture important information such as: –Issue, positions, arguments for and against positions –Commitments, action items –Video and audio transcripts of meetings By understanding the role of individuals within the organization it can help to decide who should see what information. By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”.

L C SL C S Information nodes: Goals of the project Proposed methods for the goals Arguments in favor and against methods Documents supporting these arguments Node format: Much of the content is opaque to the system (e.g. Multi-media fragments) Some slots are understood by the system And, short natural language annotations can be attached, parsed and understood by START Formal representations are also possible Relational information: Links showing the relationships between the information nodes Link types are meaningful to the system Organizational descriptions: Resource descriptions including capabilities, roles and interests (for personnel) Process goals and plans Meeting Manager Plan KB Goal Plan Knowledge Base Haystacks Knowledge acquisition Facilitator Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery web The Oxygen Collaboration Infrastructure: Knowledge Based Collaboration Webs

L C SL C S Oxygen Can Support Human Collaboration by Making Simple Inferences Oxygen deduces new information using: –The background knowledge base –The types of the links –The slots of node structures that are understood –The semantic content of the node annotations –Descriptions of the resources and people in the organization Oxygen routes information and discussion topics to those whose: –Organizational role requires it –Interests suggest it –Capabilities and skills might be useful Oxygen posts new goals and initiates new discussion processes to address these goals. Oxygen supports human interactions in a manner relevant to the organization’s problem solving context

L C SL C S Key Oxygen Components Make This Much Easier by Bringing Computers Into Our World Microphone Array Tracking Cameras Video Displays Pointing Camera Pointing Camera

L C SL C S A Meeting Web Video/Audio Transcript (quicktime movie) Position 1 Start time; End time: Position 2 Start time; End time: Discourse Structure Issue 1 Start time; End time: Issue 2 Start time; End time: Supporting Argument Start time; End time: Refuting Argument Start time; End time: Agenda Item Start time; End time: Agenda Item Start time; End time: Agenda Item Start time; End time: Meeting Structure Commitment Start time; End time: Who: Deadline Discourse Commitment Start time; End time: Who: Deadline Discourse Action Items Commitment Start time; End time: Who: Deadline Discourse People

L C SL C S 5 Keys Challenges for Adaptive Interfaces Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time Providing services in a multi-user environment while making optimal use of the currently available resources Recovering from equipment failures, information attacks, etc. Coordinating and fusing information from many sensors and modalities Capitalizing and recognizing context Maintaining Security and Privacy and trading these off against other goals

L C SL C S Challenge 1: Grounding in Real-World Semantics We want to build applications that service many individuals and groups of individuals These people will move among many physical spaces The devices and resources they use change as time progresses The context shifts during interactions The relevant information base evolves as well.

L C SL C S Research Agenda: Knowledge Representations People –Interests, skills, responsibilities, organizational role Organizations –Members, structure Spaces –Location –Subspaces –Devices and resources Resources Information nodes –Topic area, place in ontology, format Services –Methods, parameter bindings, resource requirements Agents –Capabilities, society, acting on behalf of whom Events –People identification, motion in new space, gestures

L C SL C S Challenge 2: Adaptive Resource Management In most systems, applications are written in terms of specific resources –(e.G. The left projector in michael’s office, or worse yet physical address). This is in conflict with –Portability across physical contexts –Changes in equipment availability across time –Multiple applications demanding similar resources –Need to take advantage of new resources –Need to integrate mobile devices as they migrate into a space –Need to link two or more spaces What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.

L C SL C S Abstract Service Control Parameters User’s Utility Function The binding of parameters has a value to the user Resource 1,1 Resource 1,2 Resource 1,j Each method requires different resources The system selects the method which maximizes net benefit User requests A service with certain parameters Resource Cost Function The resources Used by the method Have a cost Net benefit Each method binds the settings of The control parameters in a different way Method 1 Method 2 Method n Each service can be Provided by several Methods The system adapts by having many plans for each service

L C SL C S Security and Privacy Issues Are Addressed by Factoring in the Cost of Violating a Security or Privacy Policy Resource Cost Function Resource 1,1 Resource 1,2 Resource 1,j Abstract Service User’s Utility Function Net Benefit Method 1 Method 2 Method n Security Policies Challenge 6: Security and Privacy Is Addressed by This Infrastructure

L C SL C S Challenge 3: Robustness and Recovery From Failures The intelligent room renders services by translating them into plans involving physical resources –Physical resources have known failure modes Each plan step accomplishes sub-goals needed by succeeding steps –Each sub-goal has some way of monitoring whether it has been accomplished –These monitoring steps are also inserted into the plan If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure A recovery is chosen based on the diagnosis –It might be as simple as “try it again”, we had a network glitch –It might be “try it again, but with a different selection of resources” –It might be as complex as “clean up and try a different plan”

L C SL C S I need to ask a question of a systems wizard Plan 1: Locate A systems wizard in the E21 Monitor: check that person is still there Turn on the selected projector Monitor: check that projector turned on Project the message Done Monitor: check that the person noticed the message I don’t see light on the screen I see sally by the screen Projector-1 must be broken. We’ll try again, but using Projector-3. Plan Breakdown Recovering From Failures

L C SL C S Frobulate the sidetracker White Board Place Manager Sally is near The Whiteboard If A Person approaches A device ?x and Grammar ?y is relevant to ?x Then Activate Grammar ?y Activate the Drawing Grammar The Drawing Grammar Is Relevant to the Whiteboard Challenge 4: Exploitation of Context The attention of the system should be focused by what people in the facility do and say. Speech recognition should be biased in favor of things going on in the facility –Speech system is made up of many “ grammar fragments” –Grammar fragments are activated (and deactivated) when perceptual events (visual or speech) suggest they should be Visual Interpretation should also be biased by context

L C SL C S Challenge 5: Perceptual Coordination Separate the implementation of perceptual tasks from the uses to which perception is put All perceptual modules advertise the class of “behavioral events” they are capable of recognizing and signaling –These events are organized into a taxonomy –Some events are “ close to the physics” (e.g. an object was observed at location ) –Some events are more abstract (e.g. a person is near the white- board) –The same event can be signaled by quite different perceptual modules (e.g. both face and voice recognition can localize a person). Other modules register their interest in certain classes of events –Requests at a higher level in the taxonomy subsume lower level events Modules which receive low-level events may register for and collate many different classes of events –They combine these and signal higher-level events

L C SL C S Research Agenda: A Dynamic Event Bus For Perceptual Coordination Visual Tracke r Signal body motion Voice Identification Interested in body motion Signal the location of individuals Face Recognition Interested in face location I signal the location of individuals White Board Context Manager Interested in the location of individuals Signal people approaching the whiteboard Face Spotter Interested in body motion I signal face location

L C SL C S Plan KB Goal Plan Knowledge Base Haystacks Knowledge acquisition Facilitator Shared Plans Shared Info Shared Semantic Web Inference Resource Discovery Facilitator Agent Oxygen’s Knowledge-Based Infrastructure Helps People Collaborate: By acting as an assistant to meeting facilitators it can help to capture important information such as: –Issue, positions, arguments for and against positions –Commitments, action items –Video and audio transcripts of meetings By understanding the role of individuals within the organization it can help to decide who should see what information. By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”. By understanding the context within which people act, it can react in appropriate ways.