L C SL C S Reactive and Responsive Intelligent Environments Kevin Quigley aire group MIT AI Lab

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Reactive and Responsive Environments We are trying to build pervasive, perceptually enabled human- centered environments Such an environment must respond in reasonable ways to high level requests from its users. –It should be up to the system to figure out a reasonable way to implement the request, translating goals to plans that meet the users needs and that utilize available resources. –E.g. I might ask to light the room up; the system responds by opening the drapes. Such an environment should react to events in the environment even when there is no explicit user request. –E.g. when I walk into my room in the morning, the system should light up the room. Reactions and responses should both be contextually sensitive. Both must show human levels of adaptivity

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab 5 Challenges 1. Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time 2. Providing run-time composable services in a multi-user environment that make optimal use of the currently available resources 3. Recovering from equipment failures, information attacks, misestimates of sensors, etc. 4. Coordinating and fusing information from many sensors and modalities 5. Capitalizing on and recognizing context (task, location, personal style & state) 6.Maintaining security and privacy and trading these off against other goals

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Challenge 1: Grounding in Real-World Semantics We want to build applications that service many individuals and groups of individuals These people will move among many physical spaces The devices and resources they use change as time progresses The context shifts during interactions The relevant information base evolves as well. The system is required to respond dynamically

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Research Agenda: Knowledge Representations People –Interests, skills, responsibilities, organizational role Organizations –Members, structure Spaces –Location –Subspaces –Devices and resources Resources Information nodes –Topic area, place in ontology, format Services –Methods, parameter bindings, resource requirements Agents –Capabilities, society, acting on behalf of whom Events –E.g. Person identification, motion into a new region of space, gestures –Qualitative Changes in any of the properties in the KR

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Challenge 2: Adaptive Resource Management In most systems, applications are written in terms of specific resources –(e.g. The left projector in Michael’s office, or worse yet, a physical address). This is in conflict with –Portability across physical contexts –Changes in equipment availability across time –Multiple applications demanding similar resources –Need to take advantage of new resources –Need to integrate mobile devices as they migrate into a space –Need to link two or more spaces What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab An Example of Service Requests When I come into my office in the morning it’s dark –The lights are out –The drapes are closed I ask the office to light up the room It’s a sunny day, it opens the drapes –If I had asked it to turn on the lights, it wouldn’t have opened the drapes It’s a cloudy day, it turns on the lights

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Responsive, Goal-Directed Processing Goals Service Mapper Plans Resource Allocator Resource Pool Actions Service mapping is provided by the resource management component

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Abstract Service Control Parameters User’s Utility Function The Binding of Parameters Has a Value to the User Resource 1,1 Resource 1,2 Resource 1,j Each Plan Requires Different Resources The System Selects the Plan Which Maximizes Net Benefit User Requests A Service With Certain Parameters Resource Cost Function The Resources Used by the Method Have a Cost Net Benefit Each Method Binds the Settings of The Control Parameters in a Different Way Plan 1 Plan 2 Plan n Each Service Can Be Provided by Several Plans Services are Dynamically Mapped to Plans

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Challenge 3: Robustness and Recovery From Failures Breakdowns are inevitable –Resources sometimes fail while being used –The system acts on sensor data which has uncertainty The system renders services by translating them into plans A plan-monitor watches over the execution of a plan. Each plan step accomplishes sub-goals needed by succeeding steps –Each sub-goal has some way of monitoring whether it has been accomplished –These monitoring steps are also inserted into the plan If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure A recovery is chosen based on the diagnosis –It might be as simple as “try it again”, we had a network glitch –It might be “try it again, but with a different selection of resources” –It might be as complex as “clean up and try a different plan”

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab I need to ask a question of a systems wizard Plan 1: Locate a systems wizard in the E21 Monitor: check that person is still there Turn on the selected projector Monitor: check that projector turned on Project the message Done Monitor: check that the person noticed the message I don’t see light on the screen I see Sally by the screen Projector-1 must be broken. We’ll try again, but using Projector-3. Plan Breakdown The Plan Monitor Manages Recovery From Failures

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Making the System Responsible for Achieving Its Goals Scope of Recovery Selection of Alternative Localization & Characterization Diagnostic Service Repair Plan Selector Resource Allocator Concrete Repair Plan Resource Plan alerts Plan Monitor Rollback Designer Enactment achieves requires Step- A Step- B Condition-1 prerequisite Service Request Plan-for A plan is a partially ordered collection of steps Each step achieves a subgoal Some steps establish pre-requisites for others

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Self-Adaptive Goal Directed Processing Service Mapper Goals Plans Resource Allocator Actions Resource Pool Plan Monitor Diagnosis & Recovery

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Challenge 4: Context Awareness The context should influence how the system behaves: –Task Structure –Location –Emotional State –Personal Style Perception can help determine the context The system should choose its reactions to events based on the context Perceptual interpretation should be biased by context –E.g. a person near the White Board, is likely to start drawing Estimation of utility should be influenced by context

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab A Reactive System Responds to Events If somebody enters the room Then illuminate the room

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Structure of REBA A Reaction maps an event to a goal Reactions are grouped into behavioral bundles –Sets of reactions that are always activated as a unit The Context is determined by the task within a plan –As well as location, people present,... Each context activates a set of behavioral bundles Contexts have Sub-Contexts, activities that occur within other activities –Watching a video within a meeting The active reactions of a sub-context override the reactions of the parent context Sub-context-1 Sub-context-2 Context Stack Reactions Events Goal1 Goal2

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Context Awareness Conditions Reactions If somebody enters the room Then illuminate the room But not if a movie is being watched

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Context Sensitivity Events Context Reactive Manager Service Mapper Goals Plans Resource Allocator Actions Resource Pool Plan Monitor Diagnosis & Recovery Reactive Processing is provide by the REBA MetaGlue Component

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Challenge 5: Perceptual Integration We want to separate the implementation of perceptual tasks from the uses to which perception is put Some modules advertise the class of “behavioral events” they are capable of recognizing and signaling –These events are organized into a taxonomy –The same event can be signaled by quite different perceptual modules (e.g. both face and voice recognition can localize a person). Other modules register their interest in certain classes of events –Requests at a higher level in the taxonomy subsume lower level events Modules which receive low-level events may register for and collate many different classes of events –They combine these and signal higher-level events Modules may request perceptual services when they are uncertain of their conclusions

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab A Dynamic Event Bus For Perceptual Integration Visual Tracker Signal Body Motion Voice Identification Interested in Body Motion Signal the Location of Individuals Face Recognition Interested in Face Location I Signal the Location of Individuals White Board Context Manager Interested in the Location of Individuals Signal People Approaching the Whiteboard Face Spotter Interested in Body Motion I Signal Face Location A “Blackboard” System Publishers & Subscribers Are “Knowledge Sources” Events Are the Blackboard Data Items Highly Distributed Use of Bayesian Techniques

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Blackboards: Using Context phonemic intent acoustic Syllable sentence word R eck Og NI Z RecOgIZE Recognize Speech SP Context: Discussion of Oil Tanker Crash Fragility of Environment Wreck A Nice Beach Wreck A NICE Wreck A NIS It’s not Hard to Wreck a Nice Beach

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Integration of Reactive and Goal Directed Processing Sensory Systems Blackboard Events Context Reactions Service Mapper Goals Plans Resource Allocator Actions Resource Pool Plan Monitor Diagnosis & Recovery

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Summary Responsive processing dynamically maps goals to plans –Plans are selected by balancing the benefit to the user against the cost of resources –Plan monitoring recovers from plan breakdown –Access control is handled as part of the cost benefit analysis Reactive processing dynamically maps event to goals –Events are handled within context –Perceptions are maps to events within context

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Reactive and Goal Directed Processing Sensory Systems Blackboard Events Context Reactions Service Mapper Goals Plans Resource Allocator Actions Resource Pool Access Policies Plan Monitor Diagnosis & Recovery with Security

L C SL C S The Metaglue System Software agents for intelligent spaces

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Computational needs Smart Environments –Distributed Components –Dynamic Changes –Highly Varied Equipment –New Modes of Interaction –Frequent Failures –Abundance of Information Metaglue –Communication –Resource Management –Customization –Multi-modal HCI –Agent Recovery –Persistent Storage

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab The Metaglue System What is it? –A communication infrastructure for building systems of distributed software agents –A software architecture for creating adaptive applications for Intelligent Environments Why for Intelligent Environments? –dynamic adaptation of the applications based on *the availability of resources (hardware/software) in the current system/environment *security controls of the participants *preferences of the participants

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Metaglue Software Infrastructure What do you get with Metaglue? –Multi-modal Human-Computer Interaction *Spoken Natural Language *Perceptual (Vision) *Direct manipulation (Graphical) –Persistent storage –Multiple communication paths *Remote procedure call (Java RMI) *Publish/Subscribe message passing –Configurable setting for asserting preferences –Start on demand of Agents –Automatic recovery / handling of direct communication errors –Resource Management and Service mapping

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab What Are Software Agents? Our definition: A software agent is any software object capable of communication by exposing functionality to other agents running within the network. Metaglue is a Multi-Agent System where agents perform individually specialized, (usually) simple tasks but connect in a web of intercommunication to cooperate on more complex tasks.

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab More than Java Objects Metaglue Agents can be as complex or simple as necessary –Larger programs can be used in Java thru JNI or other interface *ViaVoice, ASSIST (a sketching tool) –Most Metaglue Agents are simpler and single-purpose *Projector display or light control –Complex agents are those that control other simpler agents *ReBa – Reactive Behavioral System *SPIe – Self-adaptive Plan-based Intelligent Environment Metaglue provides to agents –identity and occupation –the entity this agent represents –location and the ability to change locations –intrinsic communication to other agents

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab What are some of the agents in Metaglue? Device Control –lights, drapes, fans, sensors –DVD & CD players, MP3 –video multiplexers, projectors –Cameras, TVs, VCRs, Audio Data Organization Systems –Blackboard agents – data flow –START system – NLP & KB –Presence & Location agents –Newswall – visual data organization and presentation Agent Systems Applications –ReBa – interactive behaviors –SPIe and Planlet – plan monitoring –Web info display Debugging and Logging –Agent testing, Simulated devices –Log & Catalog Monitors –Notification listeners Recreation –Checkers, Boggle, RPG, HexaCheckers, Crosswords, ELIZA clone

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab System Organization - Societies Clusters of agents that operate on behalf of a real-world entity (space or person) are called Societies Societies allow the same agent to exist with different customized attributes. Agents can talk to agents in other societies as easily as their own society Societies look like agents when viewed from the outside –They exposed functionality to higher level resource management through Hyperglue

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab System Organization - Catalog The Catalog Agent is the central component which knows about all running agents

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab System Organization - MetaglueVM The Metaglue Virtual Machine (MVM) runs as a base platform for all other agents in the Metaglue system. –It handles all registration of the agent with the Catalog Agent –Provides methods for direct communication (RMI) to other agents on the current catalog To Catalog

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Calling on Other Agents – agent present 1.A reliesOn call by Agent B will create a chain of events to get the RMI stub for Agent A 2.The MVM takes the call and passes it to the Catalog Agent 3.The Catalog checks the internal table of agents to see if there is one matching the requested description 1.If the agent exists, the RMI stub for Agent A is returned

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Calling on Other Agents – agent not present 3. The Catalog Agent checks the internal table of agents to see if there is one matching the requested description 2.If the agent does not exist, it must be started locally (calling MVM) 3.The new Agent A registers its stub 4.That stub is then returned to Agent B

L C SL C S Automatic Agent Recovery Failure recovery through proxies

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Without error handling

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Without error handling

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Without error handling

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Without error handling

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Error handling with proxy objects

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Error handling with proxy objects

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Error handling with proxy objects

Oxygen Alliance Annual Meeting — February 25 – 28, 2003 Kevin Quigley — MIT Artificial Intelligence Lab Statistics on Metaglue 10 Tons of fun: –There are over 450 agents that exist within Metaglue –Between 50 and 80 agents are running the intelligent room –You are using more than 10 agents just while running the X10BasicLightControl *Test it! Use agentland.util.LogMonitor Metaglue has been in development since 1998 The system is used in several offices and homes including the office of the AI lab director, Rodney Brooks There are 2 full spaces at MIT (a 3 rd is coming soon!) and one space in Australia running Metaglue –Why not get your own?