1. Building and Using Practical Agent Applications SRI International David Martin Adam Cheyer David Martin Adam Cheyer PAAM ’98 Tutorial

2. SRI International 3/23/98 PAAM ’98 Tutorial Contents n Context: Agents & Distributed Computing n Challenges & Opportunities n Inside the Open Agent Architecture n Example Systems & Useful Techniques n Concluding Remarks

3. SRI International 3/23/98 PAAM ’98 Tutorial n Context: Agents & Distributed Computing u Areas of Agent Research u Evolving Paradigms for Distributed Systems u SRI’s Open Agent Architecture n Challenges & Opportunities n Inside the Open Agent Architecture n Example Systems & Useful Techniques n Concluding Remarks

4. SRI International 3/23/98 PAAM ’98 Tutorial What Is an Agent? Mobile Agents Programs that move among computer hosts Autonomous Agents Based on planning technologies Learning Agents User preferences, collaborative filtering,... Animated Interface Agents Avatars, chatbots,... Simulation-based Entities Data/Info finding, filtering and merging Cooperative Agents Cooperation among distributed heterogeneous programmatic components Examples Voyager, Aglets, Odyssey Robots, Softbots, BDI Microsoft Agent, Julia ModSAF, RoboCup OAA, KQML, FIPA FireFly, MIT Media Lab SIMS, InfoSleuth, IR

5. SRI International 3/23/98 PAAM ’98 Tutorial Approaches to Building Applications MonolithicApplicationsObject-OrientedApplications Distributed Object Applications OAAApplications Dynamic addition Objective Suitable for Internet environment Virtual community of dynamic services Adaptable to changing, evolving network resources Flexible interactions among components

6. SRI International 3/23/98 PAAM ’98 Tutorial Approaches to Distributed Computing n Mobile Objects n Blackboard Architectures n Agent Communication Languages (ACL) n Publish & Subscribe Brokers

7. SRI International 3/23/98 PAAM ’98 Tutorial Mobile Objects (Agents) n Objects move under their own power (e.g., Voyager, Aglets) n Advantages u Network bandwidth for certain classes of problems u Parallelism - many objects can be spawned n Disadvantages u Programmatically specify where to go and what to do, through a known interface u Little automated support for inter-object cooperation u Programming language specific (non-heterogeneous)

8. SRI International 3/23/98 PAAM ’98 Tutorial Blackboard Architectures n Knowledge Sources read and write tuples from a common information space (e.g. LINDA, FLiPSiDE) n Advantages u Eliminates explicitly programmed interactions among participants n Disadvantages u KS cannot coordinate interactions u Polling tuple(abc,1,2,3)

9. SRI International 3/23/98 PAAM ’98 Tutorial Publish & Subscribe Brokers n Clients register interest, broker routes/filters msgs n Examples: Talarian SmartSockets, Active Software’s ActiveWeb, ACL Brokers n Advantages u Destination process(es) not explicitly encoded u No polling n Disadvantages u Simple filtering, unitary messages Broker

10. SRI International 3/23/98 PAAM ’98 Tutorial Agent Communication Languages n Communication message types based on speech acts (e.g., ask, tell, deny) + conversational policies n Examples: FIPA ACL, KQML n Advantages u Rich interaction model, peer-to-peer based u Standardized message types, content-agnostic n Disadvantages u Conformance to specs not universal u Explicitly coded interactions among participants ANS, Service Broker Ask Reply

11. SRI International 3/23/98 PAAM ’98 Tutorial Comparison of Distributed Approaches

12. SRI International 3/23/98 PAAM ’98 Tutorial Overview of the OAA OAA: A framework for integrating a community of software agents in a distributed environment Facilitates flexible, adaptable interactions among distributed components through delegation of tasks, data requests & triggers Enables natural, mobile, multimodal user interfaces to distributed services Definition What, not how or who Distributed Computing Through Delegation: User Interface

13. SRI International 3/23/98 PAAM ’98 Tutorial OAA Architecture Facilitator Agent Modality Agents Application Agent Application API Meta Agent Registry NL to ICL Agent User Interface Agent Interagent Communication Language

14. SRI International 3/23/98 PAAM ’98 Tutorial Automated Office Application Main Points Mobile access to distributed services Legacy applications interacting with AI technologies Flexible interactions among components High-level tasking of agents through NL and speech Delegated Triggers

15. SRI International 3/23/98 PAAM ’98 Tutorial OAA Characteristics Open: Extensible: Distributed: Parallel: Mobile: High-level: Multimodal: agents can be created in many languages and interface with existing systems agents can be added or replaced dynamically agents are spread across many computers Parallel execution of subtasks Lightweight interfaces on phone and/or PDA hides software and hardware dependencies handwriting, speech, gestures, and direct manipulation can be combined together

16. SRI International 3/23/98 PAAM ’98 Tutorial OAA-based Applications 1. Automated Office 2. Unified Messaging 3. Multimodal Maps 4. CommandTalk 5. ATIS-Web 6. Spoken Dialog Summarization 7. Agent Development Tools 8. InfoBroker 9. Rental Finder 10. InfoWiz Kiosk 11. Multi-Robot Control 12. MVIEWS Video Tools 13. MARVEL 14. SOLVIT 15. Surgical Training 16. Instant Collaboration 17.Crisis Response 18. WebGrader 19. Speech Translation 20-25+...

17. SRI International 3/23/98 PAAM ’98 Tutorial n Context: Agents & Distributed Computing n Challenges & Opportunities u Interoperation u Coordination & Control u Information Management and Sharing u Intelligent User Interfaces n Inside the Open Agent Architecture n Example Systems & Useful Techniques n Concluding Remarks

18. SRI International 3/23/98 PAAM ’98 Tutorial Interoperation

19. SRI International 3/23/98 PAAM ’98 Tutorial Interoperation n Language, Ontology, Conversational Protocol n Imposing the Right Amount of Structure n Legacy & “Owned-elsewhere” Applications u Multi-platform, Multi-language u Wrappers & Surrogates u Backwards Compatibility With Older Paradigms n Integration with Standards n Opportunities u Support Greater Flexibility & Dynamism in Structuring Communities & Interactions u Provide Economical Means of Coding Interactions u Leverage Our Understanding Of Conversations u Minimize Platform & Language Barriers

20. SRI International 3/23/98 PAAM ’98 Tutorial Coordination & Control n No Fully General Solutions Available n Families of C & C Strategies u Knowledge-Sharing u Team Coordination u Economic (Market-Driven) u Evolutionary n Opportunities u Flexibility, Synergy u Advice and Constraints u Temporal Control u Sophisticated Facilitation, Reactive Execution

21. SRI International 3/23/98 PAAM ’98 Tutorial Alternative Agent Control Strategies n Knowledge-Sharing u Agents share knowledge about capabilities and requests. u Agent brokers dynamically match requests to capabilities. u System dynamically adjusts as capabilities are added to and removed from the environment. n Team Coordination u Agents share knowledge about goals, plans, tasks & subtasks, commitments and performance. u Teams cooperative through partially synchronized actions to accomplish individual subtasks and common goals. n Market-Driven Economy u Self-interested agents pursue personal profit. u Behavior is driven by the cost of resources. u Agents are controlled by specifying market rules, rewards and penalties. n Evolutionary Systems u Agents populations evolve over time through “reproduction”, mutation and natural selection. u Agents are controlled by specifying selection criteria and reproduction process.

22. SRI International 3/23/98 PAAM ’98 Tutorial Complexity vs. Number of Agents Applicability of Strategies Coordination and Control Strategies u Knowledge-Sharing u Team Coordination u Market-Driven Economy u Evolutionary Systems The strategies differ in the complexity and number of agents for which they are suited to control 10 1 - 10 3 10 5 - 10 7 KnowledgeSharing Number of agents in system Complexity of individual agents TeamCoord.MarketDrivenEvolutionarySystem low high

23. SRI International 3/23/98 PAAM ’98 Tutorial Information Management and Sharing n External Data u Heterogeneous, Dynamic, Unreliable Sources n Operational Data u Maintaining Consistent World-views u Transactions, Snapshots, Roll-back n Sharing Strategies u How Much to Share, Cost of Sharing u Support for Collaboration n Opportunities u Tight Integration With Service-providing & Requesting Mechanisms u Built-in Support for Handling Dynamism u Use Intelligence, Autonomy to Address Reliability

24. SRI International 3/23/98 PAAM ’98 Tutorial Intelligent User Interfaces n Make User Requests Comprehensible to System n Make System Results Comprehensible to User n Help User Understand System Complexity … u Multiple Autonomous Actors u Dynamic Communities n … Or Not Be Required to n Opportunities u Agent-based Approaches to UI Implementation u Integrate Multimodality u User As Privileged Member of Agent Community u Use of Mixed-initiative Interactions u Collaboration

25. SRI International 3/23/98 PAAM ’98 Tutorial n Context: Agents & Distributed Computing n Opportunities & Challenges n Inside the Open Agent Architecture n Example Systems & Useful Techniques n Concluding Remarks

26. SRI International 3/23/98 PAAM ’98 Tutorial OAA Architecture Facilitator Agent Modality Agents Application Agent Application API Meta Agent Registry Support Agent User Interface Agent Interagent Communication Language

27. SRI International 3/23/98 PAAM ’98 Tutorial Interagent Communication Language n Used by Agents to: u Declare Capabilities u Request Services of Community u Respond to Requests from Other Agents u Manage and Exchange Information n Conversation & Content Layers n Advice/Constraints Can Accompany Requests n Platform- and Language-Independence

28. SRI International 3/23/98 PAAM ’98 Tutorial Providing Services n Declaring Capabilities u solvable(Goal, Parameters, Permissions) n Examples of Parameters u type: {data, procedure} u private: Boolean u utility: [0.. 10] solvable(send_message(email, +ToPerson, +Params), [type(procedure), callback(send_mail)], []), solvable(last_message(email, -MessageId), [type(data), single_value(true)], [write(true)])

29. SRI International 3/23/98 PAAM ’98 Tutorial Requesting Services oaa_Solve(TaskExpr, ParamList) Expressions: logic-based (cf. Prolog) Parameters: provide advice & constraints High-level task types: query, action, inform,... Low-level: solution_limit(N), time_limit(T), parallel_ok(TF), priority(P), address(Agt), reply(Mode), block(TF), collect(Mode),... oaa_AddData(DataExpr, ParamList) oaa_AddTrigger(Typ,Cond,Action,Ps) oaa_Solve((manager(‘John Bear’,M), phone_number(M,P)), [query(var(P))]) Task Management Data & Trigger Management Example

30. SRI International 3/23/98 PAAM ’98 Tutorial Compound Queries n Address:Goal::Parameters u Address & Parameters Optional u Value-returning Parameters n Composable Using Standard Prolog Operators n Extensions u Parallel Disjunction oaa_Solve( (locate(‘Adam Cheyer’, Where)::[strategy(query)], notify(MsgRef, ‘Adam Cheyer’, [at(Where), by(fax)])::[strategy(action)]), [])

31. SRI International 3/23/98 PAAM ’98 Tutorial Facilitation Provider Requester Facilitator Goal Planning Plan Execution Registry

32. SRI International 3/23/98 PAAM ’98 Tutorial OAA Data Management n Declaring & Utilizing Data Solvables n Built-in Support n Example Parameters u single_value(t_f), unique_values(t_f) u bookkeeping(t_f), persistent(t_f) u synonym(Synonym, Original) u rules_ok(t_f) n Maintaining Data Solvables n Sharing Data

33. SRI International 3/23/98 PAAM ’98 Tutorial OAA Triggers OAA agents can dynamically register interest in any data change, communication event, or real- world occurrence accessible by any agent. oaa_AddTrigger(Type, Cond, Action, Params) comm:on_send, on_receive message time: “in ten minutes”, “every day at 5pm” data: on_change, on_remove, on_add task: “when mail arrives about...” The actions of triggers may be any ICL expression solvable by the community of agents Trigger Types Purpose Actions Adding a Trigger

34. SRI International 3/23/98 PAAM ’98 Tutorial System-Building Infrastructure n The Event Loop n Event Types u Built-In u Task-Specific u Hybrid n Libraries u Multiple Languages Supported u Minimal Structure Imposed on Agents

35. SRI International 3/23/98 PAAM ’98 Tutorial A Sample Text-to-Speech Agent in C #include ICLTerm capabilities = icl_TermFromStr(“[play(tts, Msg)]”); ICLTerm oaa_AppDoEvent(ICLTerm Event, ICLTerm Params) { if (strcmp(icl_Str(Event), “play”) == 0) { return playTTS(icl_ArgumentAsStr(Event, 2)); } else return NULL; } main() { com_Connect(“parent”, connectionInfo); oaa_Register(“parent”, “tts”, capabilities); oaa_MainLoop(True); } Include libraries List capabilities Define capabilities Agent Startup

36. SRI International 3/23/98 PAAM ’98 Tutorial A Sample Text-to-Speech Agent in Prolog :- use_module(com). :- use_module(oaa). capabilities([ solvable(play(tts, Msg), [type(procedure), callback(tts_events)], [])]). tts_events(play(tts, Msg), Params) :- tts_api(Msg). start :- capabilities(C), com_Connect(parent, ConnectionInfo), oaa_Register(parent, tts, C), oaa_MainLoop(true). Include libraries List capabilities Define capabilities Agent Startup

37. SRI International 3/23/98 PAAM ’98 Tutorial OAA and Scalability Facilitator is single point of failure Facilitator is bottleneck for communication Limitations: Solutions? Facilitator Multi-Facilitator topologies Distribution of planning & execution functions of Facilitator + peer-to-peer communication Registry & Planner Agent E Replicated Plan + Exe Facilitator

38. SRI International 3/23/98 PAAM ’98 Tutorial n Context: Agent Types & Approaches n Challenges & Opportunities n Inside the Open Agent Architecture n Example Systems & Useful Techniques u Agent & Interagent Programming Tips u Dynamic Presentation: Unified Messaging u Reference Resolution: Multimodal Map u Information Management and Collaboration: InfoBroker & Multimodal Map u Incremental System Development & Evaluation: Stimulate u Looking for the Killer App: Other Tries n Concluding Remarks

39. SRI International 3/23/98 PAAM ’98 Tutorial Agent & Interagent Programming Tips n Choosing an Agent Interface n Information Sharing Strategies n Domain-Specific vs. Domain-Independent Agents n Adding Speech & NL to Interfaces

40. SRI International 3/23/98 PAAM ’98 Tutorial Tips: Choosing an Agent Interface n Natural-language inspired interfaces u Imperative Verb, Direct Object, ParamList, (Result) u Parameter lists hold Adjs, Advs & Prepositions as well as extensible programmatic instruction n Classes tagged by type u inform(phone, ringing, Params) u send_message(MsgRef, Params) :- memberchk(by(fax), Params) n Succeed once with list vs. Multiple success u get(email, message_headers, +Params, -ListOfHeaders) u phone_number(Person, PhoneNum)

41. SRI International 3/23/98 PAAM ’98 Tutorial Tips: 3 Information Sharing Strategies n Example: Phone Dialer Agent n 1. Query u When an agent wants to know the status of the phone, it asks the Facilitator who asks the phone agent pa: oaa_Declare(status(phone, S),[]) ?a: oaa_Solve(status(phone, S), [])

42. SRI International 3/23/98 PAAM ’98 Tutorial Tips: Information Sharing Strategies - Post n 2. Post (Blackboard) u The phone agent writes its status to the Facilitator; agents can query the facilitator for status, and install a trigger which proactively monitors changes to status pa: oaa_AddData(status(phone, busy), []) ia: oaa_Solve(status(phone, S), []), oaa_AddTrigger(data, status(phone,S), notify(Me, phone(S)), [on(change)])

43. SRI International 3/23/98 PAAM ’98 Tutorial Tips: Information Sharing Strategies - Inform n 3. Inform u Broadcast time-critical events to interested parties u ia: oaa_Declare(msg(phone, Msg), []) u pa: oaa_Solve(msg(phone, ringing, []), [inform])

44. SRI International 3/23/98 PAAM ’98 Tutorial Tips: Domain-Specific vs. Domain Independent Agents n Move domain-dependent code into separate agent n Employ hooks and parameters to allow domain- specific tailoring of functionality n Always ask: Domain-specific or domain independent? u Phone agent? u Office interface? u Notify agent? u Speech recognition? u Natural language? u Facilitator?

45. SRI International 3/23/98 PAAM ’98 Tutorial Tips: Adding Speech & NL n User Interface responsible for: u accepting user input, sending requests, displaying results u controlling interactions of Speech and NL n Complex interpretation processed by external domain agent

46. SRI International 3/23/98 PAAM ’98 Tutorial Unified Messaging: Problem n Universal Access: Access to web, email, voicemail, applications (e.g., calendar, database, scheduler) from multiple interfaces (e.g., web browser, desktop, telephone) n Delegated triggers to monitor information n Message dissemination across various media (e.g., fax, printer, email, phone, pager) u Locating destination target u Plan route according to user preferences & resources u Media translation as necessary n Extensible and distributed! Minimize dependencies among component technologies

47. SRI International 3/23/98 PAAM ’98 Tutorial Unified Messaging: Components Main Points Mobile, adaptable access to distributed services Integrated Messaging: web, email, voice, fax Flexible interactions among components Distributed reference resolution and media format translation Delegated Triggers

48. SRI International 3/23/98 PAAM ’98 Tutorial Unified Messaging: Implementation 1/2 n Universal Access u Every user interface (including phone) must identify user u UI’s coordinate themselves to ensure only one “primary” interface per user, per utterance n Message Dissemination u Media agents: distributed reference resolution and translation u print(Object, Params) F ref(it): oaa_Solve(resolve_reference(the, document, Params, NewObj)) F id(Pointer): oaa_Solve(resolve_reference_id_as(id(Pointer), postscript, [], PostScript) F print TextObject or PostScript

49. SRI International 3/23/98 PAAM ’98 Tutorial Unified Messaging: Implementation 2/2 n Adaptable Presentation u GenNL agent produces simple or structured text-based response for any ICL query u Reads distributed NL vocabulary definitions in forming simple responses: F Vocabulary: noun(‘telephone number’, phone_number, []) F NL -> ICL: “What is Adam Cheyer’s telephone number?” F ICL: oaa_Solve(phone_number(‘Adam Cheyer’, X),[query(var(X))]) F Reponse: [phone_number(‘Adam Cheyer’, ‘859-4119’)] F GenNL: “The telephone number of Adam Cheyer is 859-4119.” u Structured response: description(list(EltList, AttrList)) F title(Title):Title of list, e.g. ‘Schedule’ F elt(Elt): Name of individual element in list, e.g. ‘Appointment’ F intro(Intro):Introduction to be played at start of list, e.g., ‘Here is today’s schedule for Adam Cheyer’ F max_len(Max):Num < Max Display All, else Display 1st & iterate

50. SRI International 3/23/98 PAAM ’98 Tutorial Multimodal Maps Application Main Points Natural interface to distributed (web) data Synergistic combination of handwriting, drawing, speech, direct manipulation Parallel cooperation and competition among many agents Human & Agent collaboration Adaptable displays according to user preferences

51. SRI International 3/23/98 PAAM ’98 Tutorial Multimodal Interfaces using Parallel Distributed Agents n Competition and cooperation among agents at many levels u Pen input: gesture recognizer vs. handwriting recognizer u Natural language: multiple NL systems (multilingual, diff. capabilities) u Reference Resolution n Multiple modalities for resolving ambiguities u e.g.arrow + “scroll map” vs. arrow + “photo of this hotel”

52. SRI International 3/23/98 PAAM ’98 Tutorial Multimodal Reference Resolution u Context by object type: “show photo of the hotel” u Deictic: “Find distance from here to here”, “this one” u Positional context: Write “photo?” on hotel u Visual context: “Photo of the [visible] hotel” u Database queries: “show photo of the hotel in Menlo Park” u Discourse: “No, the other one” u User disambiguation through prompting: “Which hotel?”

53. SRI International 3/23/98 PAAM ’98 Tutorial Information Broker: Requirements n Integrate Internet sources with enterprise sources n Heterogeneity handled transparently n Structured and “semi-structured” sources n Flexible access to unreliable information sources n Easily extensible to new domains n User and task models used to guide retrievals n Infrastructure must provide a basis for tools

54. SRI International 3/23/98 PAAM ’98 Tutorial Information Broker: Functionality n Mediation n Retrieval Strategies n User & Task Models

55. SRI International 3/23/98 PAAM ’98 Tutorial Mediation n Transparent access to heterogeneous sources F WWW structured and semi-structured sources F SQL sources F Knowledge bases F Multimedia repositories n Dynamic source registration & schema update n Query planning across distributed sources n Queries in broker or source schema n Domain knowledge used to increase query range n Built-in normalizations and conversions n Incomplete & inconsistent information

56. SRI International 3/23/98 PAAM ’98 Tutorial Retrieval Strategies n Identification of relevant sources n Extraction of desired information u Imposing structure on semi-structured Web pages n Local caching of virtual databases n Sensitivity to time constraints u Flexible strategies for web vs. cache retrievals n Dealing with unreliability and change u Cache maintenance u Use of alternate sources u Tracking and rating of sources

57. SRI International 3/23/98 PAAM ’98 Tutorial User and Task Modeling n Representation of salient characteristics of users and tasks n Mapping from situation to information request u What information is needed and when? u User and task models used as constraints n Mapping information retrieval to presentation u What information does the user want to see? u User and task models used as filters n User-friendly knowledge acquisition n Learning user and task models where feasible

58. SRI International 3/23/98 PAAM ’98 Tutorial Sample Queries u Mediation F “Find all hotels (meeting certain constraints) in San Francisco” u Use of domain knowledge F “Find hotels halfway between S. F. and Portland” u User modeling F “Apply my preferences” (to the same query) u Legacy and Web data source integration F “Show just the hotels for which we get a corporate discount” (Accesses WWW sources and employee db) F “Find the names and extensions of employees in the AI Center who have written about …” (Accesses Harvest index, Bibtex file and employee db) u “Persistent” queries F “Notify me of any ad selling a used color inkjet printer”

59. SRI International 3/23/98 PAAM ’98 Tutorial Information Broker: Architecture Semi-structured Source (Surrogate) Structured Source (Surrogate) HTTP Retrieval Agent Direct Query Interface Direct Query Interface UI Helper Agent (User Model) Helper Agent (User Model) Broker RDB Source (Wrapped) Broker schema Source schemas BQ BR SQ SR

60. SRI International 3/23/98 PAAM ’98 Tutorial The Broker Goal Planning Plan Execution Domain Knowledge Domain Knowledge Registry Transaction Management Transaction Management Agent Interactions Management

61. SRI International 3/23/98 PAAM ’98 Tutorial Agent Interactions Management Surrogates Batch Retrieval Routines Online Retrieval Strategies Cache Management Information Extraction Techniques Information Extraction Techniques Cache

62. SRI International 3/23/98 PAAM ’98 Tutorial Persistent Queries HTTP Retrieval Agent Direct Query Interface Direct Query Interface Broker Q T T T T UI Transaction Management Surrogate Helper Agent R

63. SRI International 3/23/98 PAAM ’98 Tutorial Useful Features of the Framework n Tight Integration of Data Capabilities n Standardized, Visible Content Language n Extension of Logic Programming Paradigm

64. SRI International 3/23/98 PAAM ’98 Tutorial Collaboration-ready Data Management n Store data using OAA Data Management u oaa_DbDeclare(icon(Id, X, Y, PictureType), [shareable, callback(icon_change)]) n Separate code which changes data from results, using callback feature u NOT: { oaa_AddData(icon(hilton, 100, 100, hotel), []) map_Display(icon(hilton, 100, 100, hotel)) } u BUT: { oaa_AddData(icon(hilton, 100, 100, hotel), []) } icon_change(add, icon(Id, X, Y, Picture)) :- map_Display(icon(Id, X, Y, Picture)).

65. SRI International 3/23/98 PAAM ’98 Tutorial Incremental System Development & Evaluation n Collaborative Multimodal Map application adapted for Wizard Of Oz (WOZ) experiment to elicit data about coordinated use of language and gesture

66. SRI International 3/23/98 PAAM ’98 Tutorial Subject Screen vs. Wizard Screen

67. SRI International 3/23/98 PAAM ’98 Tutorial Subject Video

68. SRI International 3/23/98 PAAM ’98 Tutorial Hybrid Wizard Of Oz Experiment n Naive user free to write, draw, or speak without constraints imposed by current technology n Wizard must respond quickly and accurately by using existing means, including pen and voice n Simultaneous evaluation of: u Experienced user manipulating real system u New user, providing data for future extensions n Bootstrap effect: continuous loop from data to theory, to system enhancement n Improvements from data analysis quantifiable n General-purpose approach

69. SRI International 3/23/98 PAAM ’98 Tutorial Hybrid WOZ: Implementation n System logging and playback “for free” using OAA collaboration facilities n “Subject mode”: functional interpretation (mostly) turned off n Addition of simple Wizard Feedback panel (separate agent) for text-to-speech messages (e.g., “Function not available.”)

70. SRI International 3/23/98 PAAM ’98 Tutorial Looking for Killer Apps n OAA has been used to implement more than 25 systems and prototypes n Not good for every application, but good for: u integrating numerous components which need to cooperate, often across language boundaries u supporting media translation u distributed reference resolution u tasking through adaptable or multimodal user interfaces u human/agent collaborative systems & incremental dvpt u exploring direct manipulation/task delegation tradeoffs

71. SRI International 3/23/98 PAAM ’98 Tutorial OAA-based Applications 1. Automated Office 2. Unified Messaging 3. Multimodal Maps 4. CommandTalk 5. ATIS-Web 6. Spoken Dialog Summarization 7. Agent Development Tools 8. InfoBroker 9. Rental Finder 10. InfoWiz Kiosk 11. Multi-Robot Control 12. MVIEWS Video Tools 13. MARVEL 14. SOLVIT 15. Surgical Training 16. Instant Collaboration 17.Crisis Response 18. WebGrader 19. Speech Translation 20-25+...

72. SRI International 3/23/98 PAAM ’98 Tutorial MVIEWS Application Interactive Map Main Points Multimodal annotation of video using speech & pen Automated detection, tracking, and geolocation of moving objects Search and replay of videos indexed by multimodal and auxilliary data Applications: multi-sensor surveillance, Predator UAV, Olympic bombing Live and Archived Video Interactive Map Video browser with multimedia timeline

73. SRI International 3/23/98 PAAM ’98 Tutorial MVIEWS Architecture

74. SRI International 3/23/98 PAAM ’98 Tutorial InfoWiz Kiosk Main Points An information kiosk with an animated wizard who : answers questions, gives tours, and helps navigate the information space OAA integrates SRI’s speech recognition, NL, dialogue, and knowledge representation with Microsoft Agent graphics and Netscape’s webbrowser Soon in SRI ’s lobby

75. SRI International 3/23/98 PAAM ’98 Tutorial InfoWiz Kiosk Architecture

76. SRI International 3/23/98 PAAM ’98 Tutorial Multi-Robot Control Concept Design Monitoring Maps, video, status Configurable displays Configurable displays Global or individual views Global or individual views Directed camera & robot control Delegated tasking through speech & gesture Tasking

77. SRI International 3/23/98 PAAM ’98 Tutorial Agent Development Tools Tools are implemented themselves in OAA Guide user through process of creating an agent: Definition of capabilities Documentation management (publication on Web) Code generation of agent template Definition of NL vocabulary Update NL & speech recognition systems Assembly of multiagent projects Runtime tool for launching and monitoring agent communities

78. SRI International 3/23/98 PAAM ’98 Tutorial Concluding Remarks n Many Varieties of u Agents u Agent-based Systems u Agent Frameworks n Useful Features of Agent Frameworks n Important Design Choices u Strategies for Interoperation & Coordination u Managing and Sharing Data u User Interface Functionality u Framework