1. Learning Agents Center Computer Science Department George Mason University Dorin Marcu 02-09-2004 IT 803 Spring 2004 – Mixed-Initiative Intelligent Systems – Prof. G. Tecuci COLLAGEN: Mixed-Initiative Interaction with a Collaborative Agent

2. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

3. Research Motivation  current systems lack support for the user’s problem solving process during extended periods of time: – the order in which actions must be performed by the user and the system is often inflexible – it’s hard to recover from mistakes – each system has its own interaction conventions

4. Explored Solution  support the problem-solving level of human- computer interaction via a collaborative software agent  develop a new paradigm for human-computer interaction which explicitly supports the user’s problem-solving process based on current theories of collaborative discourse

5. Assumption A human-computer interface based on familiar human discourse rules and conventions will be easier for people to learn and use than one that is not.

6. Collaborative Interface Agents From (Rich and Sidner, 1998), page 317

7. COLLAGEN: A Collaboration Manager A collaboration manager is a software component that: - mediates the interaction between a software interface agent and a user - keeps track of the linguistic and attentional state of a discourse (similarly to a discourse manager) - keeps track of the collaborative intentions of the participants

8. COLLAGEN: A Collaboration Manager (contd.) A collaboration manager is less than a fully automated planning system because: - it does not by itself decide what the agent should do or say next (though it may provide some candidates) - it provides a representation for recording the decisions that the agent has made and communicated.

9. Definition of Mixed-Initiative in COLLAGEN Mixed-initiative reasoning in COLLAGEN can be defined as a discourse-based collaboration between a human user and an interface agent that attempt to achieve shared goals by decomposing them into sub-goals and primary actions for which they have complementary solving capabilities.

10. Characteristics of Mixed-Initiative in COLLAGEN  one human – one agent, discourse-oriented collaboration  partial, task-oriented mixed-initiative system:  some tasks have fixed-initiative flags, specifying which participants can solve them  the human participant has precedence over the agent, and can ignore the contributions of the agent  initiative taking:  proposing a goal  proposing a recipe for solving a goal  solving a (part of a) recipe  delegation

11. Main Features of COLLAGEN  both participants know and intend that all their actions are observed:  reporting communication (‘I have done x’)  direct observation

12. Main Features of COLLAGEN (contd.)  the mixed-initiative capabilities of the agent arise from the interplay of two sources:  application-independent algorithms and data structures in COLLAGEN  application-specific code and libraries in the agent  a library of recipes that specify the typical steps and constraints for achieving certain goals  arbitrary pattern-action rules

13. Main Features of COLLAGEN (contd.)  supports mixed-initiative by:  interpreting discourse acts  maintaining a model of the achieved and expected tasks and goals of the user and agent  the user makes the final decision for the problem-solving process and can ignore the contributions of the agent

14. Main Features of COLLAGEN (contd.)  the interaction model is based on a formal representation of the mutual beliefs about the goals and actions to be performed, and the capabilities, intentions, and commitments of the participants

15. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

16. Air Travel Planning System You are a Boston-based sales representative planning a trip to visit customers in Dallas, Denver, and San Francisco next week. You would prefer to leave on Wednesday morning, but can leave on Tuesday night if necessary. Your customer in Denver is only available between 11 a.m. and 3 p.m. on Thursday. You would prefer to fly as much as possible on American Airlines, because you have almost enough frequent-flier miles to qualify for a free trip this summer. You absolutely must be home by 5 p.m. on Friday to attend your son’s piano recital.

17. Air Travel Planner Interface From (Rich and Sidner, 1998), page 319

18. Air Travel Planner Interface (contd.) From (Rich and Sidner, 1998), page 320

19. Traditional Use of Planner Seven visitors and staff members were asked to solve this and similar problems and their behavior was recorded via informal notes and the logging facilities built into the application A typical problem solving session lasted about 15 minutes and entailed about 150 user actions (mouse clicks).

20. User Problems with the Traditional Planner Various forms of getting stuck and getting lost:  trouble knowing what to try next when the trip has been over- or under-constrained  trouble keeping track of which combinations of routes and constraints were already examined  workflow interruptions caused by the use of application’s functions (e.g. “Snapshot” – save context)

21. Collaborative Interaction Example (Rich and Sidner, 1998), page 323

22. Collaborative Interaction Example (contd.) (Rich and Sidner, 1998), page 323

23. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

24. The Architecture of COLLAGEN (Rich and Sidner, 1998), page 335

25. The Interface Agent COLLAGEN does not provide tools for building a complete agent – different types of agents can be supported (rule-based expert systems, neural nets, or a completely ad hoc collection of code) COLLAGEN provides a generic framework for recording the decisions made and communicated by the agent (and the user), but not for making them.

26. Question Why would such a generic framework be useful? Effective?

27. Answer Why would such a generic framework be useful? Effective? Because it will allow the reuse of components for: - building collaborative agents - application-independent discourse manager

28. The Execution Cycle  a communication or observation event arrives at the discourse interpretation module  the discourse interpretation module updates the discourse state  a new agenda of expected communication and manipulation acts is computed by the discourse generation module  the agent may decide to select an entry in this new agenda for immediate execution (according to the agent’s initiative strategy)  the user communication menu is updated with all the communication actions in the agenda for which the actor is either unspecified or the user

29. The Default Agent Initiative Strategy The default agent implementation that is included in COLLAGEN always chooses to perform the highest priority action in the current agenda for which the actor is either unspecified or itself. The priorities are manually associated with actions. Previous experiments also used pair-wise comparison rules. (Rich, 2002)

30. The Architecture Issue COLLAGEN has a generic framework for developing interface agents that can collaborate with human users to solve shared goals. The generic framework must be customized for a specific application by developing application- dependent : - recipes - action models - methods for performing actions and for observing actions performed by the user - collaboration behavior

31. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  tasks  shared awareness  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

32. Window Sharing  both the user and the agent have a dedicated “home” window that is used for communication between them  each home window contains an identifying face and has an associated cursor  the home windows are serviced by separate processes to support asynchronous mixed-initiative interaction (Rich and Sidner, 1998), page 321

33. Types of Agent Communication  printing English text in the agent’s home window  acting on the application’s interface with its cursor while approved and observed by the user The user can ignore the suggestions proposed by the agent. (Rich and Sidner, 1997), page 290

34. Types of User Communication  selecting from a menu of communications expected by the discourse interpretation algorithm (that were generated from the underlying model of discourse)  acting directly on the application with his/her cursor (the agent always observes the user’s actions through a generic layer in the application that mirrors semantic actions into the input buffer of the agent process) (Rich and Sidner, 1998), page 319 (Rich and Sidner, 1998), page 325

35. The Communication Issue COLLAGEN is based on a discourse theory of collaboration in which:  the agent communicates by:  generating natural language text or user- selectable menus/buttons  performing direct actions in the interface that the user can observe  the user communicates by:  selecting agent generated menus/buttons  performing direct actions in the interface that the agent can observe

36. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

37. Collaborative Discourse Theory Collaboration is a process in which two or more participants coordinate their actions toward achieving shared goals. Most collaborations between humans involves communication. Discourse is a technical term for an extended communication between two or more participants in a shared context, such as a collaboration.

38. Collaborative Discourse Theory (contd.) Three interrelated types of collaborative discourse structures (Grosz and Sidner, 1986):  intentional structure (formalized as partial SharedPlans)  linguistic structure (includes the hierarchical grouping of actions into segments)  attentional structure (captured by a focus stack of segments)

39. Intentional Structure for the Air Travel Planner  the user knows the constraints on travel  the agent has access to a data base of all possible flights  both participants have a common goal (to find an itinerary that satisfies the constraints)

40. Intentional Structure for the Air Travel Planner (contd.) both participants:  have agreed on a sequence of actions (a recipe) to accomplish the common goal (e.g., choose a route, specify some constraints on each leg, etc.)  are capable of performing their assigned actions  intend to do their assigned actions  are committed to the overall success of the collaboration (not just the successful completion of their own parts).

41. SharedPlans SharedPlans is a formal representation of the participants’ mutual beliefs about the goals and actions to be performed, and of their capabilities, intentions, and commitments.

42. Representation of SharedPlans in COLLAGEN (Rich and Sidner, 1998), page 331

43. Features of SharedPlans  partial (due to incomplete knowledge) – communication is required to fully specify them  recursive  their planning and execution is usually interleaved for each participant and among participants

44. SharedPlans in COLLAGEN COLLAGEN provides a generic framework only for recording the order in which planning and execution occur. COLLAGEN does not currently provide a generic framework for execution (interleaving planning and execution).

45. Discussion No generally accepted domain-independent theory of how people manage the interleaving of planning and execution. Best candidate: the BDI (belief/desire/intention) frameworks.

46. Discourse Segments There is general agreement that discourse in human-human interactions has a natural hierarchical structure, the elements of which are called segments. A segment is a contiguous sequence of communicative actions that serve some purpose (e.g. to achieve shared knowledge of some fact).

47. Example of Discourse Segments (Rich and Sidner, 1998), page 329

48. Focus Stack The focus stack contains discourse segments in the order in which they are created during the natural flow of a collaborative discourse. It captures the shifting focus of attention in a discourse. New segments and sub-segments are created, pushed onto the focus stack, completed, and then popped off the stack as the SharedPlan unfolds in the conversation.

49. Example of Focus Stack (Rich and Sidner, 1998), page 331

50. Discourse State in COLLAGEN (Rich and Sidner, 1998), page 335

51. Discourse State Representation

52. Discourse State Representation (contd.) The discourse state representation contains:  the plan tree, which is an approximate representation of a partial SharedPlan)  the focus stack  the history list, which contains top level segments that have been completed and removed from the focus stack

53. Discourse Interpretation in COLLAGEN (Rich and Sidner, 1998), page 335

54. Discourse Interpretation The discourse interpretation module determines how the current direct communication or observed manipulation action can be viewed as contributing to the current discourse purpose (of the top segment in the focus stack).

55. Recognized Acts The discourse interpretation module recognizes acts that:  directly achieve the current purpose  correspond to one of the steps in a recipe for the current purpose  identify the recipe to be used to achieve the current purpose  identify who should perform the current purpose or a step in the current recipe  identify an unspecified parameter of the current purpose or a step in the current recipe.

56. Questions What types of acts that may occur are not recognized by the discourse interpretation module? What can be done in those cases?

57. Answers What types of acts that may occur are not recognized by the discourse interpretation module? What can be done in those cases?  interruptions (a segment that does not contribute to its parent)  one participant decides to address a new goal  the participants pursue multiple goal in the same time focus stack management  incomplete recipe let the user manage the focus stack

58. Questions Discourse interpretation looks similar to plan recognition, which is known to be exponential in the worst case. Is discourse interpretation necessarily exponential? What can be done to avoid this?

59. Answers Discourse Interpretation is generally not exponential because:  the participants can try to make sure that their intentions can be understood without a large cognitive search  the participants can ask clarification questions to minimize search  the search is performed through the steps of the current recipe or all known recipes for the current segment

60. Discourse Generation in COLLAGEN (Rich and Sidner, 1998), page 335

61. Discourse Generation Discourse generation is the inverse of discourse interpretation. It produces a prioritized agenda of (possibly partially specified) actions which would contribute to the current discourse segment from the focus stack, and its associated SharedPlan.

62. Dynamic Generation of Agent Responses Develop a plug-in architecture for reusable components that generate agent responses based on the last event (communication or action) and the current discourse status. Develop: - application-independent - application-dependent - role-dependent (e.g. assistant vs. tutor) plug-ins for the developed architecture.

63. Plug-in Architecture (Rich et all., 2002), page 784

64. Plug-ins for Generating Agent Responses A plug-in for generating agent responses is an object that is associated with an internal state and an algorithm for generating responses based on its state, the discourse state and the event received by the discourse interpreter module. A plug-in:  can access and modify the discourse state.  can access the recipes library.  has a priority associated with all the responses it generates

65. The Plug-in Interface update(event): Updates the private state of the plug-in (if any) and modifies the discourse state (if necessary) – for example by adding a goal or binding a parameter. generate(event): Returns a (possibly empty) list of agenda items which would be an immediate response to event. visit(node): Returns a (possibly empty) list of agenda items which would contribute to the goal of node.

66. Application-Independent Plug-ins (6) Execute(goal): (default priority 0) If goal is primitive and all of its parameters are bound and its first parameter is bound to the agent, then return goal. ProposeShouldUser(goal): (default priority 0) If goal is primitive and its first parameter is bound to the user, then return an utterance of the form “Please perform/say goal.” AskWho(goal): (default priority 0) If goal is primitive and its first parameter is unbound, then return an utterance of the form “Who should perform/say goal?”

67. Role-Dependent Plug-ins TeachStep(goal): (priority 100) If goal is teachable, then return an utterance of the form “The first/next step is goal.” TeachInitiative(goal): (priority 130) If the student model indicates that the student should already know how to achieve goal, then return “You take it from here.” PositiveFeedback(action): (priority 160) If action was live and expected, then return an utterance such as “Great”, “Nice”, “Right”.

68. Algorithm for Choosing Responses  add to the agenda the results in the order returned, of invoking the generate method of each plug-in on the received event, in the default plug-in order  add to the agenda the results in the order returned, of invoking the visit method of each plug- in, in the default plug-in order on the live, expected nodes (visited in breadth-first order)  do a post-pass to reorder the agenda based on any additional explicit preferences

69. Segmented Interaction History  provides the user with a structured guide to his/her problem solving process  can be used to retry, re-apply or undo past actions  displayed at the user’s request (Rich and Sidner, 1998), page 336

70. Producing the Segmented Interaction History Starting from the current discourse state:  list the purpose of each top level segment on the history list starting with the oldest  recursively list the purpose of each closed sub-segment followed by the purpose and elements of each open sub- segment, starting with the top level open segment (the base segment of the focus stack)  for each open sub-segment list the unexecuted recipe steps (if any) in the plan tree for that segment’s purpose, starting with the most deeply nested

71. Question What are some advantages or disadvantages of COLLAGEN’s segmented interaction history?

72. Answer Some advantages or disadvantages of COLLAGEN’s segmented interaction history:  is structured (interaction histories are typically flat)  reflects the user’s actual problem solving process  requires a structured discourse (interaction) model and its use in the problem solving process (interpretation, etc.)

73. The Shared Awareness Issue Shared awareness in COLLAGEN is maintained by the agent through the discourse state, which is accessible to the user via the segmented interaction history. It contains descriptions of the: - past interactions - current “stack” of execution reflecting the goals being solved. It is updated based on: - direct manipulation actions performed by the user or the agent - user selections of agent generated menus/buttons

74. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

75. Agent Customization Through Task Models The goal is to develop a formal model of the collaborative tasks being performed by the agent and the user. Since the agent does not have to rely on the task model alone for its decision making, the model only needs to be complete enough to support communication and collaboration with the user. The task model in COLLAGEN contains recipes and primitive actions.

76. The Recipe Library in COLLAGEN (Rich and Sidner, 1998), page 335

77. Recipes  a recipe is an application specific resource used to derive a sequence of steps to achieve a given goal (the objective of the recipe) (Garland at all., 2001), page 45

78. Recipes (contd.) It is easy to represent recipes with a fixed number of steps in a simple recipe formalism. More complicated recipes require step structures that depend on some parameters of the objective (e.g. the top level recipes for scheduling a trip working forwards or backwards).

79. Question How can a recipe with variable structure be specified?

80. Answer How can a recipe with variable structure be specified? In COLLAGEN, a general-purpose procedural approach, called recipe generators, was developed:  recipes with variable structure are represented as procedures which, given an objective, return a recipe  a predicate can also be associated with a recipe to test whether it is still applicable as it is being executed

81. Manual Development of the Task Model  initial versions of a task model are inferred from a small number of examples that show the most common solutions to key domain tasks  the model is generalized to cover additional examples that demonstrate solutions involving alternate orderings for actions, optional behavior, or alternate task decompositions (defining additional examples may force the expert to re-conceptualize the entire domain necessitating reworking many previous examples)  correct and complete learned models may need to be tweaked by the expert for other reasons (e.g. organization of a complete and accurate task model may be inappropriate for a collaborative agent)  verify the behavior of the model using the collection of past examples

82. Learning The Task Model From Examples  programming by demonstration  incremental learning from examples  allows updating past examples used in learning  regression testing with all examples

83. Learning The Task Model From Examples Step one (optional): provide an initial list of primitive and non-primitive acts Non-Primitives: MakeMeal Primitives: Boil, CookPasta, PrepareSauce, ServeDinner

84. Learning The Task Model From Examples (contd.) Step two: define un-annotated example GetPasta Boil CookPasta PrepareSauce ServeDinner

85. Learning The Task Model From Examples (contd.) Step three: (partially) annotate example [MakeMeal] [PreparePasta] GetPasta Boil CookPasta PrepareSauce ServeDinner

86. Learning The Task Model From Examples (contd.) Step four: learn (partial) recipe from example nonprimitive act PreparePasta primitive act GetPasta recipe Boil_CookPasta_GetPasta achieves PreparePasta stepsBoil boil CookPasta cookPasta GetPasta getPasta constraintsgetPasta precedes boil boil precedes cookPasta

87. Learning The Task Model From Examples (contd.) Step five: define additional example(s) [MakeMeal] GoToKitchen(kitchen2) [PrepareSauce] Boil(water3) CookClams(clams8,water3) MakeClamSauce(clams8) [PreparePasta] Boil(water3) CookPasta(linguini11,water3) ServeDinner(kitchen2)

88. Learning The Task Model From Examples (contd.) Step six: refine learned recipe nonprimitive act PreparePasta parameter Water water primitive act GetPasta recipe Boil_CookPasta achieves PreparePasta stepsBoil boil CookPasta cookPasta optional GetPasta getPasta bindings achieves.water = cookPasta.water constraintsboil.water = cookPasta.water cookPasta.water = water3 cookPasta.pasta = linguini11 getPasta precedes boil boil precedes cookPasta

89. Revive Viewer (Garland et all., 2001), page 49

90. The Learning Problem Input: (partially annotated example) Output: recipe(s) + refined examples (Garland et all., 2001), page 46

91. Experimental Results The impact of annotations on the number of needed examples: O: ordering E: equalities P: propagators All: includes optional (Garland et all., 2001), page 47

92. Recipe Library Statistics  8 recipes defined in terms of 15 different goal or action types for the air planner system (the authors estimate that the final number should be about twice as much) (Rich and Sidner, 1998)  29 recipes, 67 recipe steps, 36 primitive acts and 29 non-primitive acts for the Symbol Editor application (Garland el all., 2001)  8 recipes, 19, recipe steps, 13 primitive acts and 4 non-primitive acts for the artificial cooking world model (Garland el all., 2001)

93. The Task Issue Collaboration is modeled in COLLAGEN through recipes and primary actions. At task modeling time, some tasks can be designated to be performed by the user or by the agent, the rest being open. Example of user-designated tasks: specifying route or constraints. Example of agent-designated tasks: search data-base for flight info. Open tasks can be negotiated based on user preferences (e.g. asking the user who should perform a task) and roles (tutoring vs. assisting).

94. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

95. Control Principles  a recipe needs to be identified for the current goal (identification of a recipe may be achieved by asking the other participant)  a goal or action may be performed when all of its parameters are known and all of its predecessors (in the current recipe) have been achieved  a goal or action may be performed by any participant who is capable, unless a specific participant has been specified

96. Global vs. Local Initiative  most work is done on local initiative and its aspects:  turn taking and conversational control  interruptions, and  grounding (how the current speaker indicates that she has heard and understood the content of the previous speaker’s turn)  true mixed-initiative systems must address both global and local initiative

97. Global Initiative  application-independent: based on the current discourse state and the agenda computed from it, a COLLAGEN-based agent has an explicit choice of relevant things to say at most points in a conversation  application-dependent: the agent can choose between answering a question, performing an interface action or some other behavior, according to the specifics of the application domain

98. Negotiation as Global Initiative  negotiation is the ability to resolve differences in beliefs that are relevant to some shared goal  through negotiation, collaborators with differing points of view about the goals and recipes they undertake, as well as the state of the world at any point in time, can reach an agreement  currently, COLLAGEN does not incorporate a negotiation facility

99. Local Initiative COLLANGEN does not provide a general algorithm for turn taking because of the limitations of the current theories of local initiative in conversation Ad-Hoc mechanism developed (detailed in the following slides):  provide the user with means to relinquish control to the agent  provide the agent with means to get the user’s attention when the agent does not have control

100. Relinquish User Control The user relinquishes the control when he/she does not want to contribute any further to the current SharedPlan and instead would like to see what the agent can contribute. The agent was designed to interpret an ‘OK’ selected by the user (other than as an answer to a direct Yes/No question) as an signal of relinquishing control.

101. Request User’s Attention The agent was designed with the ability to wave its cursor ‘hand’ when it has something important to contribute to the conversation. The resulting behavior is considered very humanlike and affecting.

102. History-based Transformations History-based transformations are used to alter the course of the problem solving process by reusing past actions (without modifying the history). History-based transformation types (at segment level):  stop  revisit  retry  undo  reply The availability of such transformations is application- dependent (the extent to which the application provides methods to alter its internal state)

103. (Rich and Sidner, 1998), page 340

104. The ‘Stop’ Transformation motivation:  the user wants to let the agent know that she/he is no longer working towards the current goal  the agent has misunderstood what the current goal is actions:  pops the current segment off the focus stack without changing the application state  if the purpose of the popped segment contributes to its parent, the appropriate un-bindings are also performed in the plan tree applicability:  only to open segments

105. The ‘Retry’ Transformation motivation:  the user wants to return to working on an earlier goal – achieved or not – and try achieving it in a different way actions:  stop and reset the stack and plan tree to their states at the start of the selected segment  the recipe becomes unbound applicability:  to any type of segments

106. The ‘Revisit’ Transformation motivation:  the user wants to pick up where he/she left off working on an earlier goal actions:  stop and reset the stack and plan tree to their states at the end of the selected segment  the recipe is preserved applicability:  only to closed segments

107. The Replay Transformation motivation:  the user wants to reuse earlier work in the current context actions:  all of the application acts in the selected segment are put together into one (possibly hierarchical) recipe which is then executed by the agent in the current context

108. Questions What happens if some of the acts may not be valid in the current context? To what type of segments is the ‘Replay’ transformation applicable?

109. Answers What happens if some of the acts may not be valid in the current context? Depending on the specific details of the agent’s interface to the application, such errors may need to be handled by application-specific code in the agent, or may be taken care of by the application’s existing API or graphical interface. To what type of segments is the ‘Replay’ transformation applicable? Open and close segments. Why?

110. The Control Issue The control of initiative in COLLAGEN is partly managed: - in a generic way by the application-independent collaborative algorithms - in an application-dependent way by the related-designations in the recipes - in a role-dependent way by the discourse generation plug-ins. The agent can negotiate with the user by asking who to perform a non-designated task, or by asking whether the agent- generated solution is acceptable to the user. The user can relinquish control by asking the agent to perform a task (where possible). Through history-based transformations, the user has direct control over the problem solving process.

111. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

112. Evaluation No experimental evaluation to assess any strengths/weaknesses of the mixed-initiative approach vs. the classic approach (direct manipulation of the GUI-based applications. My assessment: since the user has the final decision on the performed steps and the communication model is straight-forward and natural, the collaborative approach should lead to better and faster problem-solving episodes.

113. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

114. Key Ideas in COLLAGEN  multidisciplinary research in user interface, linguistics, and artificial intelligence  collaboration between one user and one interface agent that is similar to human-human collaboration)  guided and structured interaction based on formal model of collaboration (SharedPlans)  separation of application-independent structures (collaboration theory, general framework) from application-dependent structures (recipes, agent customization, plug-ins)  supports a user’s problem solving process by relating current actions to the global context and history of the interaction (discourse interpretation, discourse generation, discourse status, recipe library)  oriented toward applications with direct manipulation of interfaces

115. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

116. Lessons Learned  using a formal task model for guiding collaboration and mixed-initiative interaction is effective (either SharedPlans or GOMS-like models)  using a menu-driven interaction is simpler that natural language processing (although it is not clear if it is generally sufficient)  using “plug-ins” to customize behavior seems to be an effective approach  recognizing user actions and performing agent actions is application dependent, and might be challenging with applications that are not restricted to direct manipulation of GUIs  more advanced user/agent models are necessary for different roles (e.g. tutoring) and for multi-human multi-agent collaborations

117. Presentation Outline  A brief introduction to COLLAGEN  An air travel application built with COLLAGEN  A more detailed look at COLLAGEN:  architecture  communication  shared awareness  tasks  control  evaluation  Summary of the main ideas in COLLAGEN  Lessons learned  References

118. Charles Rich and Candace L. Sidner, “COLLAGEN: A Collaboration Manager for Software Interface Agents”, User Modeling and User-Adapted Interaction 8(3), 1998, Kluwer Academic Publishers, pp 315-350. Charles Rich and Candace L. Sidner, “COLLAGEN: When Agents Collaborate with People”, in Proceedings of the First International Conference on Autonomous Agents, February 1997, Marina Del Rey, California, ACM Press, pp 284-291. Charles Rich, Neal Lesh, Jeff Rickel and Andrew Garland, “A Plug-in Architecture for Generating Collaborative Agent Responses”, in Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems – AAMAS’02, July 15-19, 2002, Bologna, Italy, ACM Press, pp 782-789. Andrew Garland, Kathy Ryall and Charles Rich, “Learning Hierarchical Task Models by Defining and Refining Examples”, in Proceedings of the International Conference on Knowledge Capture – K-CAP’01, October 22-23, 2001, Victoria, British Columbia, Canada, ACM Press, pp 44-51. Papers Used For Presentation

119. Other Interesting Related Papers Andrew Garland, Neal Lesh and Charles Rich, “Responding to and Recovering from Mistakes during Collaboration”, in Proceedings of the Workshop on Mixed- Initiative Intelligent Systems, the 18th International Joint Conference on Artificial Intelligence – IJCAI’03, August 9, 2003, Acapulco, Mexico, AAAI Press, pp 59- 64. Jacob Eisenstein and Charles Rich, “Agents and GUIs from Task Models”, in Proceedings of the 7th International Conference on Intelligent User Interfaces – IUI’02, January 13-16, 2002, San Francisco, ACM Press, pp 47-54. Charles Rich, “Window Sharing with Collaborative Interface Agents”, ACM SIGCHI Bulletin, Volume 28 Issue 1, January 1996, ACM Press, pp 70-78. Barbara J. Grosz and Candace L. Sidner, ‘Attention, Intentions, and the Structure of Discourse’. Computational Linguistics 12(3), 1986, pp 175–204. Jim R. Davies, Abigail S. Gertner, Neal Lesh, Charles Rich, Candace L. Sidner and Jeff Rickel, “Incorporating Tutorial Strategies Into an Intelligent Assistant”, in Proceedings of the 6th International Conference on Intelligent User Interfaces – IUI’01, January 14-17, 2001, Santa Fe, New Mexico, ACM Press, pp 53-56.