Click to edit the title text format Methodology & Basics of Authoring TuTalk Dialogue Agents Pamela Jordan University of Pittsburgh Learning Research and Development Center

Session Team Members Pam Jordan Yue Cui (Jenny) Moses Hall Mike Ringenberg

Agenda Review of basic authoring Methodology for creating dialogue agents Advice on computer-mediated dialogues A closer look at basic authoring

What is TuTalk? Dialogue system construction tool for learning applications  Authoring tools for setting up an artificial dialogue partner/agent  Dialogue agent that conducts: Agent-led or mixed-initiative dialogues and Tutorial or conversational dialogues

Core modules: capabilities and limitations Understanding:  minimum edit distance,  costs of insertions, deletions, substitutions,  adjustable threshold for costs allowed Generation: stored phrases Student Model:  uses dialogue history to assess student performance Dialogue Manager:  Supports authoring features that balance expressiveness and ease of use  Limited mixed initiative handling All can be replaced (e.g. LSA for Understanding)

Finite state dialogue management As a runner pushes a ball away, what horizontal forces act on it? {runner’s} {gravitational, *} Good! After the push ends, what forces….?

Example dialogue T: Let’s consider the forces acting on objects after they have been thrown. As a runner pushes a ball away, what horizontal forces are acting on it while she is pushing it? S: Gravity? T: In what direction does gravity act? S: downward T: Yes. So is a downward force a horizontal or vertical force? S: It is vertical T: Right. I asked for the horizontal forces instead and the only one is from the runner. After the push ends, what horizontal forces are acting on the ball? Subdialogue

Authoring basics Write recipes and concepts  Concepts are like the ingredient list in a cooking recipe  Recipe uses a simple referring expression (concept label) and must look at ingredient list for the details Recipe has one to many steps Step is an  Initiation that is optionally paired with an expected Response  Can be a pointer to another recipe

Examples of concept specifications (abstract) ask_share_appetizer [So, should we share an appetizer?] [I’d like to share an appetizer. What looks good to you?] skip_appetizer [I don’t want an appetizer] [Let’s skip the appetizer]

Example of a dialogue script (abstract) Response action: push to recipe named possible responses Recipe: select-appetizer Step: enthuse_about_appetizers Step: ask_share_appetizer [agree_to_share_appetizer] [skip_appetizer abort, ask-soup] [unknown abort, loose-temper] Step: agree-on-appetizer initiation Concept to realize or recognize Subrecipe: push to recipe named goal name

Authoring preparation methodologies Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-based authoring Collect corpus of humans interacting on task  Computer mediated  Non-interruptible turns Analyze for goals/topics & adjust for learning objectives Analyze goals/topics identified for student responses, look for answer categories of:  Partially correct/incomplete  Partially incorrect  Overly vague  Overly specific  Correct but premature Identify tutor tactics for each answer category Analyze student language

Tutoring tactics in ProPl

Form tactics Pump: can you say more about X? Hint & reask: fill in a possible missing piece then try again Socratic: lead through line of reasoning Simulation: lead through an example & abstract For additional ones, see chapters 7 & 8 of Evens & Michael (2006), One-on-One Tutoring by Humans and Computers

Applying tactics in ProPl

ProPl student language analysis

Theory-based authoring Same as corpus-based but  Based on theories about task & learning  Skip corpus collection Examples of theoretical conceptual tactics:  Definitions & applications of concepts (e.g. distinguish technical & lay senses of terms)  Conceptual variant of a domain principle (e.g. boundary conditions)  Variant of problem

Corpus-inspired authoring Combination of corpus-based & theory- based Locate related corpus Identify theoretical goals & refine w/ relevant ones find in corpus Identify theoretical expected responses & refine relative to corpus

Incremental refinement Author main-path dialogues w/ correct answers Refine for answer categories Author response to answer categories & attach to answer categories Pilot dialogues Analyze logs & refine authored dialogues

Author dialogues: Import a corpus Import a corpus to authoring tool as in demo  Adjust clusters to identify topics/goals  Manually extract answer categories per goal Write main-path dialogues in text editor relative to goals then import to tool Append topics to template, pair turns and annotate goals & concepts

Author dialogues: alternative approaches Write xml directly (see preliminary documentation and dtd at Skip external authoring of main-path dialogues and author all directly in tool

Advice on computer-mediated dialogues Students prone to “refusal to answer”: I don’t know, who cares  Don’t always bottom out  Prod to try Avoid interrogation:  remember coherency w/ short recaps, turn and topic transitions,  make some abstractions, meta-info explicit Assess understanding:  Avoid explicit “do you understand”  Use trick questions, after success check strength of assertion Are you sure? What other forces (when answer is no more) Don’t be interactive for sake of being interactive but for sake of adapting to individual  Dialogue slow if cover everything

Additional authoring options Turn transitions/feedback Mixed initiative Optional steps:  skip if in recent history

Turn Transitions in xml In xml add truth-val attribute to initiation and response (values = yes,no,partial,unknown) Globally enable/disable (default is enabled) Say feature in authoring tool overrides automatic transitions (currently does both)

Simple XML Script enthuse_about_appetizers ask_share_appetizer agree_to_share_appetizer skip_appetizer unanticipated_response agree-on-appetizer

XML Script w/ truth-val enthuse_about_appetizers ask_share_appetizer agree_to_share_appetizer skip_appetizer unanticipated_response agree-on-appetizer

Mixed Initiative Limited mixed initiative:  allow student to initiate a topic/question  off by default T: What is the net force on the egg? S: What is the difference between net force and force? T: Net force is the sum of all the forces on a body. S: okay T: What is the net force on the egg?

Peer agent As a runner pushes a ball away, what horizontal forces act on it? {runner’s} {gravitational, *} Good! After the push ends, what forces….? If student says Agent picks an arc

Optional Steps As a runner pushes a ball away, what horizontal forces act on it? {runner’s} {gravitational, *} Good! After the push ends, what forces….? Any others? not said said

Example of optional steps T: … what horizontal forces are acting on it while she is pushing it? S: Gravity? T: In what direction does gravity act?. T: So are there any other forces on the ball? S: no T: What about the runner?. T: Okay. After the push ends, what forces… T: … what horizontal forces are acting on it while she is pushing it? S: The runner’s T: Right! So are there any other forces on the ball? S: no T: Good. After the push ends, what forces… subdialogue

Script with an optional step and semantic labels in XML enthuse-about- appetizers ask-appetizer skip- appetizer no unanticipated_response order-appetizer

Configuring built-in dialogue behavior Automatic feedback Initiative policies:  always ignore (default) or  always accept