Click to edit the title text format Methodology for Authoring Dialogues Pamela Jordan University of Pittsburgh Learning Research and Development Center

Agenda Methodology for authoring dialogues Some lessons learned on authoring computer-mediated dialogues Next steps for projects & discussion

Authoring preparation methodologies Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-based authoring Collect corpus of humans interacting on task  Computer mediated  Non-interruptible turns Analyze for goals/topics & adjust for learning objectives Analyze goals/topics identified for student responses, look for answer categories of:  Partially correct/incomplete  Partially incorrect  Overly vague  Overly specific  Correct but premature Identify tutor tactics for each answer category Analyze student language

Tutoring tactics in ProPl

Form tactics Pump: can you say more about X? Hint & reask: fill in a possible missing piece then try again Socratic: lead through line of reasoning Simulation: lead through an example & abstract For additional ones, see chapters 7 & 8 of Evens & Michael (2006), One-on-One Tutoring by Humans and Computers

Applying tactics in ProPl

ProPL student language analysis  Use to define response concepts  Strategy: pick a minimal set of key words that will distinguish between responses

Authoring preparation methodologies Corpus-based Theory-based Corpus-inspired Incremental refinement

Theory-based authoring Based on theories about domain/task & learning Examples of theoretical conceptual tactics:  Definitions & applications of concepts (e.g. distinguish technical & lay senses of terms)  Conceptual variant of a domain principle (e.g. boundary conditions)  Variant of problem

Authoring preparation methodologies Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-inspired authoring Combination of corpus-based & theory-based Locate related corpus Identify theoretical goals Search for some of those goals within a corpus & refine relative to what can find Identify theoretically expected student responses Refine relative to those response instances can find in corpus

Authoring preparation methodologies Corpus-based Theory-based Corpus-inspired Incremental refinement

Author main-path dialogues w/ correct answers Refine according to answer categories Author responses to answer categories Pilot dialogues Analyze logs & refine authored dialogues

Agenda Methodology for authoring dialogues Some lessons learned on authoring computer-mediated dialogues Next steps for projects & discussion

Recognizing student responses Language recognizer uses simple technique of minimum edit distance  Minimum edit distance is smallest number of “edits” (insertions/deletions) needed for student response to match a response phrase  For each set of alternative NL phrases (concept) for all responses for the current question, find the minimum edit distance  Select concept with smallest minimum edit distance  If that edit distance is within threshold (default of <.5) then select that concept as the response  Else the student response is not recognized, so follow the unanticipated response path Beginnings & endings of unmatched parts of responses are not penalized Stop words (e.g. of, the) are not penalized Strategy for authoring a response: pick a minimal set of key words that will distinguish between responses for a question

Advice on computer-mediated dialogues Students prone to refusal to answer e.g., “I don’t know”, “who cares”  Don’t always bottom out  Prod student to try (e.g. “Make your best guess”) Avoid interrogation:  remember to address coherency; include short recaps, turn and topic transitions,  make some abstractions, meta-information explicit e.g., “Let’s break it down some more”, “First, we’ll identify the givens”. Assess understanding:  Avoid explicit “do you understand?”  Use trick questions; after success check strength of assertion “Are you sure?” “What other forces are there?” (when answer is no more) Don’t be interactive just for sake of being interactive, instead use it to adapt to individual  Interact in order to diagnose what the student needs  Dialogue slow if cover everything; figure out what can be skipped

Agenda Methodology for authoring dialogues Some lessons learned on authoring computer-mediated dialogues Next steps for projects & discussion

Next steps for projects Look at dialogue samples/corpus for yur project and identify goals to cover in dialogue  available corpora: For each goal author main path with only correct responses and unanticipated response follow-ups

Discussion & questions Describe your projects What help/advice do you anticipate needing?