Collecting, Storing, Coding, and Analyzing Spoken Tutorial Dialogue Corpora Diane Litman LRDC & Pitt CS

ITSPOKE Tutorial Dialogue Corpora Students engage in spoken dialogue with tutors, in the qualitative physics domain –human tutors –(fully automated) computer tutors –‘wizard’ computer tutors

Data Collection Speech-enhanced computer interfaces –Head-mounted microphones –Currently no video –Humans can be at different locations Human and Wizard Tutoring –Dialogue speech files Computer Tutoring –Utterance speech files Coordinated system logs

Data Storage Wav, raw audio, ogg formats Sampling –16k samples per second –16 bits per sample Stereo (dialogue level) and mono (utterance level)

Coding and Analysis Initially WaveSurfer –Open Source tool for sound visualization and manipulation Speech/sound analysis Sound annotation and transcription Praat is similar Recently moved to NXT (NITE XML Toolkit) –Also Open Source –

NXT Mature open-source libraries to support heavily annotated corpora whether they be multimodal; textual; monologue; dialogue A powerful integrated query language Built in tools for common tasks + Java API for custom tools Media sync built in Command line tools for data analysis

NXT meets the ICSI Corpus Jean Carletta and Jonathan Kilgour University of Edinburgh HCRC Language Technology Group

ICSI Meeting Corpus 75 natural meetings from research groups –close-talking and far-field microphones orthographic transcription "speech quality" tags (e.g., emphasis) dialogue acts hot spots

The NITE XML Toolkit library support for data handling and search using a data model that can express both timing and complex structure multiple file stand-off XML data storage some standard GUIs, data utilities library support for writing tailored GUIs

extract from Bdb001.A.words.xml time - line extract from Bdb001.A.speech-quality.xml Stand-off XML

Tasks pre-NXT: up-translation and tokenization hand annotation (topic segmentation, dialogue acts, extractive summaries,...) automatic annotation/indexing by query match

Queries in NXT ($a w):(TEXT($a) ~ /th.*/):: ($s speechquality):($s ^ $a) && Find instances of words starting with “th” For each find instances of speech quality tags of type "emphasis" that dominate the word Discard words that are not dominated by at least one such tag Use queries to understand data, verify quality, index.

NXT as Meeting Browser Browser = display + signal indexing + search NXT data displays: –synchronize with signal –highlight search results

Issues Already can't load all the ICSI data at once on some machines NXT supports display of one meeting at a time but browsing may be over several meetings Really complicated queries are often too slow for browser response times Key: Pre-indexing of query results, tailored data builds

NXT meets the BEETLE Corpus Johanna Moore’s Group University of Edinburgh

Coding Tutorial Dialogue Partitioned the dialogue into a set of non-overlapping segments with the following category names: –Content Dialogue that contains information relevant to the topics in the lessons. –Management Dialogue that does not contain information relevant to the lesson topics, but deals with the flow of the lesson. –Metacognition Dialogue that contains the student or tutor’s feeling about his or her understanding of the lesson material or each other. –Social Dialogue that serves as motivation, encouragement, humor, or establishing rapport.

Coding Student Utts for Sig Events Consider common theories of effective learning events Constructivism / generative learning –Osborne & Wittrock, 1983 Impasses –Van Lehn, et. al., 2003 Accountable talk –Wolf, Crosson & Resnick, 2006 Deep processing / cognitive effort –Thomas & Rohwer, 1993 Motivated, self-directed learner –Thomas & Rohwer, 1993 Student produces a lot of new information Student utts are incorrect or correct w/ low confidence Student utts are both accurate & deep Student utts are deep (regardless of accuracy)‏ High frequency of internally motivated student utts NOVELTY 1 ACCURACY 2 & CONFIDENCE 3 ACCURACY 2 & DEPTH 4 DEPTH 4 INITIATIVE 5

Student Utterance Coding Five major dimensions –Accuracy Correct, some missing, some errors, incorrect –Signs of “deep” processing or cognitive effort Present versus absent –Explain/justify/support statement with evidence/reasoning –Summarize or paraphrase –Express relationships or make connections between constructs –Questions or challenges statements from lesson or tutor Wolf, Crosson & Resnick (2006)‏ –Signs of low confidence Present versus absent (Bhatt, Evens & Argamon, 2004)‏ –Origin Externally versus internally motivated –Novelty Old versus new information

A B C battery X Question: If bulb B is damaged, what do you think will happen to bulbs A and C? Non-Accountable Talk: utt69: student: A and C will not light up Accuracy = Correct; Cognitive Processing = Absent Cognitive Effort and Potential Impasse: utt122a: student: bulb a will light but b and c won't since b is damaged and breaks the closed path circuit Accuracy = Incorrect; Cognitive Processing = Present Potential Impasse: utt97: student: both would be either dim or not light I would think Accuracy = Partially Correct; Cognitive Processing = Absent; Signs of Low Confidence = Yes utt83a: student: both bulbs A and C will go out because this scenario would act the same as if there was an open circuit Accuracy = Correct; Cognitive Processing = Present Accountable Talk: