Collaborative Research: Monitoring Student State in Tutorial Spoken Dialogue Diane Litman Computer Science Department and Learning Research and Development Center University of Pittsburgh

Project Participants  Post-docs –Kate Forbes-Riley, Joel Tetreault  Graduate Students –Mihai Rotaru (PhD), Beatriz Maeiereizo-Tokeshi (MS)  Undergraduate Student (REU) –Gregory Nicholas  Programmer –Scott Silliman  Columbia University –Julia Hirschberg, Jennifer Venditti (Post-doc), and Jackson Liscombe (PhD student)

Infrastructure  System Design and Implementation –ITSPOKE (Intelligent Tutoring SPOKEn dialogue system) Version 1 and 2  Corpus Collection – 128 human-human dialogues – 395 human-computer dialogues  Annotation – Positive/negative/neutral (pilot study) – Certainty and frustration/anger

Emotion Prediction  Extraction of acoustic/prosodic, lexical, and other turn-level features of student turns  Addition of word versus breath-group features  Classification using supervised and semi- supervised machine learning techniques

Adaptive Dialogue Systems  Interactions with speech recognition performance  Correlation with student learning and user satisfaction  Mining tutor responses to student emotional states

Remaining Year (No-Cost Extension Planned)  Incorporate User State Predictor into ITSPOKE  Experimentally evaluate original vs. affective ITSPOKE

Journal Publications  Diane J. Litman and Kate Forbes-Riley. Recognizing Student Emotions and Attitudes on the Basis of Utterances in Spoken Tutoring Dialogues with both Human and Computer Tutors. Speech Communication, in press.

Conference Proceedings (2004, 2005)  Kate Forbes-Riley and Diane Litman. Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2004).  Diane J. Litman and Kate Forbes-Riley, Predicting Student Emotions in Computer-Human Tutoring Dialogues. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004).  Diane Litman. Correlating Student Acoustic-Prosodic Profiles with Student Learning in Spoken Tutoring Dialogues. In Proceedings 9th European Conference on Speech Communication and Technology (Interspeech-2005/Eurospeech).  Mihai Rotaru and Diane J. Litman. Using Word-level Pitch Features to Better Predict Student Emotions during Spoken Tutoring Dialogues. In Proceedings 9th European Conference on Speech Communication and Technology (Interspeech-2005/Eurospeech).  Mihai Rotaru, Diane J. Litman, and Katherine Forbes-Riley. Interactions between Speech Recognition Problems and User Emotions. In Proceedings 9th European Conference on Speech Communication and Technology (Interspeech-2005/Eurospeech).

Conference Proceedings (2006)  Kate Forbes-Riley and Diane Litman. Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters. Proceedings of the Human Language Technology/North American Association for Computational Linguistics (HLT/NAACL).  Joel Tetreault and Diane Litman. Comparing the Utility of State Features in Spoken Dialogue Using Reinforcement Learning. Proceedings of the Human Language Technology/North American Association for Computational Linguistics (HLT/NAACL ).  Joel Tetreault and Diane Litman. Using Reinforcement Learning to Build a Better Model of Dialogue State. Proceedings 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL).

Workshop and Companion Proceedings  Diane J. Litman and Kate Forbes-Riley. Annotating Student Emotional States in Spoken Tutoring Dialogues. In Proceedings of 5th SIGdial Workshop on Discourse and Dialogue (SIGdial),  Diane J. Litman and Scott Silliman. ITSPOKE: An Intelligent Tutoring Spoken Dialogue System. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL) (Companion Proceedings),  Beatriz Maeireizo, Diane Litman, and Rebecca Hwa, Co-training for Predicting Emotions with Spoken Dialogue Data. In Companion Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona,  Kate Forbes-Riley and Diane J. Litman. Using Bigrams to Identify Relationships Between Student Certainness States and Tutor Responses in a Spoken Dialogue Corpus. In Proceedings of 6th SIGdial Workshop on Discourse and Dialogue (SIGdial), 2005.