1. Improving (Meta)cognitive Tutoring by Detecting and Responding to Uncertainty Diane Litman & Kate Forbes-Riley University of Pittsburgh Pittsburgh, PA USA

2. Outline u Motivation u Metacognitive Measures u System(s) and Corpus u Evaluation Results u Discussion

3. Background u Speaker uncertainty is of interest in several research communities u Human Language Technologies (Liscombe et al. 2005; Dijkstra et al. 2006; Pon-Barry 2008) u Psycholinguistics (Brennan & Williams 1995) u AI & Education (Tsukahara & Ward 2001; Aist et al. 2002; Craig et al, 2004; Pon-Barry et al. 2006; Forbes-Riley & Litman 2009)

4. This Paper u We show that remediating after student uncertainty has the potential to increase student's metacognitive (and cognitive) abilities u Evaluations use a corpus of previously collected dialogues between students and several versions of a Wizard of Oz spoken tutorial dialogue system

5. Metacognitive Performance u We measure metacognitive performance, in dialogues annotated for student uncertainty and correctness u Impasse severity u Monitoring accuracy (Nietfield et al. 2006) u Bias (Kelemen et al. 2000; Saadawi et al. 2009) u Discrimination (Kelemen et al. 2000; Saadawi et al. 2009) u We then conduct two evaluations u Do the measures differ across experimental conditions? u Do metacognitive and cognitive performance correlate?

6. Impasse Severity u Tutoring Theory: Uncertainty and Incorrectness both signal Learning Impasses (VanLehn et al., 2003) u Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity Nominal State:I+nonUI+UC+UC+nonU Scalar State:3 2 1 0 Severity:mostlessleastnone u This Paper: Use the scalar value associated with each student turn to compute an average impasse severity, per student

7. Impasse Severity u Tutoring Theory: Uncertainty and Incorrectness both signal Learning Impasses (VanLehn et al., 2003) u Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity Nominal State:I+nonUI+UC+UC+nonU Scalar State:3 2 1 0 Severity:mostlessleastnone u This Paper: Use the scalar value associated with each student turn to compute an average impasse severity, per student Not remediated in many systems

8. Measures from the Metacognitive Performance Literature CorrectIncorrect NonUncertainCnonUInonU UncertainCUIU The wizard's annotations for each student are first represented in an array, where each cell represents a mutually exclusive option motivated by Feeling of Knowing (FOK) research, which is closely related to uncertainty (Dijkstra et al., 2006) The array is then used to compute various standard measures

9. Monitoring Accuracy CorrectIncorrect NonUncertainCnonUInonU UncertainCUIU Ranges from -1 (no monitoring accuracy) to 1 (perfect monitoring accuracy)

10. Bias CorrectIncorrect NonUncertainCnonUInonU UncertainCUIU Bias scores greater than and less than zero indicate over-confidence and under-confidence, with zero indicating best performance

11. Discrimination CorrectIncorrect NonUncertainCnonUInonU UncertainCUIU Discrimination scores greater than zero indicate higher metacognitive performance, in terms of certainty for correct responses and uncertainty for incorrect responses

12. Prior Experiment: Does Remediating after Student Uncertainty Impact Learning?  ITSPOKE-WOZ  Speech-enabled Why2-Atlas (VanLehn, Jordan, Rosé et al. 2002), further parameterized to adapt to uncertainty and/or correctness  Speech recognition and correctness/uncertainty annotations performed in real-time by a human "Wizard"  4 Conditions  Normal Control: original system (no adaptation)  Simple Adaptation: same response for all impasses  Complex Adaptation: different responses for each impasse  Random Control: Simple Adaptation to random 20% corrects  Results (Forbes-Riley & Litman 2009)  Learning: main effect (Simple > Normal, Complex)

13. Normal (non-adaptive) System  ITSPOKE (Intelligent Tutoring Spoken Dialogue System)  Dialogue Format: Question – Student Answer – Response  Response Types:  to Corrects (C): positive feedback (e.g. “Fine”)‏  to Incorrects (I): negative feedback (e.g. “Well…”) and  Bottom Out: correct answer with reasoning  Subdialogue: questions walk through reasoning

14.  Simple Adaptation  Same response for all 3 impasses  Feedback on only (in)correctness  Complex Adaptation  Different responses for the 3 impasses  Feedback on both uncertainty and (in)correctness Two Uncertainty Adaptations

15. Simple Adaptation Example: C+U TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to? STUDENT1: The force of the car hitting it?? [C+U] TUTOR2: Fine. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]  Same TUTOR2 subdialogue if student was I+U or I+nonU

16. TUTOR2: That’s exactly right, but you seem unsure, so let’s sum up. [FEEDBACK] The net force on the truck is equal to the impact force on it. We can prove this just like we did for the car. First, we know that gravity and the normal force on the truck must cancel each other, otherwise the truck would not be at rest vertically. Second, we know that the impact force is the only horizontal force exerted on the truck. [NEW BOTTOM OUT]  Different TUTOR2 subdialogue if student was I+U or I+nonU Complex Adaptation Example: C+U

17. u Although our experiment was designed to impact learning, we hypothesized that the experimental conditions might also improve metacognitive performance u For each measure: u 1-way ANOVA with condition as between-subjects factor u Planned comparisions for each pair of conditions u complex adaptation > simple adaptation > random control > normal control Evaluation I: Does Remediating after Uncertainty Impact Metacognition?

18. Results I: Means across Conditions Metacognitive Measure Complex Adaptation (20) Simple Adaptation (20) Random Control (20) Normal Control (21) Average Impasse Severity.59.60.73 Monitoring Accuracy.58.62.52 Bias-.01-.03-.01-.02 Discrimination.34.46.48.41 u Both complex and simple reduced average impasse severity, compared to normal (p <.08 in paired contrasts)

19. Results I: Means across Conditions Metacognitive Measure Complex Adaptation (20) Simple Adaptation (20) Random Control (20) Normal Control (21) Average Impasse Severity.59.60.73 Monitoring Accuracy.58.62.52 Bias-.01-.03-.01-.02 Discrimination.34.46.48.41 u Simple (and random) increased monitoring accuracy, compared to normal (p <.06 in paired contrasts)

20. Results I: Means across Conditions Metacognitive Measure Complex Adaptation (20) Simple Adaptation (20) Random Control (20) Normal Control (21) Average Impasse Severity.59.60.73 Monitoring Accuracy.58.62.52 Bias-.01-.03-.01-.02 Discrimination.34.46.48.41 u No statistically significant differences or trends for bias

21. Results I: Means across Conditions Metacognitive Measure Complex Adaptation (20) Simple Adaptation (20) Random Control (20) Normal Control (21) Average Impasse Severity.59.60.73 Monitoring Accuracy.58.62.52 Bias-.01-.03-.01-.02 Discrimination.34.46.48.41 u Trend for discrimination differences overall (p =.09) u However, contrary to our predictions, complex reduced discrimination ability, compared to random and simple (p <.04 in paired contrasts)

22. u We also hypothesized that better metacognitive abilities would be better from a learning perspective u For each measure: u Partial Pearson's correlation over all 81 students with posttest score, controlled for pretest score to measure learning gain Evaluation II: Do metacognitive and cognitive performance correlate?

23. Results II: Significant Partial Correlations with Posttest (after controlling for Pretest) Metacognitive MeasureRp Average Impasse Severity-.56.00 Monitoring Accuracy.42.00 Discrimination.34.00 u Average Impasse Severity (where smaller is better) is negatively correlated with learning

24. Results II: Significant Partial Correlations with Posttest (after controlling for Pretest) Metacognitive MeasureRp Average Impasse Severity-.56.00 Monitoring Accuracy.42.00 Discrimination.34.00 u Better monitoring accuracy and discrimination (where higher is better) are positively correlated with learning

25. Summary u We analyzed metacognitive performance in tutorial dialogue u evaluation uses both new and traditional measures u Remediating after student uncertainty improved metacognitive performance u Impasse Severity u Monitoring Accuracy u Metacognitive performance is correlated with learning u Impasse Severity (negative) u Monitoring Accuracy, Discrimination (positive)

26. Current and Future Work u Use results to inform ITSPOKE modifications that improve both cognitive and metacognitive abilities u measure metacognition using Feeling of Knowing ratings u Replicate analysis using fully automated ITSPOKE

27. Questions? Further Information? http://www.cs.pitt.edu/~litman/itspoke.html Thank You!

28. Background u Speaker uncertainty is of interest in several research communities u Prosodic/lexical properties (Liscombe et al. 2005; Dijkstra et al., 2006; Pon-Barry 2008) u Feeling of (Another's) Knowing (Brennan and Williams, 1995) u Learning correlations (Craig et al., 2004; Forbes-Riley et al. 2008) u Tutoring evaluations have investigated the impact of responding to student uncertainty, over and above correctness u Positive feedback responses improved satisfaction (Tsukahara and Ward, 2001) and persistence (Aist et al., 2002) u More substantive responses didn’t improve learning in a fully- automated system (Pon-Barry et al., 2006), but did improve learning, efficiency, and satisfaction under semi-automated conditions (Forbes-Riley and Litman, 2009)

29. Discussion u Why didn’t Simple Adaptation and Complex Adaptation outperform Random Control? u Random Control adapted to some C+U, diminishing differences u Adapting to C+nonU may increase certainty u Why didn’t Complex Adaptation outperform Simple Adaptation? u Complex Adaptation’s human-based content responses were based on frequency, not effectiveness

30. Current and Future Work u Use results to inform ITSPOKE modifications that improve both cognitive and metacognitive abilities u Better optimize remediations for each impasse type, particularly those of highest severity u Remediate metacognitive abilities, not just domain content (e.g., Roll and Aleven, 2008; Saadawi et al., 2009) u measure metacognition using Feeling of Knowing ratings u Replicate analysis using recently collected data from fully automated ITSPOKE

31. Example Dialogue Excerpt TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to? STUDENT1: The force of the car hitting it?? TUTOR2: Fine. [FEEDBACK]

33. u Simple Adaptation: For CU, IU, InonU answers: u ITSPOKE gives same content with same dialogue act u ITSPOKE gives feedback on (in)correctness u Complex Adaptation: Depending on if answer is CU, IU, InonU: u ITSPOKE gives same content but varies dialogue act u Based on human tutor responses significantly associated with CU, IU, InonU answers u ITSPOKE gives complex feedback on affect and (in)correctness u Based on empathetic computer tutor literature (Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004) Two Uncertainty Adaptations

34. TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to? STUDENT1: Zero?? [I+U] TUTOR2: That’s not correct, but don’t worry. You seem to be aware of your mistake. Let’s resolve it. [FEEDBACK] The net force on the truck is equal to the impact force on it. Let’s walk through this answer step by step. [SHORT BOTTOM OUT] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE] Complex Adaptation Example: I+U

35. TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to? STUDENT1: The force of gravity. [I+nonU] TUTOR2: I’m sorry, but I see an error in your answer. Let’s fix it. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE] Complex Adaptation Example: I+nonU