Topics = Domain-Specific Concepts Online Physics Encyclopedia ‘Eric Weisstein's World of Physics’ Contains total 3040 terms including multi-word concepts such as “Frictional Force”, “Newton’s second law” etc. Present analysis limited to single-word concepts Ambiguity: “current problem” Vs “electric current” 36 physics concepts observed in our data 10 concepts appear in ideal answers are marked as ‘Critical’ Analyze number of times each concept is repeated per student Tutor: You need to show even more of your reasoning in your essay. What is the name of the principle of physics that you could apply to determine the difference in acceleration between the car and truck given that they both experience the same force? Student: I don’t know Tutor: Newton's Second Law. We know that Newton's Second Law expresses a relationship between force, mass, and acceleration. What is the equation for that relationship? Student: Force equals mass times acceleration Tutor: Fine. Applying this principle, if you apply equal force to two objects of differing masses, you can use the relative accelerations of the two objects to determine the relationship between the respective masses of the two objects. In this case, which vehicle will have a smaller acceleration? Student: Truck Tutor: Uh huh. So then you know that during the impact, the magnitude of the acceleration of the truck is less than the magnitude of the acceleration of the car for every time instance. Try to add something about this to your essay. Students take tests before (pre) and after (post) tutoring We study partial correlations (R) between post-test scores and concept-metrics, controlled for pre-test scores Intelligent Tutoring SPOKEn dialog-based system Interactive Dialog Sessions teaching Qualitative Physics Backend: Why-2 text-based physics tutoring [VanLehn et.al.] Tutor’s Role: Present a problem Clarify missing points, Correct misconceptions Ask Q’s to check understanding Request to revise answer Student’s Role: -Write Essays -Answer Q’s Include Multi-Word Concepts e.g. “Newton’s Second Law” Do non-physics words have any role in learning? Analyze Discourse-Level Topic Structures and Segments Concept-Level Topic Analysis of Tutoring Dialogs Amruta Purandare, Intelligent Systems Program Dr. Diane Litman, Department of Computer Science Shallow Metrics (Automatic) [Litman et. al. ITS04] Dialog Length (#turns, #words, words/turn) Distinguish Student and Tutor Contributions Deep Metrics (Manually Labeled) Student and Tutor Actions [Forbes-Riley et. al. AIED05] Misses the central theme of discussion: “Topics” or “Content” Show: Topics correlate to Learning Automatically identify Topic Structures in Dialogs Granularity Simple, Complex, Abstract Topics? 1 topic per turn/sentence? No Manual Annotations 1. Solve this problem 3. You need more details 4. What are the forces acting on the pumpkin? 5. Gravity? 2. Done! 6. Good! Revise your essay now ^^^^^^^^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^^ S+T+ES+TS+ESTE #WORDSR=0.44R=0.297R=0.516R=0.393R=0.287R=0.504 #PHY CONCEPTSR=0.488R=0.389R=0.557R=0.576R=0.355R=0.519 CONCEPTS / WORDS R=0.473R=0.505R=0.429R=0.396R=0.485R=0.329 #CRITICALR=0.52R=0.393R=0.55R=0.511R=0.364R=0.515 #CONCEPTS / TURN S+T+ES+TS+ESTE ZEROR=0.028R=0.027R=0.057R=0.055R=-0.05R=0.033 ONER=0.439R=0.466R=0.508R=0.584R=0.243R= TWOR=0.462R=0.489R=0.217R=0.303R=0.465R=-0.04 THREE and MORER=0.326R=0.252R=0.358R=0.445R=0.222R=0.314 AT LEAST ONER=0.443R=0.445R=0.582R=0.607R=0.347R=0.238 AVERAGER=0.323R=0.367R=0.261R=0.505R=0.287R=0.328 Word Level Correlations Turn Level Correlations Bold indicates correlations significant at 0.05 level Bold and Italics indicates correlations significant at 0.01 level S = Student (spoken), T = Tutor (spoken), E = Essay (written) Physics concepts show better correlations than all words #Dialog turns with 0 and 3+ concepts show no impact Student contributions matter! #physics-concepts, #critical-concepts discussed per student #dialog turns that discuss n concepts (where n = 0,1,2,3+,1+) Average #physics-concepts per turn Previous Work Our work Challenges 1. Background: ITSPOKE 2. Motivation: What Affects Learning? 3. Methodology 4. Preliminary Results 5. Current Directions Evaluation Example Automatic Topic Extraction Metrics Conclusions