Relationship between Physics Understanding and Paragraph Coherence Reva Freedman November 15, 2012

Outline What is AIED? Intelligent tutoring systems for physics Coherence: an important factor Database and experiment Results Conclusions

What is AIED? Artificial Intelligence in Education Breakthrough in last decade –Follows transformation in AI (and NLP) in the previous decade “Science of learning” –Not just intuition any more

AIED + NLP: A good match Natural Language Processing –NL = what you speak Intelligent tutoring systems often use NL for the same reasons people do –More expressive than GUI –More flexible

Teaching/Learning Physics Two goals in physics teaching –Conceptual understanding –Problem solving Does teaching one help the other? Same issue in other sciences

ITSPOKE System Intelligent tutoring system for physics –Latest in a series  Andes/Atlas ...  APE (Python/TCP-IP  C++/etc/.COM) Experiment –Students solve a physics problem –Then write essay about it –Then rewrite essay until tutor is satisfied

Sample Physics Problem A person running in a straight line at a constant velocity throws a pumpkin straight up. Where does it land and why?

Sample Student Essay 1)While you are carrying the pumpkin, it has the same horizontal velocity as you do. 2)When you throw it up, there is the force of your throw which carries it vertically up, but there is no horizontal force to affect its horizontal velocity. 3)Eventually gravity will overcome the force of your throw and the pumpkin will come back down.

Sample Student Essay 4)During the pumpkin’s entire path up and back down, there is no horizontal force acting on it, so it will maintain the same constant velocity that it had when you were carrying it. 5)So the pumpkin will land on the ground right by your feet.

Coherence Coherence used to be hard to get hold of –“Know it when you see it” –But impossible to measure Barzilay and Lapata (2005, 2008) –New theory based on the new AI –Coherence is based on the relationship between the use (e.g., pumpkin) of entities in successive sentences

Importance of Coherence Local coherence is the major cause of students’ understanding what they read –McKoon and Ratcliff (1992)

Data Structure: Entity Grid s = subjecto = object x = other role- = none Sent. #1Sent. #2 noun 1s- noun 2sx noun 3-o

Algorithm (Barzilay/Lapata) Assume human texts are more coherent than permutations of them Choose a permutation function For every permutation of each essay in the training set, compare original vs. permutation Induce coherence function that prefers the original human text most often

A Permutation Function Binary discrimination –Compare original against random permutation –Fastest and easiest –Run multiple times and see who wins

Database Data –Experiment | student | problem | essay –No. of “wins” for binary test (0 – 20) –Physics pretest and posttest 2219 data points –Small number but big records

Methods Quinlan’s C4.5 algorithm –Implementation in Weka’s J48 –Numeric fields considered as nominal values, e.g., four subgroups for posttest values

Results: Null Hypothesis! No relationship between essay sequence number and coherence –When students rewrite their essays, they do not become more coherent No relationship between posttest score and coherence –Students with better conceptual understanding of physics do not write more coherent essays

Current Work Using NLP to break down data Relationships between linguistic features of the data and –Coherence –Physics knowledge

Wider Interests Science of learning without AI –What makes CS students successful? Computational linguistics w/o learning –Machine translation of lesser studied languages Programming –Building ITSs

Acknowledgements Learning Research and Development Center –University of Pittsburgh Diane Litman –Founder and director of ITSPOKE lab Kate Forbes-Riley –ITSPOKE researcher

Coherence: Underlying Ideas Important entities are more likely to be occur in key syntactic positions such as subject or object They are more likely to be introduced in the main clause They are more likely to be referred to with pronouns in later mentions

Coherence and Entity Use Texts about the same entity will appear more coherent to the reader than texts with multiple topic switches Continuity of topic leads to consistent patterns of entity use

Permutation Functions 1)Binary discrimination –Compare original against random permutation –Fastest –Easiest 2)Insertion –Insert one sentence, given the others –Quadratic in document length

Permutation Functions 3)Ordering –Try all permutations –Exponential in document length Implementation –Brown Coherence Toolkit (Elsner, 2008)

Entity Grid: Assumptions Noun phrases with the same head noun refer to the same entity –The bank I went to –The bank up the street Salience (importance) is measured by number of uses Assumptions can be changed