1. Relationship between Physics Understanding and Paragraph Coherence Reva Freedman November 15, 2012

2. Outline What is AIED? Intelligent tutoring systems for physics Coherence: an important factor Database and experiment Results Conclusions

3. What is AIED? Artificial Intelligence in Education Breakthrough in last decade –Follows transformation in AI (and NLP) in the previous decade “Science of learning” –Not just intuition any more

4. AIED + NLP: A good match Natural Language Processing –NL = what you speak Intelligent tutoring systems often use NL for the same reasons people do –More expressive than GUI –More flexible

5. Teaching/Learning Physics Two goals in physics teaching –Conceptual understanding –Problem solving Does teaching one help the other? Same issue in other sciences

6. ITSPOKE System Intelligent tutoring system for physics –Latest in a series  Andes/Atlas ...  APE (Python/TCP-IP  C++/etc/.COM) Experiment –Students solve a physics problem –Then write essay about it –Then rewrite essay until tutor is satisfied

7. Sample Physics Problem A person running in a straight line at a constant velocity throws a pumpkin straight up. Where does it land and why?

8. Sample Student Essay 1)While you are carrying the pumpkin, it has the same horizontal velocity as you do. 2)When you throw it up, there is the force of your throw which carries it vertically up, but there is no horizontal force to affect its horizontal velocity. 3)Eventually gravity will overcome the force of your throw and the pumpkin will come back down.

9. Sample Student Essay 4)During the pumpkin’s entire path up and back down, there is no horizontal force acting on it, so it will maintain the same constant velocity that it had when you were carrying it. 5)So the pumpkin will land on the ground right by your feet.

10. Coherence Coherence used to be hard to get hold of –“Know it when you see it” –But impossible to measure Barzilay and Lapata (2005, 2008) –New theory based on the new AI –Coherence is based on the relationship between the use (e.g., pumpkin) of entities in successive sentences

11. Importance of Coherence Local coherence is the major cause of students’ understanding what they read –McKoon and Ratcliff (1992)

12. Data Structure: Entity Grid s = subjecto = object x = other role- = none Sent. #1Sent. #2 noun 1s- noun 2sx noun 3-o

13. Algorithm (Barzilay/Lapata) Assume human texts are more coherent than permutations of them Choose a permutation function For every permutation of each essay in the training set, compare original vs. permutation Induce coherence function that prefers the original human text most often

14. A Permutation Function Binary discrimination –Compare original against random permutation –Fastest and easiest –Run multiple times and see who wins

15. Database Data –Experiment | student | problem | essay –No. of “wins” for binary test (0 – 20) –Physics pretest and posttest 2219 data points –Small number but big records

16. Methods Quinlan’s C4.5 algorithm –Implementation in Weka’s J48 –Numeric fields considered as nominal values, e.g., four subgroups for posttest values

17. Results: Null Hypothesis! No relationship between essay sequence number and coherence –When students rewrite their essays, they do not become more coherent No relationship between posttest score and coherence –Students with better conceptual understanding of physics do not write more coherent essays

18. Current Work Using NLP to break down data Relationships between linguistic features of the data and –Coherence –Physics knowledge

19. Wider Interests Science of learning without AI –What makes CS students successful? Computational linguistics w/o learning –Machine translation of lesser studied languages Programming –Building ITSs

20. Acknowledgements Learning Research and Development Center –University of Pittsburgh Diane Litman –Founder and director of ITSPOKE lab Kate Forbes-Riley –ITSPOKE researcher

21. Coherence: Underlying Ideas Important entities are more likely to be occur in key syntactic positions such as subject or object They are more likely to be introduced in the main clause They are more likely to be referred to with pronouns in later mentions

22. Coherence and Entity Use Texts about the same entity will appear more coherent to the reader than texts with multiple topic switches Continuity of topic leads to consistent patterns of entity use

23. Permutation Functions 1)Binary discrimination –Compare original against random permutation –Fastest –Easiest 2)Insertion –Insert one sentence, given the others –Quadratic in document length

24. Permutation Functions 3)Ordering –Try all permutations –Exponential in document length Implementation –Brown Coherence Toolkit (Elsner, 2008)

25. Entity Grid: Assumptions Noun phrases with the same head noun refer to the same entity –The bank I went to –The bank up the street Salience (importance) is measured by number of uses Assumptions can be changed