1. Carnegie Mellon Generating Questions Automatically from Informational Text Wei Chen, Gregory Aist, and Jack Mostow Project LISTEN, School of Computer Science www.cs.cmu.edu/~listen Carnegie Mellon University The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B070458. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute and the U.S. Department of Education.

2. Carnegie Mellon Background Problem: How to improve students’ reading comprehension? Solution: Teach effective reading strategies – Activating background knowledge – Self-questioning – Summarizing – Visualizing Most effective strategy to teach -- National Reading Panel 2000 2

3. Carnegie Mellon Self-Questioning Good readers ask themselves questions during reading (Wong 1985) 3 Wong, B.Y.L. Self-questioning instructional research: A review. Review of Educational Research, 1985. 55(2): p. 227-268. Once upon a time a town mouse, on a trip to the country, met a country mouse... And when the country mouse saw the cheese, cake, honey, jam and other goodies at the house, he was pleasantly surprised. Why was the country mouse surprised?

4. Carnegie Mellon Prior Work ( Mostow & Chen 2009 ) Generating self-questioning instruction 4 text Narrative fiction Informational text Generating Instruction Automatically for the Reading Strategy of Self-questioning July 9 th, 18:00-18:30 AIED main conference Generating Instruction Automatically for the Reading Strategy of Self-questioning July 9 th, 18:00-18:30 AIED main conference questionsinstruction AIED09QG Workshop 09

5. Carnegie Mellon Goal Automatic generation of comprehension instruction Targeted population: children in grades 1-3 Targeted text genre: informational text 5 text Narrative fiction Informational text questionsinstruction

6. Carnegie Mellon Narrative vs. informational Narrative fiction: Peter thought it best to go away without speaking to the white cat. Informational text: Rainbows are seen after it rains and the sun is out. 6 CharacterMental state Objective Phenomenon Description/Explanation

7. Carnegie Mellon Example Question Title: “Life under the Sea Part 1 – the Meaning of Life” 7 What does it mean to be “alive?” … All living things are not exactly alike. For example, not all living things breathe air, or have blood, or grow hair, like we do. Likewise, we can’t live under water like fish do. Right now the question I’m thinking about is, why can’t we live under water like fish do?

8. Carnegie Mellon Example Question: Narratives 8 Text: Once upon a time a town mouse, on a trip to the country, met a country mouse... And when the country mouse saw the cheese, cake, honey, jam and other goodies at the house, he was pleasantly surprised. Question: Why was the country mouse surprised?

9. Carnegie Mellon Approach: Narratives Where to insert questions? Look for mental states When the country mouse saw the cheese, cake, honey, jam and other goodies at the house, he was pleasantly surprised. 9

10. Carnegie Mellon Question Generation Process How to generate questions? text situation model of mental states questions 10

11. Carnegie Mellon Situation Model 11 Country mouse Old knowledge New knowledge Cheese, cake, … at the house When the country mouse saw the cheese, cake, honey, jam and other goodies at the house, he was pleasantly surprised.

12. Carnegie Mellon WHAT did ? WHY/HOW did ? WHY was/were ? Why did ? What did ? Question Templates 12 the country mousethink the man to send the catdecide Why was ? the country mouse surprised

13. Carnegie Mellon Mental States in Informational Text 1.The motive force or result of some events or phenomena 2.Mental states of people outside the text 13 Fish have “noses” (called nares) that don’t look anything like our own, yet their purpose is to smell chemicals in the water. If you’re an American citizen 18 years of age or older, you probably think you have the right to vote for presidential candidates in the national election.

14. Carnegie Mellon Mental States in Informational Text 3.Beliefs of authoritative sources or the general public 4.Similar to usage in narratives 14 It is thought that they use this structure to detect prey, perhaps being able to distinguish the weak electrical signals given off by injured animals. He had thought the blue butterfly was extinct.

15. Carnegie Mellon Extension to Other Question Types Identify question indicators that can – Signal key information about the text – Be feasible to automate We chose questions about: 1. Conditions 2. Temporal context 3. Modality (i.e., possibility and necessity) 15

16. Carnegie Mellon 1. Conditional Context Example Linguistic indicators “if,” “even if,” “only if,” “as long as” Question Template What would happen if ? 16 Text: If humans removed all the kelp from the sea soon all the other sea life would start to suffer as well. Question: What would happen if humans removed all the kelp from the sea?

17. Carnegie Mellon Situation Model for Conditions 17 Condition: humans remove all the kelp from the sea The other sea life Hypothetical: would Start to suffer If humans removed all the kelp from the sea soon all the other sea life would start to suffer as well.

18. Carnegie Mellon 2. Temporal Context Linguistic indicators: temporal expressions marked by semantic role labeler (ASSERT) – Exceptions: “usually”, “sometimes” [not related to key information in text] 18 Text: Rainbows are seen [ARG-TMP after it rains and the sun is out]. Question: What happens after it rains and the sun is out? Text: They (Native American children) sometimes learn to speak a Native American language. Question: What happens sometimes?

19. Carnegie Mellon Question Templates When would happen? What happens ? 19

20. Carnegie Mellon Situation Model for Temporal Context 20 Condition: after it rains Rainbows Seen Rainbows are seen after it rains and the sun is out.

21. Carnegie Mellon 3. Modality (possibility & necessity) Example Linguistic indicators: auxiliary verbs (e.g., should, must, …) Question Template Why ? 21 Text: All goats should have covered shelters where they can escape the weather. Question: Why should all goats have covered shelters?

22. Carnegie Mellon Situation Model for Modality 22 Prescriptive: should All goats Have Covered shelters All goats should have covered shelters where they can escape the weather.

23. Carnegie Mellon Evaluation Quick evaluation for plausible questions to be shown to experts Evaluation Criteria: – Grammatical – Make sense given the story context Test set: 26 informational texts 444 sentences 23

24. Carnegie Mellon Evaluation results Question Type # Linguistic Indicators # Generated questions % Plausible questions Condition15 87% (13/15) Temporal Context 448866% (58/88) Modality337787% (67/77) 24

25. Carnegie Mellon Error Analysis: 1. Conditions Unresolved coreference Ambiguity (here, of if) 25 Text: If so, then you have eaten kelp Question: What would happen if so? Text: Sit beside a quiet pool of water and you’ll soon see water striders skating as if on ice Question: What would happen if on ice? Not indicating condition !

26. Carnegie Mellon Error analysis: 2. Temporal Parsing error Unfiltered temporal expression: 26 Text: At present totem poles are sold to people who collect them and to museums Question: What happens at present? Parse: If the pressure changes over a large area it can cause [ARG1 winds] to [TARGET blow] [ARGM-TMP in a huge circle] Question: What happens in a huge circle? Not a temporal expression! Just like “usually” and “sometimes”

27. Carnegie Mellon Error Analysis: 3. Modality Parsing error 27 Parse: [ARG0 Skin cells] [ARGM-MOD must] [ARGM- DIS also] [TARGET make] [ARG1 sure] to keep harmful things out of the body Question: Why must skin cells make sure? Incomplete!

28. Carnegie Mellon Contributions Extension from narratives Unchanged: Question Generation Mechanism Added: – Discourse markers for opportunities of questioning – Rules for building situation model – Question templates 28 text situation model questions

29. Carnegie Mellon Comparison 29 GenreLinguistic Patterns# Inference/Schema building rules # Question Templates NarrativeModal verbs (239)294 InformationalIf-constructions (4), temporal expressions, auxiliary verbs (8) 64