Download presentation
Presentation is loading. Please wait.
1
An Embedded Experiment to Evaluate the Effectiveness of Vocabulary Previews in an Automated Reading Tutor Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN Carnegie Mellon University Funding: NSF IERI, Heinz Endowments The Reading Tutor helps children read stories aloud. Should it preview new vocabulary? Within-student experiment design Effects on vocabulary and comprehension When does taking time to preview a new word before reading a story improve vocabulary and comprehension more than encountering the word in context? To address this question, the version of Project LISTEN's Reading Tutor embedded an automated experiment to compare three types of vocabulary preview -- defining the word, giving a synonym, or just asking about the word -- and a control condition. Outcomes included within-story comprehension as measured by performance on multiple-choice cloze questions, and post-story vocabulary as measured by matching words to their definitions. We analyze results based on thousands of randomized trials.
2
Before story: randomly assign 4 new vocabulary words in story to different previews
Control: Do not pretest or explain word meaning. Test-only: Pretest but do not explain word meaning. Pretest word meaning, then teach a synonym: Relate to a simpler word appropriate to the story. Pretest word meaning, then teach a definition: Give a short explanation appropriate to the story. How does Reading Tutor pretest word meaning?… Say “as I’ll show you in a moment” 2
3
Before story: pretest word meaning
3
4
Before story: teach a synonym
4
5
Before story: teach a definition
5
6
During story: assist reading (only with decoding, not word meaning)
6
7
During story: test comprehension by automatically inserting cloze questions
7
8
… then continue assisted reading
8
9
After story: post-test vocabulary
9
10
… by matching the 4 words to their definitions
10
11
Effects on vocabulary: post-test results
Percent correct on post-test N = 5668 trials 1417 story readings 364 students in grades K-9 7 urban & suburban schools “Untaught”: just encounter control test-only “Taught”: preview meaning synonym definition 56% 54% 52% 50% 48% control test-only synonym definition (95% confidence intervals of means) 11
12
Exclude words with hasty pretest responses
Correctness varies with speed N = 4251 pretest responses 10.8% are faster than 2 seconds 3.3% are slower than 15 seconds “Hasty” responses are guesses At chance level (25%) Exclude those words in subsequent analyses Percent correct on pretest 80% 60% 40% 20% 0% Response time on pretest (in seconds) 12
13
More taught than untaught words were learned
Split by whether already knew “Knew” = correct on pretest Excluding hasty pretests Compare per-student % correct N = 291 students If didn’t already know word: Taught: 17% over chance Untaught: 10% over chance Preview > encounter alone! P = .031 Who learned?… Percent correct on post-test 60% 50% 40% 30% N=277 N=242 N=291 N=250 No Yes 13
14
Who learned taught words better than untaught?
Disaggregate by WRMT N = 291 students No difference: For WC up to GE 3 Taught > untaught words: WC above GE 3 Percent correct on post-test 80% 60% 40% 20% 0% Word Comprehension GE pretest score 14
15
Effects on comprehension: cloze results
Did teaching target, distractor(s), or sentence word(s) help? N = 2671 cloze questions on vocabulary words Logistic regression variables Beta P Whether target word was taught .170 .097 Number of distractors taught .082 .183 Number of sentence words taught .256 .216 Number of sentence words untaught -.229 .232 WRMT Word Comprehension GE score .231 .000 Time since start of story till cloze question .548 15
16
Who did better on cloze if target word was taught?
N = 152 students’ cloze performance with target word taught or not Exclude cloze question if a distractor or sentence word was taught Percent correct on post-test 70% 60% 50% 40% 30% 20% 10% 0% Word Comprehension GE pretest score 16
17
Benefits to vocabulary and comprehension
Briefly explaining a new word before a story helped child Match the vocabulary word to its definition after the story Better than just encountering it in the story For students with Word Comprehension above GE 3 And apparently Answer comprehension questions involving the word Whether as cloze target, distractor, or context For students with Word Comprehension above GE 1 (!) Do previews really help comprehension before vocabulary? Thank you! Questions? Suggestions? 17
18
Project LISTEN’s Reading Tutor (video)
John Rubin (2002). The Sounds of Speech (Show 3). On Reading Rockets (Public Television series commissioned by U.S. Department of Education). Washington, DC: WETA. 18
19
Project LISTEN’s Reading Tutor (video)
19
20
Problem: support vocabulary learning
Efficacy varies by age and ability (NRP 2000). What are the specific vocabulary instruction needs of students at different grade and ability levels? Computers can facilitate vocabulary learning (NRP 2000). What is the optimal use of computers in vocabulary instruction? Our question: when does previewing new words help? Which previews? Which outcomes? Which students? Which words? 20
21
Within-student experiment design
The Reading Tutor helps children read stories aloud. Before story: pick 4 random new vocabulary words in story Pretest word meaning: “Which word means (definition)?” Teach 2 of the 4 words During story: Assist reading: scaffold decoding but not meaning Assess comprehension: insert cloze questions After story: Post-test all 4 words: “Which word means (definition)?” 21
22
show example -- simulate screen shot sequence
Before story show example -- simulate screen shot sequence 22
23
Before story: ask familiarity with word
23
24
During story: assist and assess reading
Assist: scaffold decoding but not word meaning Assess: automatically insert cloze questions Before a sentence containing a vocabulary word Delete the word and use it as the target Use 3 other vocabulary words as distractors (not necessarily the preview words) 24
25
Effect of previews on post-test
Results Effect of previews on post-test Effect of previews on cloze performance By age By word frequency 25
26
ANOVA for % correct on post-test
Source df F Sig Corrected Model 6 3.6 .002 Intercept 1 .48 .491 WRMT WC GE 7.3 .007 Age .61 .437 Word Length 1.1 .290 Gender .41 .521 Treatment 4.7 .031 Gender * Treatment .47 .493 Significant predictors: Previous vocabulary Preview 26
27
Previous work Aist 2002: factoids helped rare, single-sense words (grades 2-3) on next-day test Vocabulary learning literature Reinking & Rickman 1990: computer vocabulary help for 6th graders Beck: McKeown guidelines for explaining words Mostow, Tobin, Cuneo 2002: cloze validation see Aist thesis 27
28
Who learned taught words better than untaught?
28
Similar presentations
