1. Assessing Comprehension with Verbal Protocols and LSA
Keith Millis
Joe Magliano
Northern Illinois University

2. HOW TO FOOL A CAT

3. Once upon a time there was a rich lord.

4. He liked to collect little wood carvings of animals.

5. He had many kinds, but he had no carved mouse.

6. He called two skilled carvers to him.

7. PLEASE REPORT YOUR THOUGHTS
He said, “I want each of you to carve a mouse for me.”
PLEASE REPORT YOUR THOUGHTS

8. Sample thoughts
The, uh, he had all kinds of carved animals except a mouse. So he sent for some men to carve him a mouse. I guess he was going to pick which one he liked best.
He didn’t have a mouse for his collection of wood carved animals. And he wanted a mouse.
He really wanted a mouse.

9. What is LSA? Latent semantic analysis
Statistical representation of word knowledge
Computes a metric of semantic similarity between any two units of language
Cosine ranging (usually) between 0 and 1.0

10. Why use LSA to assess comprehension?
Comprehension tests typically use multiple-choice formats
The “off-line” limitation, “surfacy” information
LSA enables an assessment of on-line processing as expressed by verbal protocols
Verbal protocols
Express thoughts of the reader, which convey the use of reading strategies
LSA can categorize thoughts on reading strategies
R-SAT: Reading Strategy Assessment Tool

11. Summary of Approach
Compute similarity between verbal protocols and “semantic benchmarks” – words representing different strategies
The magnitude and pattern of cosines reveal strategies
Benchmark 1
at sentence N
Benchmark 3
Benchmark 2
At sentence N
Verbal protocol
.55
.04
.18

12. Types of reading/ comprehension strategies
Specific strategies
Paraphrasing, bridging & associative inferences
General strategies
The extent that the current text representation is elaborated with information from the reader’s world knowledge and the mental representation of the text

13. General reading strategies
Sentence-focused: primarily paraphrases the current sentence
Local: some elaboration of current sentence from the prior sentence
Global: uses prior text and world knowledge to integrate sentence with the theme of a text

14. Example: the Heart
(text…) The blood cannot get rid of carbon dioxide through the lungs. It becomes purplish, and the baby's skin looks blue.
Sentence focused: “The blood turns to a purplish color and the baby's skin turns blue.”
Local: “When the carbon dioxide does not escape the body, the baby's skin looks blue.”
Global: “The infant is essentially choking from the inside. The skin turning blue might have something to do with not receiving enough oxygen connected to the heart problems.”

15. General strategy and Information Source
Current sentence
Prior text
World knowledge
General strategy
Sentence-focused X
Local
Global

16. Benchmarks and source of information
(from theory) Good readers attempt to integrate current sentence with prior causal text along with appropriate inferences
Three benchmarks for each sentence
Current sentence
World knowledge
Prior causal
sentences
Verbal protocol
at sentence N
.75
.04
.18

17. Example Benchmarks Sentence from “Thunderstorm”:
“Usually, within an hour the amount and size of precipitation becomes too much for the updraft to support.”
Current sentence benchmark:
hour size precipitation amount becomes updraft support
Prior sentence:
cloud release develops start storm
World knowledge:
fall hold down heavy

19. Do cosines predict narrative comprehension
Do cosines predict narrative comprehension? (from Magliano & Millis, 2003)
Simple fairy tales
½ stories, participants thought-aloud to selected sentences
“Write down thoughts that come to mind as you read the current sentence in the context of the passage.”
½ stories, participants read silently
Answered comprehension questions & recalled the passages
Took the Nelson-Denny
Cosines, between protocol and
Current sentence benchmark
Prior causal sentences benchmark

20. Beta Weights from regression
Dependent Variable
T/F questions
Recall
TA-passages
Silent passages
LSA Current sentence
-.61**
-.36**
-.22*
LSA prior causal
.36**
.58**
.47**
Nelson-Denny
.02
.11
.23* R2
.38
.45

21. Beta Weights from regression
Dependent Variable
Textbase
Sit. Model
Total
LSA Current sentence
-.46*
-.31
LSA Prior Cause
.66**
.62**
.76**
Nelson-Denny
.27*
.31**
.34**
R2:
.20**
.27**
* p< .05; ** p < .01

22. Can feedback based on LSA change reading strategies?
Feedback to student who is typing in self-explanations to a text
If feedback based on LSA changes the quality of self-explanations, then
It supports the validity of LSA for identifying strategies
Useful in a tutoring environment

23. The cumulonimbus clouds are the first to appear in a thunderstorm
All thunderstorms have a similar life history. Thunderstorms start with the development of large cumulonimbus clouds.
Type your self-explanation below:
The cumulonimbus clouds are the first to appear in a thunderstorm
Get feed back
Tutor’s feedback:
Rate feed
back
OK.
How appropriate/fitting is the feedback?
Get Next
sentence
Not at all very

24. Method Participants receive brief self-explanation training
Participants told to evaluate feedback from new tutor
Randomly placed into one of four conditions: no feedback, general, full, random

25. Feedback conditions General: Full: Random feedback
Sentence focused = “Ok.” Local = “Very good.” Global = “Excellent.”
Full:
General + specific
Specific feedback associated with paraphrases, bridges, association
“Very good. I like it when you use your world knowledge.”
Random feedback
Like Full but content is randomly chosen.

26. Example Sentence: It becomes purplish, and the baby's skin looks blue.
Self-explanation: A lack of oxygen to the heart generates a blue hue on a baby's skin.
General: Excellent.
Full: Excellent. It seems to me that you are bringing up prior text in your self-explanation.
Random: Very good. It appears that you are paraphrasing for the most part.

27. General reading strategy

28. Specific reading strategies

29. Student appropriate ratings

30. Summary LSA can be used to assess text comprehension
Narrative and expository texts
Identify general and specific reading strategies
Does not have the “off-line” limitations associated with MC tests
What are important factors to consider with young children?