1. Scalable and Sustainable Technologies for Reading Instruction
An Interagency Education Research Initiative: NSF - IES - NIH
Principal Investigators:
Walter Kintsch, Donna Caccamise, Ron Cole, Lynn Snyder, & Richard Olson

2. The Push to Improve Literacy
Every child can learn to read, yet
Literacy is a national problem
We have good basic research, problem is
To develop practical implementations
To get them used widely in schools
IERI initiative

3. Colorado Literacy Tutor
Tools for Teachers
K-6th grade:
emphasis on foundational reading skills: Interactive books for practice plus tutors for special problems
6th-14th grade:
emphasis on comprehension and learning from text

4. Underlying Cognitive Science Research
Tutoring is an effective method of instruction:
Human tutors produce gains between .4 and 2.3 standard deviations over classroom teaching
Most human tutors have only modest domain knowledge and rarely use sophisticated tutoring strategies
Existing Artificial Intelligence tutors such as AUTOTUTOR (Graesser et a., 1999) and ATLAS (van Lehn et al., 2000) approach the effectiveness of human tutors (Graesser et al., 2001)

5. Underlying Cognitive Science Research
Creating the Next Generation of Intelligent Animated Conversational Agents:
Use of synthesized speech and speech recognition
Conversation with a talking head
Learners respond in English in addition to click-and-point

6. Underlying Cognitive Science Research
Latent Semantic Analysis: Theory, Technology and Practice
LSA as a map of meaning
High-dimensional semantic space
Text meaning as the sum of the words
Limitations:
Learns only from words
Neglect of order and syntax

7. A map is a 2-dimensional representation of distance and direction data: Data: Map:

8. A semantic space is a k-dimensional representation of word co-occurrence data (k<<<m,n): Data: Semantic Space:

9. High-dimensional spaces are capable of representing complex semantic relations:
TREE
DOG
.70
BARK
.06
.66

10. The meaning of a text is the sum of the words in LSA:
The accident victim was flown to the hospital
This sentence is more similar to (1) than (2), although it shares two content words with (2) and none with (1):
(1) A helicopter brought the man hurt in the crash to the emergency room cosine = .29
(2) An accident occurred in the street where the victim could not get assistance
cosine = .24

11. Interactive Books & Reading Tutors
Ron Cole, Donna Caccamise, Lynn Snyder and the Colorado Literacy Tutor Team
Center for the Study of Spoken Language
University Of Colorado

12. Components of the Reading Tutor:
Foundational Skills Tutors
Teach underlying reading skills
Interactive Books
Teach fluent reading & comprehension
Managed Learning Environment
Enroll students, track and display progress, manage individual study plans, etc.

13. Beginning sounds

14. Letter sounds

15. Word reading

16. Silent “e”

17. Interactive Books Teacher or child selects appropriate book
Child selects animated tutor or “coach”
Animated coach guides child through book
Coach can read words, sentences paragraphs or whole page
Coach asks comprehension, inference and summary questions using speech recognition to determine accuracy of responses
Each level of a book has the capability of invoking specific reading tutors if a child needs more foundational skill training

18. Read to me and read aloud

19. Click on image question interaction

20. Multiple choice question interaction

21. Summarization using Summary StreetTM

22. Comprehension and Learning
Walter Kintsch, Donna Caccamise, Lynn Snyder, and the LSA Research Group
Institute of Cognitive Science
University of Colorado
Tom Landauer
Pearson Knowledge Technologies & University of Colorado

23. Comprehension and Learning
LSA computes how similar in meaning two texts are
The contents of a student’s essay can be compared with other essays and standards
Students receive feedback about the content of their writing - guidance for revising
A tool for self-assessment: feedback allows students to judge how well they are doing and what needs more work

24. Summary Street®
An existing, classroom-tested tested system that provides content-based feedback to middle-school students summarizing a text:
indicates what content is missing
what might be overemphasized
flags apparent problems sentences
helps with the organization of the material

29. The teacher keeps track of how much and how well the student did:

31. Provides hints about how the summary could be shortened:
Sentences are flagged that are very similar in meaning:
…...They also wrote books on paper. The books were made from bark paper that they folded together…..
Sentences that appear unrelated to the topic are questioned:
…..We also learned about the Incas…..

33. What makes Summary Street effective? Student Interviews:
The computer is sometimes wrong
Students are never told what to do
Problems are identified and hints are provided about how to solve them
Students always make their own decisions
Anonymity of the computer
It is not threatening to be corrected by a sub machine!

34. How effective is Summary Street?
David Wade-Stein & Eileen Kintsch (2004) Summary Street: Interactive computer support for writing. Discourse Processes, 22,
6th-graders write summaries, one with Summary Street, one on a word processor that provides only length and spelling feedback

35. Students working with Summary Street spend
much more time on task:

36. Summaries written with Summary Street receive better teacher grades:

37. Of the 10 texts used in this study, the biggest improvements were observed for the most difficult texts
The best students did not need Summary Street; for the poorest students, Summary Street did not provide enough support; the middle group profited the most.

38. How effective is Summary Street?
Marita Franzke, Eileen Kintsch, Donna Caccamise, Nina Johnson, & Scott Dooley (submitted) Summary Street®:Computer Support for Comprehension and Writing.
8th-graders practice summary writing for four weeks, with and without Summary Street; four classrooms, same teacher

39. Teacher grades assigned to summaries written with and
without Summary Street over the 4-week study period;
texts get progressively more difficult.
Summary Street
Control

40. Essentially identical results were observed for
Content scores
Inappropriate Details
Organization
Style
No improvements for
Mechanics
Plagiarism was infrequent in both groups and did not increase with practice

41. Performance on a test patterned after the Colorado Student Assessment Program (CSAP):
All students take the test under standard, paper-and-pencil conditions, so this is a test of distant transfer
Students trained with Summary Street outperform students who only used a word processor on questions requiring summarization; p < .05, effect size is .42
No statistically reliable difference between conditions on other types of test items:
Inference
Vocabulary
Fact Finding
Other

42. Summary Street in Colorado schools:
7 school districts - urban, suburban, rural
85 teachers
77 classrooms
2292 students
about 140 texts

43. A Valid Test of Reading Comprehension
Current tests are not theoretically motivated
Items have been selected because they discriminate reliably between people
Levels of comprehension are not distinguished
Multiple-choice questions are used to score tests objectively and automatically
A valid test should
Distinguish between deep and shallow comprehension
Employ a free-response format

44. Beth Mulligan, Katherine Rawson, Praful Mangalath, & Walter Kintsch Designs for a comprehension test.
In two experiments 241 college students read 6 expository texts ( words), recalled what they had written, and answered an extended inference question
Recall - shallow comprehension
Ability to reproduce the text
Inference - deep comprehension
Inference questions required a paragraph length response. Information from the text as well as general world knowledge had to be combined to obtain the answer
A variety of inference types were used, e.g.,
what were the causes of an event described in the text?
what is the general theme of the examples given in the text?

45. Recall and Inference are separate components of comprehension
Recall performances sets upper bound for inference performance

46. 21% of the students received a significantly higher grade on the recall test than on the inference test
For 58 students who received a memory score of B or better, inference scores ranged from F to B+
A confirmatory factor analysis yielded two significant factors, a memory factor and an inference factor

47. Grading Essays with LSA: Classifying the vectors representing the students’ responses
Support-vector regression segments the semantic space into areas according to the grades human raters assign to essays
New essay is given a grade according to the area of the space it is in

48. LSA grading of protocols:
Memory
Human rater inter-reliability r = .83
LSA-human correlation r = .80
Inference
Human rater inter-reliability r = .80
LSA-human correlation r = .68

49. Tools for Teachers
The success and reputation of many professions is based on the use of powerful tools
Teachers employ few tools, mostly because versatile, powerful tools were not available
Tools to empower teachers - not to substitute or replace them

50. Handouts are at: