Scalable and Sustainable Technologies for Reading Instruction An Interagency Education Research Initiative: NSF - IES - NIH Principal Investigators: Walter Kintsch, Donna Caccamise, Ron Cole, Lynn Snyder, & Richard Olson

The Push to Improve Literacy Every child can learn to read, yet Literacy is a national problem We have good basic research, problem is To develop practical implementations To get them used widely in schools IERI initiative

Colorado Literacy Tutor Tools for Teachers K-6th grade: emphasis on foundational reading skills: Interactive books for practice plus tutors for special problems 6th-14th grade: emphasis on comprehension and learning from text

Underlying Cognitive Science Research Tutoring is an effective method of instruction: Human tutors produce gains between .4 and 2.3 standard deviations over classroom teaching Most human tutors have only modest domain knowledge and rarely use sophisticated tutoring strategies Existing Artificial Intelligence tutors such as AUTOTUTOR (Graesser et a., 1999) and ATLAS (van Lehn et al., 2000) approach the effectiveness of human tutors (Graesser et al., 2001)

Underlying Cognitive Science Research Creating the Next Generation of Intelligent Animated Conversational Agents: Use of synthesized speech and speech recognition Conversation with a talking head Learners respond in English in addition to click-and-point

Underlying Cognitive Science Research Latent Semantic Analysis: Theory, Technology and Practice LSA as a map of meaning High-dimensional semantic space Text meaning as the sum of the words Limitations: Learns only from words Neglect of order and syntax

A map is a 2-dimensional representation of distance and direction data: Data: Map:

A semantic space is a k-dimensional representation of word co-occurrence data (k<<<m,n): Data: Semantic Space:

High-dimensional spaces are capable of representing complex semantic relations: TREE DOG .70 BARK .06 .66

The meaning of a text is the sum of the words in LSA: The accident victim was flown to the hospital This sentence is more similar to (1) than (2), although it shares two content words with (2) and none with (1): (1) A helicopter brought the man hurt in the crash to the emergency room cosine = .29 (2) An accident occurred in the street where the victim could not get assistance cosine = .24

Interactive Books & Reading Tutors Ron Cole, Donna Caccamise, Lynn Snyder and the Colorado Literacy Tutor Team Center for the Study of Spoken Language University Of Colorado

Components of the Reading Tutor: Foundational Skills Tutors Teach underlying reading skills Interactive Books Teach fluent reading & comprehension Managed Learning Environment Enroll students, track and display progress, manage individual study plans, etc.

Beginning sounds

Letter sounds

Word reading

Silent “e”

Interactive Books Teacher or child selects appropriate book Child selects animated tutor or “coach” Animated coach guides child through book Coach can read words, sentences paragraphs or whole page Coach asks comprehension, inference and summary questions using speech recognition to determine accuracy of responses Each level of a book has the capability of invoking specific reading tutors if a child needs more foundational skill training

Read to me and read aloud

Click on image question interaction

Multiple choice question interaction

Summarization using Summary StreetTM

Comprehension and Learning Walter Kintsch, Donna Caccamise, Lynn Snyder, and the LSA Research Group Institute of Cognitive Science University of Colorado Tom Landauer Pearson Knowledge Technologies & University of Colorado

Comprehension and Learning LSA computes how similar in meaning two texts are The contents of a student’s essay can be compared with other essays and standards Students receive feedback about the content of their writing - guidance for revising A tool for self-assessment: feedback allows students to judge how well they are doing and what needs more work

Summary Street® An existing, classroom-tested tested system that provides content-based feedback to middle-school students summarizing a text: indicates what content is missing what might be overemphasized flags apparent problems sentences helps with the organization of the material

The teacher keeps track of how much and how well the student did:

Provides hints about how the summary could be shortened: Sentences are flagged that are very similar in meaning: …...They also wrote books on paper. The books were made from bark paper that they folded together….. Sentences that appear unrelated to the topic are questioned: …..We also learned about the Incas…..

What makes Summary Street effective? Student Interviews: The computer is sometimes wrong Students are never told what to do Problems are identified and hints are provided about how to solve them Students always make their own decisions Anonymity of the computer It is not threatening to be corrected by a sub machine!

How effective is Summary Street? David Wade-Stein & Eileen Kintsch (2004) Summary Street: Interactive computer support for writing. Discourse Processes, 22, 333-362. 6th-graders write summaries, one with Summary Street, one on a word processor that provides only length and spelling feedback

Students working with Summary Street spend much more time on task:

Summaries written with Summary Street receive better teacher grades:

Of the 10 texts used in this study, the biggest improvements were observed for the most difficult texts The best students did not need Summary Street; for the poorest students, Summary Street did not provide enough support; the middle group profited the most.

How effective is Summary Street? Marita Franzke, Eileen Kintsch, Donna Caccamise, Nina Johnson, & Scott Dooley (submitted) Summary Street®:Computer Support for Comprehension and Writing. 8th-graders practice summary writing for four weeks, with and without Summary Street; four classrooms, same teacher

Teacher grades assigned to summaries written with and without Summary Street over the 4-week study period; texts get progressively more difficult. Summary Street Control

Essentially identical results were observed for Content scores Inappropriate Details Organization Style No improvements for Mechanics Plagiarism was infrequent in both groups and did not increase with practice

Performance on a test patterned after the Colorado Student Assessment Program (CSAP): All students take the test under standard, paper-and-pencil conditions, so this is a test of distant transfer Students trained with Summary Street outperform students who only used a word processor on questions requiring summarization; p < .05, effect size is .42 No statistically reliable difference between conditions on other types of test items: Inference Vocabulary Fact Finding Other

Summary Street in Colorado schools: 7 school districts - urban, suburban, rural 85 teachers 77 classrooms 2292 students about 140 texts

A Valid Test of Reading Comprehension Current tests are not theoretically motivated Items have been selected because they discriminate reliably between people Levels of comprehension are not distinguished Multiple-choice questions are used to score tests objectively and automatically A valid test should Distinguish between deep and shallow comprehension Employ a free-response format

Beth Mulligan, Katherine Rawson, Praful Mangalath, & Walter Kintsch Designs for a comprehension test. In two experiments 241 college students read 6 expository texts (147 - 322 words), recalled what they had written, and answered an extended inference question Recall - shallow comprehension Ability to reproduce the text Inference - deep comprehension Inference questions required a paragraph length response. Information from the text as well as general world knowledge had to be combined to obtain the answer A variety of inference types were used, e.g., what were the causes of an event described in the text? what is the general theme of the examples given in the text?

Recall and Inference are separate components of comprehension Recall performances sets upper bound for inference performance

21% of the students received a significantly higher grade on the recall test than on the inference test For 58 students who received a memory score of B or better, inference scores ranged from F to B+ A confirmatory factor analysis yielded two significant factors, a memory factor and an inference factor

Grading Essays with LSA: Classifying the vectors representing the students’ responses Support-vector regression segments the semantic space into areas according to the grades human raters assign to essays New essay is given a grade according to the area of the space it is in

LSA grading of protocols: Memory Human rater inter-reliability r = .83 LSA-human correlation r = .80 Inference Human rater inter-reliability r = .80 LSA-human correlation r = .68

Tools for Teachers The success and reputation of many professions is based on the use of powerful tools Teachers employ few tools, mostly because versatile, powerful tools were not available Tools to empower teachers - not to substitute or replace them

Handouts are at: http://www.reading.org/association/meetings/annual_handouts.html