I C A R E: Independent Comprehensive Adaptive Reading Evaluation: WHY & HOW Barbara Wise, Ron Cole, Bryan Pellom, Sarel Van Vuuren Center for Spoken Language Research University of Colorado www.cslr.colorado.edu & with Lynn Snyder, Tim Weston, Richard Olson, Jan Keenan, Jariya Tuantranont, Nattawut Ngampatipatpong, Andi Hagen, & Sharon Vaughn Institute for Educational Science November 12, 2004

Colorado Literacy Project A comprehensive reading program: Teach any child to read and learn from text Optimize instruction for different kids Scale up nationally and globally A test bed for research on Reading and comprehension instruction Individual differences in instructional needs Interactions with Virtual Teachers

Components of Colorado Literacy Project ICARE: 3rd -7th (K--3rd??) Independent Comprehensive Adaptive Reading Evaluation: (also Lu Sessions, Tammy Tomczyk) Foundations to Literacy: KG--3rd (--> 5??) Foundational reading exercises Books for fluent reading and understanding (also Scott Schwartz, Taylor Struemph, Gordon Golding, Pam Cole) Summary Street: 5-->; (K--5??) Comprehension and learning Interactive Books and LSA (Walter & Eileen Kintsch, Tom Landauer, Donna Caccamise, Dave Wade-Stein)

Sponsors & Grants IES: ICARE: Independent Comprehensive Adaptive Reading Evaluation $1,600,000 Wise, B., Cole, R., Pellom, B, & Van Vuuren, S. 10/1/04--9/30/08 NICHD/IERI : 1R01HD-44276.01 Cole, R., Barker, L., Schwartz S., Snyder, L., Wise, B. "IERI: Scaling up Reading Tutors," $1,000,000.00, NIH. 9/27/02 - 9/30/04. NSF/ITR: REC-0115419 - Kintsch, W., Landauer T., Caccamise, D., Cole, R., "ITR/PE: Latent Semantic Analysis Theory and Technology," $2,400,000, NSF, 09/01/01 - 08/31/06. NSF/IERI: EIA-0121201 - Kintsch, W., Caccamise, D., Cole, R., Olson, R., Snyder, L., "IERI: Scalable and Sustainable Technologies for Reading Instruction and Assessment," $5,997,404, NSF, 07/01/01 - 06/26/06 NSF/ITR: IIS-0086107 - Cole, R., van Santen, J., Movellan, J., "ITR: Creating the Next Generation of Intelligent Animated Conversational Agents," $4,000,000, NSF, 09/01/00 - 08/31/05. and past NIH grants HD11683 and HD22223 Talking Computers Help Children with Reading Disabilities, and for the bases of many of the word-level tests. Olson, R. & Wise, B. 1989-1999.

Outline Why build I C A R E ? Goal: Independent Comprehensive Adaptive How? Simple Model for Comprehensive & Adaptive Technology for Independence 4 stages for building and validating Where are knowledge and measures strong, where gaps Discussion and Plans

ICARE Independent Comprehensive Adaptive Reading Evaluation Why build It? Need: Instructional Profiling and RtI measures Problem: Not enough trained evaluators nor time Goal: Positive, Engaging, Efficient Good readers out in < 15 minutes Kids in Need Out in 20-- 45 minutes Work in English, ESL, Spanish

ICARE Independent Comprehensive Adaptive Reading Evaluation Two Kinds of Assessment Needed (Fuchs et al 2004) Evaluation Level for Identification and Evaluation of Programs WJ-III, CTOPP, & ICARE are this kind Response to Intervention RtI, CBM for stability, ongoing evaluation DIBELS and FtL are this kind

ICARE Independent Comprehensive Adaptive Reading Evaluation Efficient, Effective, Engaging Find kids in need or at risk for Difficulties in Reading Comprehension For different reasons Funded Grades 3rd -- 7th Proposed KG -- 3rd

ICARE Independent Comprehensive Adaptive Reading Evaluation Goals: Instructional Needs Profiles to Guide areas of needed Instruction For classroom Instruction For Study Plan for FtL Evaluate effectiveness of Programs 2 forms for repeated measure

All ICARE measures (eventually) Adaptive Only take tests if you need them Only take items you need Independent Version with Speech Recognition Version without Speech Recognition (Forced Choice, mouse-clicked)

Can Speech Rec do it? Speech recognition for Reading out loud, for fluency, accuracy: SONIC: Bryan Pellom Accuracy 94% with known text, & improving Spoken responses to focused questions Transcription of spoken summaries for automatic grading And if not-- Mouse Clicks!

How to Make it Adaptive ? How will we build it and validate it? Use Simple Model for basic profiles What underlies reading comprehension ? Get Profile of Instructional Needs for all with needs or at risk How will we build it and validate it? Where are knowledge and measures strong, where gaps

Theoretical Base Simple Model of Reading (Gough et al., 1996) Skilled Reading = product of Processes underlying Word Reading: Alphabet, Phonological Awareness, Decoding, Sight, Automaticity times Listening Comprehension: Construct Gist , Vocabulary , Working Memory.... Text Cohesion (cross-text connections, bridging inferences) Rapid Naming affects both

Basic Profiles from Simple Model

ICARE: Efficient and Adaptive.. Time Goals for Good Readers Exit Kids (<15 min) if at grade level on “screening measures” Fluent and Accurate Word Reading Fluent Reading with Comprehension Spelling ( risk for later problems ? ) Vocabulary (risk for later problems ?) Congratulations! Keep reading and writing well!

ICARE Time Goals for Kids At Risk for or with Difficulties in Reading Comprehension Problem at Word Level? (20-25 min) Investigate processes under Word Reading Problem w/ Listening Comp (20-25 min) Problem with both (40-45 min)

Possible Word Level Tests if difficulty with Time limited Word reading Untimed Word Reading Rapid Naming Letters & Numbers Spelling Time-limited or Untimed Decoding Orthographic Coding Phonological Awareness Phoneme Deletion (if Decode < = 3rd or) Phoneme Reversal (if Decode > 3rd

Comprehension Measures if Reading Comprehension < Grade Listening Comp and Vocab: OK? Just do Word Level Measures, otherwise: Paragraph Listening Main idea , summaries Concrete and abstract inferencing Cross-text cohesion, pronoun referents Reading Comp at TWR level Prosody and fluency at TWR level Reflects and develops comprehension Phonological or Working Memory

Concurrent Validity Measures For ICARE measures & Adaptive System 1. Woodcock-Johnson III Untimed Word Read, Spell, Reading Comp, Fluency, Vocabulary, Abstract Words, Analogies 2. CELF: Listening Comprehension 3. GRADE: Reading Comp, Analogies, Inferencing 4. WORD: Vocabulary, Multiple Meanings 5. TOWRE Time-limited word reading TWR & Time-limited decoding 6. CTOPP Phoneme Deletion, Reversal, Rapid Naming, Phonological Memory 7. PIAT Spelling: Orthographic Coding

ICARE will Build on FtL Interface and Functions

What is a Virtual Evaluator? A Lifelike 3D Computer Character with personality, attitude, and built-in knowledge about language that engages users in natural face-to-face conversation to produce engaging and effective learning experiences

Marge

Foundational Reading Skills Exercises More than 30 exercises: Alphabet, Phonological awareness, Decoding and encoding (spelling) sound-out words Reading and Spelling Sight words Automaticity Vocabulary Contextual feedback, reinforcement and individualized instruction by the Virtual Teacher

Interactive Books Student can listen as teacher reads Student can read aloud, get help, & computer listens and tracks reading Student can read silently & get help Multimodally assess comprehension

Interactive Books Engage kids in comprehension activities Background Check Main Idea and Inference questions Summarize Word- spotting Base for assessments of Reading and Listening Comprehension Summarizing Inferencing (Abstract and Concrete) Vocabulary (Single and Multiple meanings)

Click on image ? interaction

ICARE measures In various stages of development 1. Develop Items..... Have for many word level, from Twin & ROSS Develop for new measures, under LC especially 2. Test in Long Form To develop ordering, speech recognition 3. Test in Adaptive Form Skip up to error, down to basal, up to ceiling 5 / 7 X Concurrent validity 4. Adaptive System ... ESL and Bilingual Spanish Versions

Comprehension, Fluency and Prosody at TWR level

Main Idea Multiple choice

Summarization using Summary StreetTM

Untimed Letter Sound ID Instruction: Click on the letter that makes the beginning sound of the word I say. pig

Rapid Naming of Letters with speech recogntion Instruction: Read as many letters as you can in 45 seconds

Untimed Word reading without speech recogntion Instruction: Click on the word I say fed

Time-limited Word Reading without speech recognition Words stay on screen for 2 sec. then bubbles pop. Score final response girl

Time-limited Word Reading with speech recognition Word stays on screen for 1 sec. then bubble pops. Child reads. Score final response Keep up the good work!

Time-limited Word Reading with speech recognition Word stays on screen for 1 sec. then bubble pops. Child reads. Score final response Keep up the good work!

Phoneme Elision without speech recogntion Say “nest.” What sounds like “nest” without sss?

Articles and Websites for Foundations to Literacy, COLit, ICARE www.cslr.colorado.edu Learning to Read with a Virtual Tutor: Foundations to Literacy Wise, Cole, Van Vuuren, Schwartz, Snyder, Ngampatipatpong, Tuantranont, & Pellom. In Kinzer, C. & Verhoeven, L. (Eds). Interactive Literacy Education. Mahwah, NJ: Lawrence Erlbaum and on website above www.colit.edu