The Challenge of Assessing Reading for Understanding to Inform Instruction Barbara Foorman, Ph.D. Florida Center for Reading Research Florida State University

Problem What is reading comprehension and how is it measured? Dissatisfaction with traditional “mean proficiency” approaches that judge students’ achievement relative to a benchmark has led to an interest in basing accountability on individual academic growth. Yet the challenge of informing instruction remains

Simple View of Reading The ability to read and obtain meaning from what was read. Decoding of text Comprehension of language Reading to gain meaning Gough and Tunmer (1986) Recognizing words in text & sounding them out phonemically The ability to understand language Multiplied byEquals

Scarborough (2002)

Constructs/Measures by Grade ConstructMeasureGrade Phonological AwarenessTOPEL; CTOPPpreK-G1 Print LN/LSTOPEL; FAIRpreK-K Word ReadingTOWRE; FAIR; SARAG1-G9 Vocabulary (+MORPH)TOPEL; PPVT; FAIR; SARApreK-G9 SyntaxCELFpreK-G9 Listening ComprehensionCELF; FAIRpreK-G6 Reading Comprehension EfficiencyTOSREC; FAIR; SARAG1-G9 TextGMRT; FAIR; SARAG1-G9

TEXT ACTIVITY READER A heuristic for thinking about reading comprehension (Sweet & Snow, 2003) Word recognition, vocabulary, background knowledge, strategy use, inference-making abilities, motivation Text structure, vocabulary, genre discourse, motivating features, print style and font Purpose, social relations, school/classroom/peers/ families Environment, cultural norms

Components of Reading Comprehension (Perfetti, 1999) Comprehension Processes Inferences Situation Model Text Representation Parser General Knowledge Linguistic System Phonology Syntax Morphology Meaning and Form Selection Word Representation Orthographic Units Phonological Units Visual Input Word Identification Lexicon Meaning Morphology Syntax Orthography Mapping to phonology

Goal of RFU Assessment Grant ETS and FSU/FCRR are designing a new assessment system consisting of: 1.Purpose-driven, scenario-based, summative assessment (textbase + situation model; motivation; prior knowledge; text complexity. 2.Component skill measures to predict achievement trajectories and provide additional information about non-proficient readers.

NAEP Framework Literary Text ● Fiction ● Literary Nonfiction ● Poetry Informational Text ● Exposition ● Argumentation and Persuasive Text ● Procedural Text and Documents Cognitive Targets Distinguished by Text Type Locate/Recall Integrate/Interpret Critique/Evaluate

10 Advanced G8 students at the Advanced level should be able to :  Make complex inferences  Critique point of view  Evaluate character motivation  Describe thematic connections across literary texts  Evaluate how an author uses literary devices to convey meaning G8 students at Advanced level should be able to:  Make complex inferences  Evaluate the author’s purpose  Evaluate strength & quality of supporting evidence  Compare & contrast ideas across texts  Critique causal relations Proficient G8 students at the Proficient level should be able to: , Make inferences that describe problem and solution, cause and effect  Analyze character motivation  Interpret mood or tone  Explain theme  Identify similarities across texts  Analyze how an author uses literary devices to convey meaning  I nterpret figurative language G8 students at Proficient level should be able to:  Summarize major ideas  Draw conclusions  Provide evidence in support of an argument  Describe author’s purpose  Analyze and interpret implicit causal relations Basic G8 students at the Basic level should be able to:  I nterpret textually explicit information  Make simple inferences  Identify supporting details  Describe character’s motivation  Describe the problem  Identify mood G8 students at the Basic level should be able to:  Locate the main idea  Distinguish between fact and opinion  Make inferences  Identify author’s explicitly stated purpose  Recognize explicit causal relations Achievement Level LiteraryInformational Achievement Levels for Grade 8 NAEP Reading

Current Approaches to Measuring Growth TN uses EVASS (Sanders, 2000): deviation from mean level of growth. Ohio uses achievement plus growth from previous and current year: above, met, and below expected growth. Colorado uses the Student Growth Percentile Model based on conditional percentile ranks and quantile regression (Betebenner, 2008).

Interim Assessments Why not measure growth within a year? Interim assessments are mid-way between formative and summative and can be aggregated up above the classroom level. Interim assessments are ideal for informing learning, instruction, and placement.

Florida Assessments for Instruction in Reading A K-2 assessment system administered to individual students 3 times a year, with electronic scoring, Adobe AIR version, and PMRN reports linked to instructional resources. A 3-12 computer-based system where students take the assessments 3 times a year. Several tasks are adaptive. PMRN reports are available, linked to instructional resources. Printed toolkit available.

The K-2 “Big Picture” Map Broad Screen/Progress Monitoring Tool (BS/PMT) “All” students Letter Naming & Sounds Phonemic Awareness Word Reading Broad Diagnostic Inventory (BDI) “All” students “Some” students for vocabulary Listening Comprehension Reading Comprehension Vocabulary Spelling (2 nd grade only) Targeted Diagnostic Inventory (TDI) “Some” students; some tasks K = 9 tasks 1 st = 8 tasks 2 nd = 6 tasks Ongoing Progress Monitoring (OPM) “Some” students K – 2 = TDI tasks 1 – 2 = ORF

K-2 Targeted Diagnostic Inventory (TDI) Kindergarten Print Awareness Letter name and sound knowledge Phoneme Blending Phoneme Deletion Word Parts/Initial Letter Sound Connection Initial Letter Sound Connection Final Word Building –Initial Consonants Word Building –Final Consonants Word Building –Medial Vowels First Grade Letter Sound Knowledge Phoneme Blending Phoneme Deletion Initial Phoneme Deletion Final Word Building –Consonants Word Building –Vowels Word Building –CVC /CVCe Word Building –Blends Second Grade Phoneme Deletion Initial Phoneme Deletion Final Word Building –Consonants Word Building –CVC /CVCe Word Building –Blends & Vowels Multisyllabic Word Reading

The K – 2 “Score” Map BS/PMTPRS = Probability of Reading Success BDI LC = Listening Comprehension Total questions correct (implicit/explicit) RC = Reading Comprehension Total questions correct (implicit/explicit), Fluency, Percent Accuracy Target Passage VOC = Vocabulary Percentile Rank SPL = Spelling Percentile Rank TDI ME = Meets Expectations BE = Below Expectations OPM ORF = Adjusted Fluency OPM TDI Tasks = ME or BE and Raw Score 16

Broad Screen/Progress Monitoring Tool Reading Comprehension Task ( 3 Times a Year ) Targeted Diagnostic Inventory Maze & Word Analysis Tasks Diagnostic Toolkit ( As Needed ) Ongoing Progress Monitoring ( As Needed ) If necessary Grades 3-12 Assessments Model

RC Screen –Helps us identify students who may not be able to meet the grade level literacy standards at the end of the year as assessed by the FCAT without additional targeted literacy instruction. Mazes –Helps us determine whether a student has more fundamental problems in the area of text reading efficiency and low level reading comprehension. Relevant for students below a 6 th grade reading level. Word Analysis –Helps us learn more about a student's fundamental literacy skills- -particularly those required to decode unfamiliar words and read and write accurately. Purpose of Each 3-12 Assessment

How is the student placed into the first passage/item? TaskPlacement Rules Reading Comprehension - Adaptive For AP 1, the first passage the student receives is determined by: Grade level and prior year FCAT (if available) If no FCAT, students placed into a specific grade-level passage All 3 rd grade students are placed into the same initial passage For AP 2 and 3, the first passage is based on students’ final ability score from the prior Assessment Period (AP). Maze – Not adaptive Two predetermined passages based on grade level and assessment period (AP). WA - Adaptive AP 1-3 starts with predetermined set of 5 words based on grade level. Student performance on this first set of 5 words determines the next words the student receives words given at each assessment period based on ability.

How is the student placed into subsequent passages? Based on the difficulty of the questions the student answers correctly on the first passage, the student will then be given a harder or easier passage for their next passage. –Difficulty of an item is determined using Item Response Theory (IRT). Estimates based on theta/SE. Because of this, the raw score of 7/9 for Student A and 7/9 for Student B, when reading the same passage, does not mean they will have the same converted scores.

21 Florida Assessments for instruction in Reading (FAIR): 3-12 Measures Type of AssessmentName of Assessment Broad Screen/Progress Monitoring Tool (BS/PMT) – Appropriate for ‘All’ students Reading Comprehension (RC) Targeted Diagnostic Inventory (TDI) – “Some” students Maze Word Analysis (WA) Ongoing Progress Monitoring (OPM) – “Some” students Maze ORF RC Informal Diagnostic Toolkit (Toolkit) – “Some” students Phonics Inventory Academic Word Inventory Lexiled Passages Scaffolded Discussion Templates

22 FAIR 3-12 Score Types Reading Comprehension - BS/PMT  FCAT Success Probability (FSP) Color- coded  Percentile  Standard Score  Lexile®  Ability Score and Ability Range  FCAT Cluster Area Scores Maze - TDI  Percentile  Standard Score  Adjusted Maze Score Word Analysis - TDI  Percentile  Standard Score  Ability Score (WAAS) OPM  RC – Ability Score, Ability Range, Cluster Scores  Maze – Adjusted Maze Score  ORF (3 rd – 5 th ) Adjusted Fluency Score

Research Questions How do FAIR RC scores correlate to FCAT? What is the value-added of FAIR RC to prior FCAT? Does FAIR RC significantly reduce identification errors above and beyond prior FCAT? What is the value added of growth vs. difference? What is the value added of growth and prior FCAT?

FL Comprehensive Assessment Test (FCAT): Grades 3-10 FCAT Reading is a group-administered, criterion- referenced test consisting of 6-8 informational & literary passages, with 6-11 multiple choice items per passage. 4 content clusters in 0910: 1) words & phrases in context; 2) main idea; 3) comparison/cause & effect; 4) reference & research. Reliability up to.9; content & concurrent validity (FLDOE, 2001; Schatschneider et al., 2004)

Participants: 951,893 students in FL Public Schools in PNRN GradeNumber of Students Grade 3156,265 Grade 4130,119 Grade 5129,856 Grade 6107,737 Grade 7108,394 Grade 8105,071 Grade 9112,686 Grade 10101,765

Analyses Correlations of FCAT SSS & FAIR’s FSP & SS Multiple regression of current FCAT by: prior FCAT; FCAT plus FAIR RCA; FAIR RCA alone. Comparisons of NPP in predicting current FCAT: 1) prior FCAT, or 2) prior FCAT + FAIR’s RCA. HLM comparisons of current FCAT by: Bayesian growth in RCA vs. difference scores. HLM comparisons of current FCAT: Bayesian growth with & without prior FCAT

Fall Winter Spring Grade RC Screen Standard ScoreFSP RC Screen Standard ScoreFSP RC Screen Standard ScoreFSP Table 1: Correlations between the FCAT and both RC Screen & FSP

Grade Variables Prior FCAT 49.5%58.6%53.0%60.0%64.7%57.5%59.0% Prior FCAT + RCA 53.3%62.5%60.3%63.0%68.4%59.2%61.3% Unique Var. 3.8%3.9%7.3%3.0%3.7%1.7%2.3% Table 2: Estimates of Variance Explained by Prior FCAT and FCAT + RCA

Grade Variables Prior FCAT86% 84%78%70%61% Prior FCAT + RCA98%94%93%85%92%90%54% Table 3: Comparing Negative Predictive Power in Predicting Current FCAT with either Prior FCAT or Prior FCAT + FAIR’s RCA

GradeBaseBayesian SlopeSimple Difference Table 4. HLM Estimates of FCAT Comparing R 2 in Growth vs. Difference Score Models

GradeFCAT FCAT + RCA Bayesian Slope Simple Difference Table 5. HLM Estimates of FCAT Comparing R 2 in Autoregressive, Growth, and Difference Score Models

Improvements on “Mean Proficiency” approach FAIR + Prior FCAT accounts for up to 7% unique variance; Simple Difference approach to measuring FAIR growth accounts for up to 2-3% unique variance beyond prior FCAT + FAIR; Improvements in prediction lead to: –Reduction in mis-identification of risk (from 14%-30% with prior FCAT to 2%-15% with prior + FAIR) –Better placement for reading intervention

Challenges to Implementing FAIR How is progress monitored? (score types, RTI decision-making) What is the value of an adaptive test? Benchmark mania Scaling professional development, making instructional resources available, and building local capacity

Reading Comprehension MazesWord Analysis AP ScorePM score AP ScorePM scoreAP ScorePM score student lexile score student lexile score %ile & SS RCAS Percentile rank Percentile rank FSP Adj. Maze SS WAAS AP = Assessment Period; PM = progress monitoring; SS = standard score

Value of Computer-Adaptive Tests Provides more reliable & quicker assessment of student ability than a traditional test, because it creates a unique test tailored to the individual student’s ability. Provide more reliable assessments particularly for students at the extremes of ability (extremely low ability or extremely high ability). Grade-level percentiles are currently provided; Grade Equivalent scores will be provided next year.

Benchmark Conundrum Benchmark tests rarely have enough items to be reliable at the benchmark level. Besides, teaching to benchmarks (e.g., “the student will use context clues to determine meanings of unfamiliar words”) results in fragmented skills. Teach to the standard(s) (e.g., “The student uses multiple strategies to develop grade appropriate vocabulary). Assess at aggregate levels (e.g., Reporting Categories), if CFA show categories are valid.

FCAT 2.0 Reporting Categories Reporting Category 1: Vocabulary Reporting Category 2: Reading Application Reporting Category 3: Literary Analysis- Fiction/Nonfiction Reporting Category 4: Informational Text/ Research Process

FCAT 2.0: Benchmarks x Grade Category 1Category 2Category 3Category 4Total Grade /

Possible Benchmark Solutions Stop-gap: start each students with grade-level passage. Provide % correct on Reporting Categories. Then continue to current adaptive system to obtain reliable, valid FSP and RCAS. For the future: –Align FAIR to the Common Core. Develop grade- level CAT that is item adaptive. Incorporates vocabulary depth/breadth. –Challenges: Dimensionality; multi-dimensional IRT; testlet effects.

Kinds of Vocabulary Knowledge Word Meanings Text Definitional Usage/contextual Definitional/contextual Relational Morphological

Thank You Comments or Questions?