Designing Accessible Reading Assessments National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of DARA Project Research Linda Cook

Designing Accessible Reading Assessments Background Research Questions –Do students with reading-based learning disabilities receive differential performance gains from an audio (read aloud) accommodation/modification when compared to students without disabilities? Differential boost –Do reading comprehension tests and test items taken with and without a read aloud accommodation/modification measure similar constructs? Factor analysis Differential item functioning analysis –Distractor analysis –Cognitive labs

Designing Accessible Reading Assessments Background DARA Tests and Samples –Two different tests and several different samples used for the studies –Gates-MacGinitie Reading Test (GMRT) Fourth and eighth grade students (randomized design) –State standards-based assessment of English- language arts (ELA) Fourth and eighth grade students (archival data)

Designing Accessible Reading Assessments Summary of DARA Analyses

Designing Accessible Reading Assessments Overview of Presentation Gates MacGinitie Reading Test (GMRT) –Differential boost –DIF and factor analysis studies English-language arts assessment (ELA) –DIF and factor analysis studies Next steps

Designing Accessible Reading Assessments Differential Boost Description Framework for studying impact of changes in testing conditions (accommodations or modifications to the standard testing procedures) A change in testing condition may be viewed as an accommodation if students with disabilities receive a significant increase in scores compared to students without disabilities

Designing Accessible Reading Assessments Differential Boost Study Two forms of GMRT (reading comprehension section only) –Form S –Form T Two grades –Grade 4 –Grade 8 All forms and grade levels contained 48 passage based questions All questions were multiple-choice

Designing Accessible Reading Assessments Differential Boost Samples All NJ public school 4 th and 8 th grade students were invited to participate Samples –Fourth grade 1170 students –522 (RLD) –648 (NLD) –Eighth grade 855 students –394 (RLD) –461 (NLD)

Designing Accessible Reading Assessments Differential Boost Data Collection Design Group Session 1Session 2 FormTesting ConditionFormTesting Condition 1SStandardTAudio 2S TStandard 3T SAudio 4T SStandard

Designing Accessible Reading Assessments Grade 4 Differential Boost Analysis

Designing Accessible Reading Assessments Grade 8 Differential Boost Analysis

Designing Accessible Reading Assessments Differential Boost ANOVA Results of RM-ANOVA showed differential boost –Students with disabilities had significantly larger boost than students without disabilities –Significant interaction between score gain (boost) and disability status was observed Answer to first question: Students with reading based learning disabilities do exhibit differential performance gains when they take a reading test with a read aloud test change

Designing Accessible Reading Assessments Purpose of Differential Item Functioning and Factor Analyses of GMRT Determine if the test is measuring the same underlying construct (s) for the following comparisons: –Students without disabilities who take the test with and without a read aloud test change –Students with disabilities who take the test with and without a read aloud test change

Designing Accessible Reading Assessments Differential Item Functioning and Factor Analysis of GMRT Data Comparisons of item responses for four groups –Reading based learning disability (RLD), no accommodation (Standard) –Reading based learning disability (RLD), audio modification (Audio) –No disability (NLD), no accommodation (Standard) –No disability (NLD), audio modification (Audio)

Designing Accessible Reading Assessments Differential Item Functioning of GMRT Data Analyzed Form S (4 th and 8 th grade levels) and Form T (4 th and 8 th grade levels) Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change Does the GMRT measure the same underlying construct (s) for the study groups?

Designing Accessible Reading Assessments Summary Statistics for Grade 4 Form S DIF and Factor Analysis Samples

Designing Accessible Reading Assessments Differential Item Functioning (DIF) Test takers matched on proficiency level –Used total test score as matching criterion Reference Groups –Students with and without disabilities who took test under standard conditions Focal Groups –Students with and without disabilities who took test with read aloud test change

Designing Accessible Reading Assessments Differential Item Functioning (DIF) Analyses Used Mantel-Haenszel procedure with total score as criterion Mantel-Haenszel categorization –Anegligible DIF –Bslight to moderate DIF –Cmoderate to large DIF Direction of DIF Flags –Negative favors reference group –Positive favors focal group

Designing Accessible Reading Assessments Comparison 1: RLD Standard (reference group) vs. RLD Audio (focal group)

Designing Accessible Reading Assessments Comparison 2: NLD Standard (reference group) vs. NLD Audio (focal group)

Designing Accessible Reading Assessments Factor Analysis of GMRT Data Analyzed Form S (4 th and 8 th grade levels) and Form T (4 th and 8 th grade levels) Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change Does the GMRT measure the same underlying construct (s) for the study groups?

Designing Accessible Reading Assessments Factor Analysis of GMRT Data Single group exploratory factor analysis of four groups –RLD who took the test under standard conditions –RLD who took the test with an audio test change –NLD who took the test under standard conditions –NLD who took the test with an audio test change Confirmatory factor analysis –Single-group to establish single factor –Multi-group to establish base line model –Multi-group comparisons to establish measurement invariance Comparison 1RLD standard vs. RLD read aloud Comparison 2NLD standard vs. NLD read aloud

Designing Accessible Reading Assessments Summary of Multi-group CFA for RLD Standard vs. RLD Audio

Designing Accessible Reading Assessments Summary of GMRT Analyses Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without a read aloud test change Results of the differential boost study showed that students with reading based learning disabilities receive a differential boost on a reading test taken with a read aloud test change Results of DIF and factor analysis studies indicate that test measures the same underlying construct for the RLD group, or for the NLD group when taken with and without a read aloud test change

Designing Accessible Reading Assessments Analyses of English-language Arts Assessment Purpose – To examine if the test measures the same construct (s) for Students without disabilities Students with learning disabilities who took the test with and without an accommodation/modification

Designing Accessible Reading Assessments Analyses of ELA Assessment Analyses –Factor analyses –Differential item functioning analyses and distractor analyses Groups of Interest –Students with learning disabilities who took the test with and without a change in testing conditions Test –Grade 4 and grade 8 English-language Arts (ELA) assessment Focus –Determine if the test measures the same constructs for Examinees without disabilities Examinees with learning disabilities who took the test with and without a change in testing conditions

Designing Accessible Reading Assessments

English-Language Arts Grade 4 Summary Statistics

Designing Accessible Reading Assessments Factor Analyses of ELA Assessment Purpose: to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study Exploratory analyses (separately in each group) – how many factors Confirmatory (multi-group) –Establish base-line model –Confirm number of factors needed to describe data across all groups

Designing Accessible Reading Assessments Differential Item Functioning (DIF) Analyses The purpose of the DIF study was to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study Used Mantel-Haenszel procedure with total score as criterion Mantel-Haenszel categorization –Anegligible DIF –Bslight to moderate DIF –Cmoderate to large DIF Direction of DIF Flags –Negative favors reference group –Positive favors focal group

Designing Accessible Reading Assessments Comparisons Made in the DIF Study

Designing Accessible Reading Assessments Interpreting the Results of the DIF Study Fourth grade results –1 C DIF item, 10 B DIF items –5 B DIF items were reading items that favored students with disabilities who took test with read-aloud change in testing conditions –5 B DIF items (3 reading and 2 writing) favored students without disabilities –Results consistent with factor analyses results

Designing Accessible Reading Assessments Interpreting the Results of the DIF Study Eighth grade results –1 C DIF item, 7 B DIF items –Five B DIF items (4 reading and 1 writing) favored students who took test with read- aloud change in testing conditions –Again, results consistent with factor analyses results

Designing Accessible Reading Assessments Summary of Research Results GMRT –Differential boost study Students with reading disabilities show differential performance gains when taking a test with read aloud –DIF and Factor Analysis Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without read aloud very similar ELA –DIF and Factor Analysis Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without test changes very similar

Designing Accessible Reading Assessments Research Answers Do students with reading-based learning disabilities receive differential performance gains from read aloud? –yes Do reading tests and test items taken with and without a read aloud accommodation/modification measure similar constructs? –Measure the same single factor for all groups –Show little evidence of differential item performance

Designing Accessible Reading Assessments Next Steps Complete analysis of data from cognitive labs Design and carry out simulations for multi-stage reading assessment Design and carry out field trial of prototype assessment

Designing Accessible Reading Assessments Analyzing Data from Cognitive Labs Cognitive labs are a means of measuring mental processes through a think aloud protocol Administered GMRT to 34 fourth grade students and 15 eighth grade students Gained insight into test taking strategies as well as issues related to test format and content

Designing Accessible Reading Assessments Reading Comprehension Routing Test Reading Fluency Decoding and Extended Comprehension Test with Audio Extended Comprehension Test with Audio Extended Reading Comprehension Test

Designing Accessible Reading Assessments NARAP Field Trial Large data collection effort Design and develop accessible reading assessment prototypes based on NARAP Principles and Guidelines Will field trial DARA, PARA, and TARA prototypes

Designing Accessible Reading Assessments Questions?