1. Designing Accessible Reading Assessments Reading Aloud Tests of Reading Review of Research from the Designing Accessible Reading Assessments Projects Cara Cahalan Laitusis

2. Designing Accessible Reading Assessments Designing Accessible Reading Assessments (DARA) Project One of 3 National Accessible Reading Assessment Projects (NARAP) funded by OSEP/NCSER DARA research has focused on development of a multistage tailored test which measures the components of reading (fluency, decoding, comprehension, vocabulary) in isolation for some students so that read aloud of some items (vocabulary and comprehension) can be an allowable accommodation

3. Designing Accessible Reading Assessments Reading Comprehension Routing Test Reading Fluency Decoding and Extended Comprehension Test with Audio Extended Comprehension Test with Audio Extended Reading Comprehension Test

4. Designing Accessible Reading Assessments Background Research Questions –Do students with reading-based learning disabilities receive differential performance gains from read aloud when compared to students without disabilities? –Do reading comprehension tests and test items taken with and without a read aloud measure similar constructs? Predictive Validity Factor analysis Differential item functioning analysis –Distractor analysis –Cognitive labs

5. Designing Accessible Reading Assessments Background DARA Tests and Samples –Two different tests and several different samples used for the studies –Gates-MacGinitie Reading Test (GMRT) Fourth and eighth grade students (randomized design) –State standards-based assessment of English- language arts (ELA) Fourth and eighth grade students (archival data)

6. Designing Accessible Reading Assessments Summary of DARA Analyses

7. Designing Accessible Reading Assessments Differential Boost Study Two forms of GMRT (reading comprehension section only) –Form S –Form T Two grades –Grade 4 –Grade 8 All forms and grade levels contained 48 passage based questions All questions were multiple-choice

8. Designing Accessible Reading Assessments Differential Boost Samples All NJ public school 4 th and 8 th grade students were invited to participate Samples –Fourth grade 1170 students –522 (RLD) –648 (NLD) –Eighth grade 855 students –394 (RLD) –461 (NLD)

9. Designing Accessible Reading Assessments Differential Boost Data Collection Design Group Session 1Session 2 FormTesting ConditionFormTesting Condition 1SStandardTAudio 2S TStandard 3T SAudio 4T SStandard

10. Designing Accessible Reading Assessments Differential Boost ANOVA Results of RM-ANOVA showed differential boost –Students with disabilities had significantly larger boost than students without disabilities –Significant interaction between score gain (boost) and disability status was observed Answer to first question: Students with reading based learning disabilities do exhibit differential performance gains when they take a reading test with a read aloud test change

11. Designing Accessible Reading Assessments Validity of Scores (regression) Answer to second question (predictive validity): –Test taken with read aloud do not predict teachers ratings of reading comprehension as well as tests taken under standard conditions. –However combining read aloud scores with reading fluency scores results in equal (or better) prediction of teacher ratings than tests taken under standard conditions.

12. Designing Accessible Reading Assessments Differential Item Functioning and Factor Analysis of GMRT Data Comparisons of item responses for four groups –Reading based learning disability (RLD), no accommodation (Standard) –Reading based learning disability (RLD), audio modification (Audio) –No disability (NLD), no accommodation (Standard) –No disability (NLD), audio modification (Audio)

13. Designing Accessible Reading Assessments Differential Item Functioning (DIF) Test takers matched on proficiency level –Used total test score as matching criterion Reference Groups –Students with and without disabilities who took test under standard conditions Focal Groups –Students with and without disabilities who took test with read aloud test change

14. Designing Accessible Reading Assessments Differential Item Functioning (DIF) Analyses Used Mantel-Haenszel procedure with total score as criterion Mantel-Haenszel categorization –A—negligible DIF –B—slight to moderate DIF –C—moderate to large DIF Direction of DIF Flags –Negative favors reference group –Positive favors focal group

15. Designing Accessible Reading Assessments Factor Analyses Purpose: to examine whether or not the assessments (GMRT and State ELA) measure the same construct(s) for the groups in our study Exploratory analyses (separately in each group) – how many factors Confirmatory (multi-group) –Establish base-line model –Confirm number of factors needed to describe data across all groups

16. Designing Accessible Reading Assessments Summary of GMRT Analyses Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without a read aloud test change Results of the DIF study indicated very few C level DIF items (1-2% of items). More items (10- 15%) were flagged for B level DIF but the direction of the DIF was equally distributed between the reference and focal groups. Results of the factor analysis studies the test is measuring the same single factor for all subgroups.

17. Designing Accessible Reading Assessments Summary of State ELA Analyses Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without read aloud test. Results of DIF analyses indicated the level of DIF was less than the DIF found in the GMRT analyses but did not include a non-disabled read aloud comparison group. Results indicated same single factor structure for all of the subgroups studied.

18. Designing Accessible Reading Assessments Summary of Research Results Differential boost study –Students with reading disabilities show differential performance gains when taking a test with read aloud (findings consistent with Fletcher et al. 2005 and Crawford & Tindal, 2004). –Read aloud and standard scores were highly correlated (same as two forms of test) for students without disabilities –Read aloud and standard scores were moderately correlated for students with learning disabilities –Read aloud score was not a good predictor or reading comprehension (based on teacher ratings) for students with reading- based learning disabilities (RLD) –Read aloud score plus reading fluency score predicted teachers ratings of reading comprehension as well or better than standard reading comprehension for RLD students

19. Designing Accessible Reading Assessments Summary of Research Results GMRT and State ELA DIF and Factor Analysis –Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without read aloud are very similar Few items with C level DIF under both conditions Measure same single factor

20. Designing Accessible Reading Assessments Next Steps Complete analysis of data from cognitive labs Design and carry out simulations for multi-stage reading assessment Design and carry out field trial of prototype assessment

21. Designing Accessible Reading Assessments Questions?