1. Designing Accessible Reading Assessments A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis ETS

2. Designing Accessible Reading Assessments ETS Contributors Linda Cook* Kelly Bruce Jennifer Dean Dan Eignor Lois Frankel Gena Gourley Eric Hansen Branden Hart Teresa King* Skip Livingston Pavan Pillarisetti Kitty Sheehan Elizabeth Stone* Klaus Zechner

3. Designing Accessible Reading Assessments DARA Goal 4 Field test a multi-stage component-based reading assessment. –Reduce number of students performing at “chance level” –Allow students to show what they know –Push instructional to include both comprehension and reading fluently for students with reading-based LD

4. Designing Accessible Reading Assessments DARA Test Design Reading Comprehension Routing Test Component Test 1 Reading Comprehension Subtest (Audio, MP3) + Reading Fluency Subtest Component Test 2 Reading Comprehension Subtest (Standard)

5. Designing Accessible Reading Assessments Accessibility Elements Students with disabilities included in pilot test “Higher” interest passages selected based on student ratings Single column question format (increased white space and reduced wrapping of text) Included “context” sentence Panel of disability experts reviewed items and made suggested revisions (simplified language)

6. Designing Accessible Reading Assessments

10. Data Collection Design Routing Test Criterion Test (Part 1) Component Test 1 Criterion Test (Part 2) Component Test 2 Criterion Test (Part 2)

11. Designing Accessible Reading Assessments Primary Research Questions For accountability purposes, is it possible to combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? Is the Component test more accessible than the state assessment –Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments?

12. Designing Accessible Reading Assessments Other Research Questions Can we reduce the number of students scoring at chance level? Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure? Can we accurately route students based on 7, 14, 21, and 28 items? What is the best measure of oral reading fluency? How do we combine fluency and comprehension test scores (50/50, 25/75, 75/25)?

13. Designing Accessible Reading Assessments Sample 8 th Grade Students 26 Middle Schools 294 RLD (final sample=275) 194 LP (not include in this presentation) 500 Non-Disabled (final sample=486)

14. Designing Accessible Reading Assessments Description of Sample by NLD/RLD Race, Gender, and cut score impact Group SEXRACE % M% F% A.I.% A% B% H% M% P.I.% W NLD 146.1853.820.002.822.428.472.023.6380.65 NLD 246.8453.160.422.974.248.052.973.8177.54 RLD 163.1236.880.001.424.9622.706.382.8461.70 RLD 262.6937.310.75 5.228.965.224.4874.63

15. Designing Accessible Reading Assessments

17. Test Score Summaries: Route 1 N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component Fluency (obs max=222.75) Component Total: Scaled (max=48) RLD (Route 1)141 14.92 (3.96) 141 19.14 (6.16) 141 71.44 (32.37) 141 20.61 (5.60) NLD (Route 1)249 36.34 (7.95) 249 33.71 (6.12) 249 145.49 (30.18) 249 37.34 (6.27)

18. Designing Accessible Reading Assessments Test Score Summaries: Route 2 N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component Total:Scaled (max=48) RLD (Route 2)134 27.21 (6.01) 134 22.66 (7.43) 134 25.90 (8.49) NLD (Route 2)237 34.29 (8.05) 237 30.65 (7.41) 237 35.03 (8.47)

19. Designing Accessible Reading Assessments Primary Research Questions For accountability purposes, is it possible to combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? YES Is the Component test more accessible than the state assessment –Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments? YES, for Route 1

20. Designing Accessible Reading Assessments Can we reduce the number of students scoring at chance level?

21. Designing Accessible Reading Assessments Routing decision Best passage (2) Passage 1 Passages 1, 2 Passages 1, 2, 3 Full routing (8 items; Route 1: <= 3) (16 items; Route 1: <= 6) (24 items; Route 1: <= 10) (32 items; Route 1: <= 13) Reliability0.630.600.740.770.79 % (N) students assigned to Route 1 40.73% (112) 41.45% (114) 34.55% (95) 43.64% (120) 51.27% (141) % (N) students assigned to Route 1 on this test and full routing test 70.21% (99) 72.34% (102) 65.96% (93) 82.98% (117) 100% (141) Can we accurately route students based on 7, 14, 21, and 28 items?

22. Designing Accessible Reading Assessments Fluency Test Human vs. Automated Scoring ALL Passage 1 Passage 2 Passage 3 Passage 4 N547126151148122 Pearson r0.680.810.710.600.76 Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure?

23. Designing Accessible Reading Assessments Future Questions for Study and Policy Q: What is the best measure of oral reading fluency? Corrected words per minute in 1 st minute Words per minute, corrected words per minute, percent correct, rating Q: How do we combine comprehension and fluency scores 25% fluency + 75% comprehension 50/50, 75/25

24. Designing Accessible Reading Assessments Contact information Cara Cahalan Laitusis Senior Research Scientist Educational Testing Service Mailstop 09R Princeton, NJ 08541 CLaitusis@ETS.org

25. Designing Accessible Reading Assessments Extra Slides

26. Designing Accessible Reading Assessments Test Score Correlations: Route 1 Criterion Component Comprehension Component Fluency Component Total (Scaled) Criterion1.000.800.550.83 Component Comprehension 0.301.000.460.97 Component Fluency 0.27-0.021.000.67 Component Total (Scaled) 0.380.940.311.00 NLD RLD

27. Designing Accessible Reading Assessments Test Score Correlations: Route 2 Criterion Component Total (Scaled) Criterion 1.000.82 Component Total (Scaled) 0.761.00 NLD RLD