Designing Accessible Reading Assessments A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis ETS

Designing Accessible Reading Assessments ETS Contributors Linda Cook* Kelly Bruce Jennifer Dean Dan Eignor Lois Frankel Gena Gourley Eric Hansen Branden Hart Teresa King* Skip Livingston Pavan Pillarisetti Kitty Sheehan Elizabeth Stone* Klaus Zechner

Designing Accessible Reading Assessments DARA Goal 4 Field test a multi-stage component-based reading assessment. –Reduce number of students performing at “chance level” –Allow students to show what they know –Push instructional to include both comprehension and reading fluently for students with reading-based LD

Designing Accessible Reading Assessments DARA Test Design Reading Comprehension Routing Test Component Test 1 Reading Comprehension Subtest (Audio, MP3) + Reading Fluency Subtest Component Test 2 Reading Comprehension Subtest (Standard)

Designing Accessible Reading Assessments Accessibility Elements Students with disabilities included in pilot test “Higher” interest passages selected based on student ratings Single column question format (increased white space and reduced wrapping of text) Included “context” sentence Panel of disability experts reviewed items and made suggested revisions (simplified language)

Designing Accessible Reading Assessments

Data Collection Design Routing Test Criterion Test (Part 1) Component Test 1 Criterion Test (Part 2) Component Test 2 Criterion Test (Part 2)

Designing Accessible Reading Assessments Primary Research Questions For accountability purposes, is it possible to combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? Is the Component test more accessible than the state assessment –Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments?

Designing Accessible Reading Assessments Other Research Questions Can we reduce the number of students scoring at chance level? Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure? Can we accurately route students based on 7, 14, 21, and 28 items? What is the best measure of oral reading fluency? How do we combine fluency and comprehension test scores (50/50, 25/75, 75/25)?

Designing Accessible Reading Assessments Sample 8 th Grade Students 26 Middle Schools 294 RLD (final sample=275) 194 LP (not include in this presentation) 500 Non-Disabled (final sample=486)

Designing Accessible Reading Assessments Description of Sample by NLD/RLD Race, Gender, and cut score impact Group SEXRACE % M% F% A.I.% A% B% H% M% P.I.% W NLD NLD RLD RLD

Designing Accessible Reading Assessments

Test Score Summaries: Route 1 N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component Fluency (obs max=222.75) Component Total: Scaled (max=48) RLD (Route 1) (3.96) (6.16) (32.37) (5.60) NLD (Route 1) (7.95) (6.12) (30.18) (6.27)

Designing Accessible Reading Assessments Test Score Summaries: Route 2 N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component Total:Scaled (max=48) RLD (Route 2) (6.01) (7.43) (8.49) NLD (Route 2) (8.05) (7.41) (8.47)

Designing Accessible Reading Assessments Primary Research Questions For accountability purposes, is it possible to combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? YES Is the Component test more accessible than the state assessment –Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments? YES, for Route 1

Designing Accessible Reading Assessments Can we reduce the number of students scoring at chance level?

Designing Accessible Reading Assessments Routing decision Best passage (2) Passage 1 Passages 1, 2 Passages 1, 2, 3 Full routing (8 items; Route 1: <= 3) (16 items; Route 1: <= 6) (24 items; Route 1: <= 10) (32 items; Route 1: <= 13) Reliability % (N) students assigned to Route % (112) 41.45% (114) 34.55% (95) 43.64% (120) 51.27% (141) % (N) students assigned to Route 1 on this test and full routing test 70.21% (99) 72.34% (102) 65.96% (93) 82.98% (117) 100% (141) Can we accurately route students based on 7, 14, 21, and 28 items?

Designing Accessible Reading Assessments Fluency Test Human vs. Automated Scoring ALL Passage 1 Passage 2 Passage 3 Passage 4 N Pearson r Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure?

Designing Accessible Reading Assessments Future Questions for Study and Policy Q: What is the best measure of oral reading fluency? Corrected words per minute in 1 st minute Words per minute, corrected words per minute, percent correct, rating Q: How do we combine comprehension and fluency scores 25% fluency + 75% comprehension 50/50, 75/25

Designing Accessible Reading Assessments Contact information Cara Cahalan Laitusis Senior Research Scientist Educational Testing Service Mailstop 09R Princeton, NJ

Designing Accessible Reading Assessments Extra Slides

Designing Accessible Reading Assessments Test Score Correlations: Route 1 Criterion Component Comprehension Component Fluency Component Total (Scaled) Criterion Component Comprehension Component Fluency Component Total (Scaled) NLD RLD

Designing Accessible Reading Assessments Test Score Correlations: Route 2 Criterion Component Total (Scaled) Criterion Component Total (Scaled) NLD RLD