1. National Accessible Reading Assessment Projects DARA Research and Next Steps Cara Cahalan-Laitusis & Linda Cook Educational Testing Service

2. National Accessible Reading Assessment Projects Presentation Experimental Study of Read Aloud Psychometric Research Research Plans for Year 3 –Psychometric analysis of experimental data –Tailored Test Design –Cognitive labs –IEP Decision Making for read aloud

3. National Accessible Reading Assessment Projects Differential Boost from Read Aloud (Non-disabled vs. RLD) 1.Is there a Differential Boost from read aloud? 2.How well do test scores (standard, audio, and fluency) predict variance in teacher ratings of reading comprehension? 3.Are teachers able to predict which students will benefit from read aloud?

4. National Accessible Reading Assessment Projects Prior Research No Differential Boost –Kosciokek & Ysseldyke (2000)- Small sample size (n=31) –Meloy, Deville, and Frisbie (2002) – Between subjects design (n=260, 76% non-disabled, randomly assigned to audio or standard) –McKevitt & Elliott (2003)-Small sample size (n=39) Differential Boost –Crawford and Tindal (2004)-(n=338, 78% non-disabled) –Fletcher, et. al (2006)-Between subjects design (randomly assigned to audio or standard). Sample included 91 Dyslexic (poor decoder) and 91 average decoders

5. National Accessible Reading Assessment Projects Data Collected GMRT Forms S and T –Extra Time –Extra Time with Read Aloud via CD 2 Fluency Measures –WJ Reading Fluency –Test of Silent Word Reading Fluency 2 Decoding Measures (4 th grade only) –WJ Letter Word ID –WJ Word Recognition Demographic and Survey Data

6. National Accessible Reading Assessment Projects Sample 1170 4 th Graders –522 Students with RLD –648 Students without a disability 855 8 th Graders –394 Students with RLD –461 Students without a disability

7. National Accessible Reading Assessment Projects Design Group Session 1Session 2 FormAccommodationFormAccommodation 1SStandardTAudio 2S TStandard 3T SAudio 4T SStandard

8. National Accessible Reading Assessment Projects Means for Grade 4 Non-LDRLD Test/ConditionNMeanSDNMeanSD WJ Letter Word ID6045042146947329 WJ Word Attack6045041546948420 TOSWRF604102104698912 WJ Fluency6045012446947421 Audio6045023246947730 Standard6044973746945731 Boost6045244691927

9. National Accessible Reading Assessment Projects Means for Grade 8 Non-LDRLD Test/ConditionNMeanSDNMeanSD TOSWRF463103133739012 WJ Fluency4635604237351434 Audio4635553137352127 Standard463553337351128 Boost4632213731023

10. National Accessible Reading Assessment Projects Scores by RLD and Grade

11. National Accessible Reading Assessment Projects 1.Is there a Differential Boost from read aloud? Repeated Measures ANOVA and ANCOVA Dependent Variables: –GMRT Standard –GMRT Audio Independent Variables: –Disability Status (RLD vs. NLD) –Form/Order (STSA, STAS, TSSA, TSAS) Covariate: Decoding and Fluency Measures

12. National Accessible Reading Assessment Projects ANOVA Findings Yes, students with reading-based learning disabilities have larger gains (on average) from read aloud than students without disabilities –Finding consistent at both grades 4 and 8, but boost is larger at grade 4 –Controlling for Decoding and/or Fluency as a covariate did not alter findings

13. National Accessible Reading Assessment Projects Multiple regression analyses to determine how much variance in teachers rating of reading comprehension (5-point scale) were predicted by three test scores: –Standard –Audio –Fluency 2.How well do test scores predict reading comprehension?

14. National Accessible Reading Assessment Projects Regression Findings Audio score does not significantly predict variance in Teacher Ratings of Reading Comprehension (beyond standard and fluency) for Grade 8 RLD Audio score adds to prediction of reading comprehension (beyond standard and fluency scores) for three groups (NLD grade 4, NLD grade 8, and RLD grade 4), but incremental change is small

15. National Accessible Reading Assessment Projects Analyses: Analysis of variance in boost by teacher predictions Cross-tabulations of teacher ratings by degree of boost (more than on SEM, less than one SEM, neither) 3.Are teachers able to predict which students will benefit from read aloud?

16. National Accessible Reading Assessment Projects Accuracy of Teacher Prediction For this study each student took a reading comprehension test that was read aloud by a CD player and another reading comprehension test that they read to themselves. Which test do you predict the student did better on? Test read aloud by CD player Test the student read to themselves No difference

17. National Accessible Reading Assessment Projects Findings from Teacher Predictions ANOVA indicated that on average teachers were able to predict score gain from audio at grade 4 but not grade 8 At the individual level teachers accurately predicted if a student would benefit from the audio version about 35% of the time and were completely wrong about 5% of the time

18. National Accessible Reading Assessment Projects DARA Psychometric Research Purpose of psychometric research: To help us understand how an examinee's disability or the accommodations he or she receives impacts the psychometric properties of a reading test

19. National Accessible Reading Assessment Projects Results of This Years Psychometric Analyses Psychometric Analyses –Factor analyses –Differential item functioning analyses Populations –Students with learning disabilities who took the test with and without accommodations Test –Grade 4 and grade 8 English-language arts (ELA) assessment Focus –Determine if the test measures the same constructs for Examinees without disabilities Examinees with disabilities who took the test with and without accommodations

20. National Accessible Reading Assessment Projects

22. STAR ELA Grade 4 and Grade 8 Summary Statistics Group Grade 4 Total GroupsGrade 8 Total Groups NMeanSDNMeanSD (1) Students without disabilities 298,6224814357,3744612 (2) LD, without accommodations 9,045291218,5122910 (3) LD, 504/IEP accommodations 4,72427104,325279 (4) LD, read- aloud accommodation 1,3672911874279

23. National Accessible Reading Assessment Projects Factor Analyses of ELA Assessment Exploratory analyses (separately in each group) – how many factors Confirmatory (multi-group) –Establish base-line model –Confirm number of factors needed to describe data across all groups

24. National Accessible Reading Assessment Projects Differential Item Functioning (DIF) Analyses The purpose of this study was to examine differential item functioning on the same English- Language Arts assessment that was used for the factor analyses DIF is a statistical observation that involves matching test takers from different groups on the characteristic measured [by the test] and then looking at performance differences on an item. (Sireci, 2006)

25. National Accessible Reading Assessment Projects Method Mantel-Haenszel Categorization3 Levels –A Negligible DIF –B Slight to Moderate DIF –C Moderate to Large DIF Directions of DIF Flags - Favors reference group + Favors focal group

26. National Accessible Reading Assessment Projects Comparisons Made in the Study

27. National Accessible Reading Assessment Projects DIF Categories ELA Grade 4 LD Without Accommodations Easy Difficult Favors Students Without Disabilities Favors LD Students

28. National Accessible Reading Assessment Projects DIF Categories ELA Grade 4 LD With Accommodations (IEP/504) Easy Difficult Favors Students Without Disabilities Favors LD (IEP/504)

29. National Accessible Reading Assessment Projects DIF Categories ELA Grade 4 LD With Accommodations (Read-Aloud) Easy Difficult Favors Students Without Disabilities Favors LD (Read-Aloud)

30. National Accessible Reading Assessment Projects DIF Categories ELA Grade 8 LD Without Accommodations Easy Difficult Favors Students Without Disabilities Favors LD Students

31. National Accessible Reading Assessment Projects DIF Categories ELA Grade 8 LD With Accommodations (IEP/504) Easy Difficult Favors Students Without Disabilities Favors LD (IEP/504)

32. National Accessible Reading Assessment Projects DIF Categories ELA Grade 8 LD With Accommodations (Read-Aloud) Easy Difficult Favors Students Without Disabilities Favors LD (Read-Aloud)

33. National Accessible Reading Assessment Projects Interpreting the Results of the DIF Study Grade 4 –1 C DIF item, 8 B DIF items Grade 8 –1 C DIF item, 6 B DIF items Majority of flagged items were reading items that favored students who took test with read-aloud accommodation Consistent with Factor Analysis Results

34. National Accessible Reading Assessment Projects Next Steps Psychometric Research Examination of Tailored Testing Cognitive Labs IEP decision making

35. National Accessible Reading Assessment Projects Psychometric Research

36. National Accessible Reading Assessment Projects Plans for Next Years Psychometric Analyses Psychometric analyses –Factor Analyses Differential item functioning analyses Populations –Students with learning disabilities who took the test with and without an audio accommodation Test –Gates-McGinitie Reading Test Focus –Aid in interpretation of results of differential boost study –Increase understanding of impact of disability and audio accommodation on reading test scores

37. National Accessible Reading Assessment Projects Factor Analyses We Plan to Carry Out Aid in interpretation of results of differential boost study –Compare factor structures for students without disabilities who took test with and without accommodation –Compare factor structures for students with disabilities who took test with and without accommodation

38. National Accessible Reading Assessment Projects Factor Analyses We Plan to Carry Out Increase understanding of impact of disability and accommodation on reading test scores –Compare factor structures of test given to examinees with and without disabilities under standard conditions –Compare factor structure of test given to examinees with disabilities who take test with accommodations and examinees without disabilities who take test without accommodations

39. National Accessible Reading Assessment Projects Purpose of Doing DIF and DDF Analyses on Data From the Differential Boost Study Aid in interpretation of results of differential boost study Increase understanding of impact of disability and accommodation on reading test scores

40. National Accessible Reading Assessment Projects Possible Comparisons for DIF Analyses

41. National Accessible Reading Assessment Projects Procedures for Analyzing Data Differential Item Functioning: Mantel- Haenszel Differential Distractor Analysis: Standardized Distractor Analysis

42. National Accessible Reading Assessment Projects Two Staged Tailored Testing

43. National Accessible Reading Assessment Projects Operational Data GradeDisabilityFormat Percent Below Chance Test TakersItems 4LD Audio17.421.3 Other23.330.7 Standard21.221.3 None Standard2.21.3 8LD Audio20.032.0 Other19.732.0 Standard16.624.0 None Standard1.50.0

44. National Accessible Reading Assessment Projects GMRT Data Percent Below Chance GradeDisabilityFormat Test Takers Items 4RLD Audio4.45.2 Standard20.523.0 None Audio1.51.1 Standard2.62.1 8RLD Audio4.811.5 Standard12.215.7 None Audio0.21.1 Standard1.1

45. National Accessible Reading Assessment Projects DARA Tailored Testing Model Two (or three) stages of testing Students subtests on stage 2 are determined by performance on routing test administered in stage 1 Ideally computer administered but can be paper administered Some parts could be individually administered (e.g., decoding) if only a few students are routed into a decoding measure and this format reduces the number of students receiving individualize testing accommodations (e.g., read aloud by human)

46. National Accessible Reading Assessment Projects Reading Comprehension Routing Test Reading Fluency Decoding and Extended Comprehension Test with Audio Extended Comprehension Test with Audio Extended Reading Comprehension Test

47. National Accessible Reading Assessment Projects Advantages of Model Score is more reliable estimate since items are targeted to students ability level Students may feel less frustrated if they can do some of the items on the routing test Teacher receives more information on low performing students strengths and weaknesses Fundamental Skills and Comprehension are not confounded for students with poor fundamental skills (some LD) or poor comprehension (some LD and ELL) Growth can be more accurately measured in students working significantly below (or above) grade level

48. National Accessible Reading Assessment Projects Disadvantages of Model Requires computer administration or teacher scoring of items after stage 1 Students who are routed to fluency test may be embarrassed Routing decision is made before test is scaled or standard setting is completed Design could route more that 2% of students to modified test

49. National Accessible Reading Assessment Projects Questions for Year 3 How many items (and of what difficulty) are needed for an accurate routing test? Can we equate the audio extended and standard extended using the routing test? What portion of students would be routed to fluency measure and what portion would be routed to decoding?

50. National Accessible Reading Assessment Projects Questions for Year 3 (continued) Are the 2 alternate routes highly correlated with the standard administration? What is the impact audio, fluency, and decoding scores on total test score. –If student is not a fluent reader should the total test score be non-proficient? Is the routing test accurate for all students? –Do some students do better on hard items? –Do some students having trouble with the first few items on the test?

51. National Accessible Reading Assessment Projects Questions for Year 3 (continued) How should we weight different measures and what impact will this have on subpopulations? Could we compose a tailored test from a states current operational item pool? –If not how many additional items would be required and at what difficulty level?

52. National Accessible Reading Assessment Projects Cognitive Labs

53. National Accessible Reading Assessment Projects Background Cognitive labs using the think aloud method on reading comprehension questions Build off the findings of last years large scale differential boost study –Gates MacGinitie Reading Comprehension Test –Use items found in preliminary findings of the DIF analysis of the GMRT data

54. National Accessible Reading Assessment Projects Cognitive Labs Advantages Beneficial to learn about components of mental processes of reading (Afflerbach & Johnston, 1984, Alavi, 2005; Pressley & Afflerbach, 1995) Beneficial in the development of assessments (Caspar, Lessler, & Willis, 1999; Desimone & LeFloch, 2004; Willis, 2005) Open flexible procedure can be catered to the specific situation and activity (Davison, Vogel & Coffman, 1997) May use a small sample size Procedure has been successfully conducted with children as young as 3 rd grade (Laing & Kamhi, 2002; Paulsen & Levine, 1999; Trambasso & Magliano, 1996)

55. National Accessible Reading Assessment Projects Cognitive Labs Disadvantages Thinking aloud is an unnatural step which may affect or interfere ones normal mental processes Students with disabilities may have difficulty with the procedure (Johnstone, Miller, & Thompson, in press) Responses have the potential to be incomplete or incorrect –Lack of desire/motivation –Embarrassment –Inability to understand the task

56. National Accessible Reading Assessment Projects Purpose of Study This study is being conducted to serve the following purposes: 1.How do students with and without reading-based learning disabilities differ as they approach a reading comprehension assessment? 2.Is this type of information gathering and data quality worthwhile to conduct in future large scale studies considering: –Age of students –Students with disabilities

57. National Accessible Reading Assessment Projects Research Questions 1.In what way do students with reading- based disabilities respond differently to reading comprehension questions compared to students without disabilities? 2.What errors occur while reading the passage/reading the items?

58. National Accessible Reading Assessment Projects IEP Decision Making

59. National Accessible Reading Assessment Projects IEP Decision Making What factors contribute to boost? –Low standard score –WJ Reading Measures –Teacher Predictions –Student Preference

60. National Accessible Reading Assessment Projects Analyses planned Regression analyses to predict boost for RLD students using –WJ scores –Standard score –Use of read aloud in class or on tests –Teacher predictions –NJ ASK from prior year