1. Copyright © 2006 Educational Testing Service Listening. Learning. Leading. Using Differential Item Functioning to Investigate the Impact of Accommodations on the Scores of Students with Disabilities on English-Language Arts Assessments Mary Pitoniak, Linda Cook, Frederic Cline, and Cara Cahalan-Laitusis Educational Testing Service NCME Presentation April 10, 2006

2. Copyright © 2006 Educational Testing Service 2 P 2 Purpose and Overview of the Study The purpose of this study was to examine differential item functioning on the English- Language Arts assessment described by Linda DIF analyses are statistical procedures that are used to identify items that function differently for different subgroups of examinees DIF exists when examinees of equal ability differ, on average, according to their group membership in their responses to a particular item (Standards)

3. Copyright © 2006 Educational Testing Service 3 P 3 Purpose and Overview of the Study (continued) Issues investigated: – Do 2 different DIF detection methods yield the same results? – Are the results interpretable in terms of a priori or a posteriori evaluation of item content? – Of particular interest: When the read-aloud modification is used, do the items function differentially for students?

4. Copyright © 2006 Educational Testing Service 4 P 4 Purpose and Overview of the Study (continued) Features of study: – 2 DIF detection methods – Large enough sample sizes (not always the case) – Looked at 3 different criteria (total score, Reading score, Writing score); we decided to go with total score for several reasons – Used purification step, as recommended by literature

5. Copyright © 2006 Educational Testing Service 5 P 5 Comparisons Made in the Study

6. Copyright © 2006 Educational Testing Service 6 P 6 DIF Methods Used Mantel-Haenszel Logistic Regression For both methods, we used ETS classification system: – Category A contains items with negligible DIF; – Category B contains items with slight to moderate values of DIF; – Category C contains items with moderate to large values of DIF.

7. Comparison of Mantel-Haenszel vs. Logistic Regression

8. Example of Uniform DIF

9. Example of Non-Uniform DIF

10. Copyright © 2006 Educational Testing Service 10 P 10 Results Within this presentation, I will present results only for Reading items (and not Writing), both for time reasons and because we were most interested in the effects of the accommodations on performance on the Reading items

11. Copyright © 2006 Educational Testing Service 11 P 11 Results (continued) Overall – No items flagged as C – Each method flagged 9 items as B (out of 42 items X 5 comparisons, or 210 possible flags) – However, those 9 items were not the same itemsin all, 12 different items were flagged by at least one of the methods – There were inconsistencies between methods

12. Copyright © 2006 Educational Testing Service 12 P 12 Number of Items Flagged by Each Method

13. Agreement Between Flags for Methods by Comparison Type

14. Non-LD vs. LD No Accommodation

15. Non-LD vs. LD IEP/504 Accommodation

16. Non-LD vs. LD Read-Aloud Modification

17. LD Non-Accommodated vs. LD IEP/504 Accommodation

18. LD Non-Accommodated vs. LD Read-Aloud Modification

19. Example of Discrepancies in Flags Item Flags M-HUniform LRNo flag The items flagged by MH (but not LR) as favoring students with read-aloud modification did show differences such as these graphically for LR

20. Example of Discrepancies in Flags Item Flags M-HUniform LRNo flag

21. Copyright © 2006 Educational Testing Service 21 P 21 A Priori Theories About Read-Aloud Modification Results 5 items were easier for students who received the read-aloud modification than for non-LD students. A priori theories were not that accurate! – Item A: harder (refer back) – Item B: easier (short item; intonation/body language) – Item C: easier (intonation/body language) – Item D: harder (char. of options) – Item E: harder (length of options)

22. Copyright © 2006 Educational Testing Service 22 P 22 A Posteriori Interpretation About Read-Aloud Modification Results The reasons why these 5 items were easier with read-aloud accommodation were not obvious to test developers

23. Copyright © 2006 Educational Testing Service 23 P 23 What Do the Results Say About the 3 Questions Posed Do 2 different DIF detection methods yield the same results? – Neither flagged an item as C. – There were discrepancies in B flags, however. – Some discrepancies are explainable in terms of advantages/disadvantages of methods as listed earlier.

24. Copyright © 2006 Educational Testing Service 24 P 24 Are the results interpretable in terms of a priori or a posteriori evaluation of item content? – Not consistently Of particular interest: When the read-aloud modification is used, do the items function differentially for students? – Yes, some items were easier when read-aloud, which supports this states decision to view read-aloud as a modification 3 Questions (continued)

25. Copyright © 2006 Educational Testing Service 25 P 25 ELL and ELL/LD groups to be compared Grade 8 ELA to be evaluated DIF analysis paradigm to be utilized Next Steps