Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Using DIF to Examine the Validity and Fairness of Assessments for Students With Disabilities Cara Cahalan-Laitusis & Linda Cook Educational Testing Service

Copyright © 2004 Educational Testing Service Page 2 Examining Assessments for Students with Disabilities DARA Research Using Archival Data –State standards based assessment –English-language Arts Purpose of Studies –Determine how a students score is effected by disability or accommodation –To improve assessments for all students

Copyright © 2004 Educational Testing Service Page 3 Overview of Presentation What is DIF and why are DIF procedures useful Some issues related to using DIF procedures for students with disabilities How has DIF been applied to evaluate assessments for students with disabilities A summary of a DARA DIF study

Copyright © 2004 Educational Testing Service Page 4 Why DIF Procedures are Useful DIF studies first carried out on a frequent basis in 1960s –Evaluate ethnic differences –Identify and remove biased items DIF exists if examinees at same ability level perform differently on the item Examples –Deaf and HH examinees –Test item draws on knowledge of popular music

Copyright © 2004 Educational Testing Service Page 5 Some Issues Related to Using DIF Procedures For Students With Disabilities There are some differences between DIF for well defined subgroups and DIF for students with disabilities –Definition of groups –Matching criterion –Ability differences between groups

Copyright © 2004 Educational Testing Service Page 6 Reviews of Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities Tindal & Fuchs, 2000; Sireci, Li & Scarpati, 2003; Thompson, Blount & Thurlow, 2002; Pitoniak & Royer, 2001 Very little research focusing on DIF

Copyright © 2004 Educational Testing Service Page 7 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities Lewis, Green & Miller, 1999 –Read aloud and extended time –35 different groups with disabilities –More ELA items had DIF than Math items –DIF balanced out

Copyright © 2004 Educational Testing Service Page 8 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Bielinski, Thurlow, Ysseldyke, Freidebach & Freidebach, 2001 –Read aloud accommodations –Reading and math tests for grades 3 and 4 –41 multiple choice grade 3 reading items –32 multiple choice grade 4 math items –DIF for 30 reading and 7 math items –Results indicated read aloud accommodation may affect comparability of items

Copyright © 2004 Educational Testing Service Page 9 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Bolt,2004 –Read aloud accommodation is less appropriate for reading tests than for non-reading tests –Accommodations are more appropriate for students with sensory and physical disabilities than for students with cognitive disabilities –Accommodations fall on a continuum in terms of the degree to which they affect the validity of a test

Copyright © 2004 Educational Testing Service Page 10 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Barton& Finch, 2004 –Second edition of the TerraNova –Language and math subtests for grades 3,5,8 DIF for items with high amount of text provided via read aloud conditions to accommodated students –Concluded some level of boost for students with disabilities with read aloud accommodation

Copyright © 2004 Educational Testing Service Page 11 DARA DIF Analysis Purpose is to provide information that will lead to the development of more accessible tests for students with disabilities Series of studies –English language arts –Grades 3,4,7, and 8 –Learning disabled student –Accommodations Extended time, read aloud

Copyright © 2004 Educational Testing Service Page 12 Purpose 1.Identify characteristics of test questions that result in Differential Item Functioning (DIF) for students with Learning Disabilities. 2.Make recommendations to test developers regarding the improvement of test questions for students with Learning Disabilities.

Copyright © 2004 Educational Testing Service Page 13 Reference-Focal Comparisons Reference Group No Disability Focal Group LD No Accommodation LD Extra Time LD Read Aloud and Extra Time

Copyright © 2004 Educational Testing Service Page 14 Content of Tests Examined English Language Arts Grades 3 71 multiple-choice items 86% reading 14% writing strategies English Language Arts Grade 7 81 multiple-choice items 77% reading 23% writing strategies

Copyright © 2004 Educational Testing Service Page 15 Proficiency Levels for ELA, Grade 3 SubgroupBelow Basic Basic and Above Total No LD33.5%66.5%437,398 LD Time86.8%13.3%2,069 LD Read & Time 79.6%20.4%2,003

Copyright © 2004 Educational Testing Service Page 16 Proficiency Levels for ELA, Grade 7 SubgroupBelow Basic Basic and Above Total No LD26.4%73.6%437,689 LD Time82.3%17.6%1,787 LD Read85.1%14.9%2,154

Copyright © 2004 Educational Testing Service Page 17 DIF Categories ELA Grade 3 Extra Time Easy Difficult Favors SWOD Favors LD

Copyright © 2004 Educational Testing Service Page 18 DIF Categories ELA Grade 3 Read Aloud and Extra Time Easy Difficult Favors SWOD Favors LD

Copyright © 2004 Educational Testing Service Page 19 DIF Categories ELA Grade 7 Extra Time Easy Difficult Favors SWOD Favors LD

Copyright © 2004 Educational Testing Service Page 20 DIF Categories ELA Grade 7 Read Aloud and Extra Time Easy Difficult Favors SWOD Favors LD

Copyright © 2004 Educational Testing Service Page 21 Conclusions DIF used in an exploratory manner may provide useful insights that will help us understand how to develop good assessments for students with disabilities.

Copyright © 2004 Educational Testing Service Page 22 Areas for Further Research Choice of Criteria Choice of Method DARA is Currently Carrying Out Research to Address Both Questions

Copyright © 2004 Educational Testing Service Page 23 Next Steps Replicate the findings in this study using data from other ELA tests Follow up the study with other types of research –Item manipulation –Protocol analysis –Factor analysis