Jamal Abedi, UCLA/CRESST Major psychometric issues Research design issues How to address these issues Universal Design for Assessment: Theoretical Foundation and Practical Implications Paper presented at 34 th Annual Conference on Large-Scale Assessment Boston, June 20-23, 2004

Major Psychometric Issues Unique to EL and SD Assessment  Reliability: Language factors (unnecessary linguistic complexity of test items) may be source of measurement error, affecting the reliability of EL and SD assessment  Validity: Language factors may be a source of construct irrelevant variance, threatening the construct validity of EL and SD assessment

Reliability Issues Classical Test Theory  2 X =  2 T +  2 E  XX’=  2 T /  2 X  XX’= 1-  2 E /  2 X Textbook examples of possible sources that contribute to the measurement error: X: Observed Score T: True Score Error Score E: Error Score  Rater  Occasion  Item  Test Form

Assumptions of Classical True- Score Test Theory 1. X = T + E (Total observed score is the sum of true score plus error score) 3.  ET = 0 (Correlation between error and true scores is zero) 2.  (X) = T (Expected value of observed score is true score) 5.  E1T2 = 0 (Correlation between a true score and an error score is zero) 4.  E1E2 = 0 (Correlation between two error scores is zero)

Performance Difference Between SD and Non-SD in Reading

Performance Difference Between SD and Non-SD in Content-Based Areas

A sample Representing Subgroup of Students with Disabilities 1

A sample Representing Subgroup of Students with Disabilities 2

ReadingMathMath CalculationMath Analytical Non-LEP/Non- SWD Mean SD N LEP only Mean SD N SWD only Mean SD N LEP/SWD Mean SD N Site 4 Grade 8 Descriptive Statistics for the SAT 9 Test Scores by Strands

Validity Issues  Linguistically complex, content-based test items, may confound knowledge of language and of content  Confounding complicates interpretation of assessment results for EL students  Construct validity of assessments with unnecessarily complex language is questionable

How to Improve Assessment Quality for EL Students  Reduce unnecessary linguistic complexity, based on content experts’ judgment  Examine the impact of linguistic modification on EL students under experimentally controlled conditions with the following characteristics: Manipulate the level of linguistic complexity of assessments and see the effect of manipulation (IV) on the outcome variable (DV) Randomly assign students (intact classrooms) to linguistically complex/non-complex groups Examine the effect of linguistic modification (cause & effect Control relationship) Control for students’ reading ability by using reading score as a covariate

Design Considerations Threats to internal validity:  Selection  Maturation  Instrumentation  Statistical regression  Testing  History  Diffusion of treatment  Experimenter bias

Design Considerations, continues… Threats to external validity: Random selection of subjects (LEP, SD, non-LEP/non-SD) Intact classrooms versus randomly assigning subjects Willingness to participate Legal issues in random sampling

Design Considerations, continues… Hierarchical versus completely randomized factorial design How to control for nuisance variables Power & Sample size Variability at different levels of HLM

CRESST Studies to Examine the Effectiveness of Accommodations Study #1. (Abedi, Lord, & Plummer, 1997, n=1,031). EL students with linguistically modified version performed significantly better than others. Study #2. (Abedi, Lord, & Hofstetter, 1997, n=1,394). Linguistic modification contributed to improved performance on 49% of the items. Study #3. (Abedi, Hofstter, & Lord, 1998, n=946). Linguistically modified version helped EL students. Study #4. (Abedi, Lord, Kim, & Miyoshi, 2000, n=422). Linguistic modification as a form of accommodation was helpful. Study #5. (Abedi, Courtney, Leon, Mirocha, & Goldberg, 2001, n=612). Linguistic modification was found to be helpful.

Studies, continued Study #6. (Abedi, Courtney, & Leon, 2001, n=1,856 G 4 and 1,512 G 8). Linguistically modified version improved performance for Grade 8 EL students. Study #7. (Abedi & Leon, 1999, n=900,000) students from four sites in different parts of the nation. Study #8. (Abedi, Leon, & Mirocha, 2001, n=700,000). The higher the level of language demand, the higher the performance gap between EL and non-EL students. Study #9. (Abedi, Courtney, & Leon, 2002, n=607 Grade 4 and 542 Grade 8 students). Computer testing and customized dictionary were shown to be effective forms of accommodation. Study #10. (Abedi, Courtney, & Leon, 2002, n=607 Grade 4 and 542 Grade 8 students). Study #11. (Abedi, Courtney, & Leon, 2002, n=700 G 8). Student self-reported Opportunity-to-Learn correlated with actual performance in math.

Conclusion and Recommendations  Existing and newly developed content-based achievement tests (NRT and CRT) may have complex linguistic structures and may not be relevant for EL and other subgroups of students.  To increase authenticity of assessments for all, irrelevant constructs must be identified and removed.  Assessment accommodations can help EL students to show what they know without disadvantaging other students. Such accommodations must be considered to provide more valid assessment of EL students.

Contribution of Our Research Findings to Educational Assessments  Test publishers incorporated our suggested approaches for reducing the level of linguistic complexity into their item-development process (ETS, AIR, CTB/McGraw Hill, etc)  States are increasingly asking us to assist them in the language modification of their instruments  NAEP is considering language modification as a form of accommodation in its future assessments  Our published articles and reports attract the attention of policy makers and practitioners across the nation and the world