Are Accommodations Used for ELL Students Valid? Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (CRESST) Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (CRESST)

Validity of accommodations Results of several studies have indicated that some accommodation strategies help both ELL and non-ELL studentsResults of several studies have indicated that some accommodation strategies help both ELL and non-ELL students These accommodations do more than just “level the playing field” because they provide unfair advantages to the recipientsThese accommodations do more than just “level the playing field” because they provide unfair advantages to the recipients For some accommodations there is enough research to judge the appropriateness and validity of the accommodations for ELL studentsFor some accommodations there is enough research to judge the appropriateness and validity of the accommodations for ELL students

Validity of Accommodations Research has shown that providing an English dictionary ( Abedi, Courtney, & Leon, 2003; Abedi, Lord, Kim, & Miyoshi, 2000) and extra time (Abedi, Lord, Hofstetter, & Baker, 2000; Hafner, 2001; Thurlow, 2001) affects performance of all students (see also, Maihoff, 2002; Thurlow & Liu, 2001)Research has shown that providing an English dictionary ( Abedi, Courtney, & Leon, 2003; Abedi, Lord, Kim, & Miyoshi, 2000) and extra time (Abedi, Lord, Hofstetter, & Baker, 2000; Hafner, 2001; Thurlow, 2001) affects performance of all students (see also, Maihoff, 2002; Thurlow & Liu, 2001) That is, the results of accommodated and non-accommodated assessment may not be aggregatedThat is, the results of accommodated and non-accommodated assessment may not be aggregated Research also suggests that translation of assessment tools into students’ native language may not produce desirable results if the language of instruction and assessment is not aligned (Abedi, Lord, Hofstetter, & Baker, 2000)Research also suggests that translation of assessment tools into students’ native language may not produce desirable results if the language of instruction and assessment is not aligned (Abedi, Lord, Hofstetter, & Baker, 2000) The purpose of accommodation for ELL students is to reduce the impact of language factors in order to provide a fair assessment of the content knowledge for these studentsThe purpose of accommodation for ELL students is to reduce the impact of language factors in order to provide a fair assessment of the content knowledge for these students

This particular study addresses the following four questions: 1. Do accommodation strategies help reduce the performance gap between ELL and non-ELL students by removing language barriers? (Effectiveness)1. Do accommodation strategies help reduce the performance gap between ELL and non-ELL students by removing language barriers? (Effectiveness) 2. Do accommodation strategies impact performance of non-ELL students on content-based assessment? (Validity)2. Do accommodation strategies impact performance of non-ELL students on content-based assessment? (Validity) 3. Do student background variables impact performance on the accommodated assessments? (Differential impact)3. Do student background variables impact performance on the accommodated assessments? (Differential impact) 4. Are the accommodations easy to apply? (Feasibility)4. Are the accommodations easy to apply? (Feasibility)

Methodology A total of 1,854 Grade 4 students and 1,594 Grade 8 students in 132 classes at 40 school sites participated in this studyA total of 1,854 Grade 4 students and 1,594 Grade 8 students in 132 classes at 40 school sites participated in this study The English proficiency designation of students was determined based on school recordsThe English proficiency designation of students was determined based on school records Out of 3,448 students, 1,712 (49.7%) students were identified as being ELLOut of 3,448 students, 1,712 (49.7%) students were identified as being ELL Three language categories were targeted for this study: Spanish, Chinese, and “other Asian languages”Three language categories were targeted for this study: Spanish, Chinese, and “other Asian languages”

Methodology (continues…) Of the 1,712 ELL students, 1,614 (94.3%) students had Spanish language backgroundOf the 1,712 ELL students, 1,614 (94.3%) students had Spanish language background A science test of multiple-choice and open-ended questions was administered in four forms: (1) original, (2) either a Customized English Dictionary, (3) a Bilingual/English Glossary, or (4) a Linguistic Modification of the test that addressed the challenge of understanding the English lexicon and, possibly, its syntaxA science test of multiple-choice and open-ended questions was administered in four forms: (1) original, (2) either a Customized English Dictionary, (3) a Bilingual/English Glossary, or (4) a Linguistic Modification of the test that addressed the challenge of understanding the English lexicon and, possibly, its syntax These three forms also allowed additional time (50%)These three forms also allowed additional time (50%) In addition to the science test, an English language reading battery was also administered to determine students’ reading proficiency levelsIn addition to the science test, an English language reading battery was also administered to determine students’ reading proficiency levels

Methodology (continues…) The linguistic complexity of the science items—but not the science content—was simplifiedThe linguistic complexity of the science items—but not the science content—was simplified The reading measure was important because students at different levels of reading proficiency may benefit differently from any accommodationThe reading measure was important because students at different levels of reading proficiency may benefit differently from any accommodation The study also included a student background questionnaire, an accommodation follow-up questionnaire, and teacher and school questionnairesThe study also included a student background questionnaire, an accommodation follow-up questionnaire, and teacher and school questionnaires The questionnaire included language background, country of origin, and length of time in the United States, and also asked students to self-assess their proficiency both in English and in their home languageThe questionnaire included language background, country of origin, and length of time in the United States, and also asked students to self-assess their proficiency both in English and in their home language

Methodology (continues…) The accommodation follow-up questionnaire asked students whether the dictionary/glossary helped them during the science test and how the language in the test could have been made easier to understandThe accommodation follow-up questionnaire asked students whether the dictionary/glossary helped them during the science test and how the language in the test could have been made easier to understand To control for teacher, school, and class effects, test materials and accommodations were distributed randomly among studentsTo control for teacher, school, and class effects, test materials and accommodations were distributed randomly among students

Null Hypotheses H01: In the science assessment, ELL students do not benefit from any of the accommodations used in this study. (Effectiveness)H01: In the science assessment, ELL students do not benefit from any of the accommodations used in this study. (Effectiveness) H02: Accommodation does not impact performance of non-ELL students on the science test. (Validity)H02: Accommodation does not impact performance of non-ELL students on the science test. (Validity) H03: Student background variables do not impact performance on accommodated science assessment. (Differential impact)H03: Student background variables do not impact performance on accommodated science assessment. (Differential impact)

Methodology (continues…) We created a latent composite of all the three components that measured English reading efficiency/proficiency: (1) Fluency section of the Language Assessment Scales (LAS), (2) one intact block of the 1994 NAEP Reading assessment (released items), and (3) an experimental word recognition section.We created a latent composite of all the three components that measured English reading efficiency/proficiency: (1) Fluency section of the Language Assessment Scales (LAS), (2) one intact block of the 1994 NAEP Reading assessment (released items), and (3) an experimental word recognition section. We used this latent composite as a covariateWe used this latent composite as a covariate Each open-ended science and reading item was scored independently by two ratersEach open-ended science and reading item was scored independently by two raters Interrater reliability indices (percent of exact and within one- point agreement, P.M. correlation, intraclass correlation, and kappa, and alpha coefficients) were computedInterrater reliability indices (percent of exact and within one- point agreement, P.M. correlation, intraclass correlation, and kappa, and alpha coefficients) were computed For the open-ended science items in Grade 4, percent of agreement ranged from a low of 69% to a high of 97%. The kappa coefficient ranged from a low of.42 to a high of.94For the open-ended science items in Grade 4, percent of agreement ranged from a low of 69% to a high of 97%. The kappa coefficient ranged from a low of.42 to a high of.94

Findings The internal consistency coefficient for the entire set of Grade 8 reading items was.78; for Grade 4, the overall internal consistency coefficient for reading was.82The internal consistency coefficient for the entire set of Grade 8 reading items was.78; for Grade 4, the overall internal consistency coefficient for reading was.82 Students with Spanish as a primary home language performed slightly higher on the Linguistic Modification accommodation than under the Standard condition.Students with Spanish as a primary home language performed slightly higher on the Linguistic Modification accommodation than under the Standard condition. Students whose primary home language is neither English nor Spanish benefited the most from the Linguistic Modification version of the testStudents whose primary home language is neither English nor Spanish benefited the most from the Linguistic Modification version of the test

Item-level analysis The higher the level of the linguistic complexity, the larger the performance difference between ELL and non-ELL studentsThe higher the level of the linguistic complexity, the larger the performance difference between ELL and non-ELL students The larger the performance difference between ELL and non-ELL students, the more linguistic modification of test items helped reduce the performance gapThe larger the performance difference between ELL and non-ELL students, the more linguistic modification of test items helped reduce the performance gap Out of 30 science items, students performed better on 22 items under the Linguistic Modification conditionOut of 30 science items, students performed better on 22 items under the Linguistic Modification condition

Findings (continue…) Neither ELL nor non-ELL Grade 4 students benefited from any of the three accommodation strategies that were usedNeither ELL nor non-ELL Grade 4 students benefited from any of the three accommodation strategies that were used We believe this may be because language demand in textbooks and tests may be less in lower than in higher gradesWe believe this may be because language demand in textbooks and tests may be less in lower than in higher grades The lack of significant impact on Grade 4 non-ELL students is an encouraging result because it suggests that the accommodation did not alter the construct under measurementThe lack of significant impact on Grade 4 non-ELL students is an encouraging result because it suggests that the accommodation did not alter the construct under measurement

Results (continues…) The Linguistic Modification accommodation was shown to have a significant impact on the ELL students’ performance in grade 8The Linguistic Modification accommodation was shown to have a significant impact on the ELL students’ performance in grade 8 Accommodation helped ELL students to increase their performance while the accommodated performance of non-ELL students was unchangedAccommodation helped ELL students to increase their performance while the accommodated performance of non-ELL students was unchanged

Summary & Conclusion A non-significant impact of the linguistically modified test on the non-ELL group assures the validity of this accommodationA non-significant impact of the linguistically modified test on the non-ELL group assures the validity of this accommodation Increase performance of ELL students (grade 8) under the linguistically modified test shows effectiveness of this accommodationIncrease performance of ELL students (grade 8) under the linguistically modified test shows effectiveness of this accommodation As for feasibility, this accommodation requires up-front preparation, but is easy to implement in the field; therefore, it is feasible for large-scale assessmentsAs for feasibility, this accommodation requires up-front preparation, but is easy to implement in the field; therefore, it is feasible for large-scale assessments

For more information contact: Jamal Abedi: (530)