CRESST ONR/NETC Meetings, July 2003, v1 1 ONR Advanced Distributed Learning Language Factors in the Assessment of English Language Learners Jamal Abedi University of California, Los Angeles National Center for Research on Evaluation, Standards, and Student Testing (CRESST) July 18, 2003

CRESST ONR/NETC Meetings, July 2003, v1 2 The “No Child Left Behind Act” mandates inclusion of ALL students Goals 2000 Title I and VII of the Improving America’s School Act of 1994 (IASA) However, language factors create a major obstacle in including English language learners (ELLS) Because of possible English language deficiencies, ELL students have been traditionally excluded from large-scale National and State assessments.

CRESST ONR/NETC Meetings, July 2003, v1 3 CRESST Studies on the Impact of Language Factors on the Assessment of ELL Students:

CRESST ONR/NETC Meetings, July 2003, v1 4 Study #1 Analyses of extant data (Abedi, Lord, & Plummer, 1995). Used existing data from NAEP 1992 assessments in math and science. SAMPLE: ELL and non-ELLs in grades 4, 8, and 12 main assessment. NAEP test items were grouped into long and short and linguistically complex/less complex items. Findings l ELL students performed significantly lower on the longer test items. l ELL students had higher proportions of omitted and/or not-reached items. l ELL students had higher scores on the linguistically less-complex items.

CRESST ONR/NETC Meetings, July 2003, v1 5 Study #2 Interview study (Abedi, Lord, & Plummer, 1997) 37 students asked to express their preference between the original NAEP items and the linguistically modified version of these same items. Math test items were modified to reduce the level of linguistic complexity. Findings l Over 80% interviewed preferred the linguistically modified items over the original version.

CRESST ONR/NETC Meetings, July 2003, v1 6 The revised items need less time to respond:  “It’s easier to read, and it gets to the point, so you won’t have to waste time.”  “I might have a faster time completing that one cause there’s less reading.”  “Less reading; then I might be able to get to the other one in time to finish both of them.”  “Cause it’s, like, a little bit less writing.”

CRESST ONR/NETC Meetings, July 2003, v1 7 The vocabulary in the revised items was more familiar:  “This one uses words like ‘approximation,’ and this one uses words that I can relate to.”  “It doesn’t sound as technical.”  “I can’t read that word.”  “Because it’s shorter and doesn’t have, like, complicated words.”

CRESST ONR/NETC Meetings, July 2003, v1 8 Study #3 Impact of linguistic factors on students’ performance (Abedi, Lord, & Plummer, 1997). Two studies: testing performance and speed. SAMPLE: 1,031 grade 8 ELL and non-ELL students. 41 classes from 21 southern California schools. Findings l ELL students who received a linguistically modified version of the math test items performed significantly better than those receiving the original test items.

CRESST ONR/NETC Meetings, July 2003, v1 9 Study #4 The impact of different types of accommodations on students with limited English proficiency (Abedi, Lord, & Hofstetter, 1997) SAMPLE: 1,394 grade 8 students. 56 classes from 27 California schools. Findings Spanish translation of NAEP math test l Spanish-speakers taking the Spanish translation version performed significantly lower than Spanish-speakers taking the English version. l We believe that this is due to the impact of language of instruction on assessment. Linguistic Modification l Contributed to improved performance on 49% of the items. Extra Time l Helped grade 8 ELL students on NAEP math tests. l Also aided non-ELL students. Limited potential as an assessment accommodation.

CRESST ONR/NETC Meetings, July 2003, v1 10 Study #5 Impact of selected background variables on students’ NAEP math performance (Abedi, Hofstetter, & Lord, 1998). SAMPLE: 946 grade 8 ELL and non-ELL students. 38 classes from 19 southern California schools. Findings Four different accommodations used (linguistically modified, a glossary only, extra time only, and a glossary plus extra time). The glossary plus extra time was the most effective accommodation. Glossary plus extra time accommodation Non-ELLs showed a greater improvement (16%) than the ELLs (13%). This is the opposite of what is expected and casts doubt on the validity of this accommodation.

CRESST ONR/NETC Meetings, July 2003, v1 11 Study #6 The effects of accommodations on the assessment of LEP students in NAEP (Abedi, Lord, Kim, & Miyoshi, 2000) SAMPLE: 422 grade 8 ELL and non-ELL students. 17 science classes from 9 southern California schools. Findings Some forms of accommodations may help the recipients with the content of assessment. For example, a dictionary defines all the words in a test, both content and non-content. A Customized Dictionary l Easier to use than a published dictionary l Included only non-content words in the test. l ELL students showed significant improvement in performance. l No impact on the non-ELL performance.

CRESST ONR/NETC Meetings, July 2003, v1 12 Study #7 Language accommodation for large-scale assessment in science (Abedi, Courtney, Leon, Mirocha, & Goldberg, 2001). SAMPLE: 612 grades 4 and 8 students. 25 classes from 14 southern California schools. Findings l A published dictionary was both ineffective and administratively difficult as an accommodation. l Different bilingual dictionaries had different entries, different content, and different format.

CRESST ONR/NETC Meetings, July 2003, v1 13 Study #8 Language accommodation for large-scale assessment in science (Abedi, Courtney, & Leon, 2001) SAMPLE: 1,856 grade 4 and 1,512 grade 8 ELL and non-ELL students. 132 classes from 40 school sites in four cities, three states. Findings l Results suggested: linguistic modification of test items improved performance of ELLs in grade 8. l No change on the performance of non-ELLs with modified test. l The validity of assessment was not compromised by the provision of an accommodation.

CRESST ONR/NETC Meetings, July 2003, v1 14 Study #9 Impact of students’ language background on content-based performance: analyses of extant data (Abedi & Leon, 1999). Analyses were performed on extant data, such as Stanford 9 and ITBS SAMPLE: Over 900,000 students from four different sites nationwide. Study #10 Examining ELL and non-ELL student performance differences and their relationship to background factors (Abedi, Leon, & Mirocha, 2001). Data were analyzed for the language impact on assessment and accommodations of ELL students. SAMPLE: Over 700,000 students from four different sites nationwide. Findings l The higher the level of language demand of the test items, the higher the performance gap between ELL and non-ELL students. l Large performance gap between ELL and non-ELL students on reading, science, and math problem solving (about 15 NCE score points). l This performance gap was zero in math computation.

CRESST ONR/NETC Meetings, July 2003, v1 15 Study #11 Research-supported accommodations for English language learners in NAEP (Abedi, Courtney & Leon, 2002) SAMPLE: 607 grade 4 students (46% ELLs and 54% non-ELLs) and 542 grade 8 students (47% ELLs and 53% non-ELLs) Accommodations: Computer testing, customized dictionary, and extra time. A reading composite score was used as a covariate. Student responses to accommodation follow-up questionnaires and background questionnaires were analyzed. Findings l The computer testing was the most effective accommodation. It provided an alternative test item delivery and an easy-to-access gloss of non-math lexicon. l The customized dictionary was also shown to be effective. l Since non-ELLs who were accommodated performed the same as non-ELLs who were not accommodated, the two effective accommodations are deemed valid.

CRESST ONR/NETC Meetings, July 2003, v1 16 Study #12 Opportunity to Learn for English Language Learners (Abedi, Courtney, & Leon, 2002) SAMPLE: 607 grade 4 students (46% ELLs and 54% non-ELLs) and 542 grade 8 students (47% ELLs and 53% non-ELLs) Findings Student self-reported OTL correlated with their actual performance in math. Teacher-reported OTL (their indication that they taught the materials) did not correlate as high with the student performance.

CRESST ONR/NETC Meetings, July 2003, v1 17 Study #13 Issues and problems in classification of students with limited English proficiency (Abedi & Leon, 2002)  This study examined the validity of LEP classification scheme by analyzing extant data.  LEP classification codes correlated poorly with test scores.  In lower grades, low-performing ELLs tend to remain classified as LEP.  There appears to be a tendency to reclassify these students in higher grades.  Correlation between test scores and LEP classification varies substantially among districts.  The results of longitudinal analyses indicated that in addition to language proficiency, student background variables were also predictors of LEP classification.

CRESST ONR/NETC Meetings, July 2003, v1 18 Study #14 Opportunity to learn for English language learners: OTL and language interaction (Abedi, Courtney, & Leon, 2002) This study examines the differences, if any, in opportunity to learn (OTL) between ELLs and their non-ELL peers in grade 8 math. SAMPLE: 700 grade 8 algebra students (in the 2-year track) Research Questions: Do ELL students receive the same level of OTL as non-ELL students? (Observation/teacher interview/ student OTL questionnaire and field testing) Are the OTL factors influenced by student level of English language proficiency? (Observation/teacher interview/ student OTL questionnaire) Are the OTL factors for ELL students influenced by the teacher’s impression of the ELL students’ ability to learn? (Observation/teacher interview/ student OTL questionnaire)

CRESST ONR/NETC Meetings, July 2003, v1 19 References to CRESST Studies

CRESST ONR/NETC Meetings, July 2003, v1 20 Reports

CRESST ONR/NETC Meetings, July 2003, v1 21 Abedi, Courtney & Leon (2002) Research-Supported Accommodations for English Language Learners in NAEP. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. Abedi, J., Lord, C., & Hofstetter, C. (2001). Impact of Selected Background Variables on Students’ NAEP Math Performance. National Center for Education Statistics (NCES), Working Paper, Publication #: (NCES ). Abedi, J. (2001). Assessment and Accommodations for English Language Learners: Issues and Recommendations. Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. Policy Brief 4. Abedi, J.; Courtney, M. and Leon, S. (2001). Language Accommodation for Large-scale Assessment in Science: Assessing English Language Learners. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

CRESST ONR/NETC Meetings, July 2003, v1 22 Abedi, J.; Lord, C.; Kim, C. & Miyoshi, J (2001). The effects of accommodations on the assessment of LEP students in NAEP. National Center for Education Statistics (NCES), Working Paper, Publication #: (NCES ). Abedi, J., Courtney, M., Mirocha, J., Leon, S., & Goldberg. J. (2000). Language Accommodation for Large-scale Assessment in Science. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. Abedi, J., Lord, C., Kim, C., & Miyoshi, J (2000). The effects of accommodations on the assessment of LEP students in NAEP. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #537. Abedi, J., Leon, S., & Mirocha, J. (2001). Students’ performance differences in standardized achievement tests and background factors: Analyses of Extant Data. University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

CRESST ONR/NETC Meetings, July 2003, v1 23 Abedi, I. Leon, S. (1999). Impact of students’ language background on content-based performance: Analyses of extant data. University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. Abedi, J., Hofstetter, C., Baker, E. & Lord, C. (1998). NAEP math performance and test accommodations: Interactions with student language background, Draft Report. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #536. Abedi, J., Lord, C., & Hofstetter, C. (1997). Impact of selected background variables on students’ NAEP math performance. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #478. Abedi, J., Lord C., & Plummer, J. R. (1997). Language Background as a Variable in NAEP Mathematics Performance. Los Angeles: Center for the Study of Evaluation, CSE Technical Report # 429.

CRESST ONR/NETC Meetings, July 2003, v1 24 Publications

CRESST ONR/NETC Meetings, July 2003, v1 25 Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics and linguistics issues. Educational Assessment (accepted for publication in Educational Assessment). Abedi, J., Hofstetter, C., & Lord, C. (2002) Assessment Accommodations for English Language Learners: A Review of Empirical Research and Policy Issues. Review of Educational research (submitted for publication). Abedi, J. (2002). Issues and problems in classification of students with limited English proficiency. Educational Measurement: Issues and Practice. (submitted for publication). Abedi, J. (2002). Assessing and Accommodations of English language learners: Issues, concerns and recommendations. Journal of School Improvement. v3, n1, Spring 2002.

CRESST ONR/NETC Meetings, July 2003, v1 26 Abedi, J., Lord, C (2001). The Language Factor in Mathematics Tests. Applied Measurement in Education, 14, 3, June Abedi, J. (2000). Loaded Questions? American Language Review, The Magazine for Language Teaching Professional. July/August Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000) Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19, 3, pp