1. Jamal Abedi University of California, Davis/CRESST Validity, Effectiveness and Feasibility of Accommodations for English Language Learners With Disabilities (ELLWD)

2. Accommodations for ELLWD The process of provision of accommodations to ELLWD is more complex than the process used for either SD or ELL students. The process of provision of accommodations to ELLWD is more complex than the process used for either SD or ELL students. Care must be taken to choose accommodations that are appropriate for this particular subgroup of students. Care must be taken to choose accommodations that are appropriate for this particular subgroup of students. ELLWDs need accommodation addressing both their language needs and their disabilities. ELLWDs need accommodation addressing both their language needs and their disabilities.

3.  Why Should English Language Learners Be Accommodated? Their possible English-language deficiency may interfere with their content knowledge performance. Assessment tools may be culturally and linguistically biased for these students. Linguistic complexity of the assessment tools may be a source of measurement error. Language factors may be a source of construct irrelevant variance.

4. Why Should Students With Disabilities be Accommodated? Their disabilities put them at disadvantage. Accommodations must be provided to level the playing field.

5. SubgroupReadingMathLanguageSpelling ELL Status ELL Mean26.334.632.328.5 SD15.2 16.616.7 N62,27364,15362,55964,359 Non-ELL Mean51.752.055.251.6 SD19.520.720.920.0 N244,847245,838243,199246,818 SES Low SES Mean34.338.138.936.3 SD18.917.119.820.0 N92,30294,05492,22194,505 Higher SES Mean48.249.451.747.6 SD21.821.622.622.0 N307,931310,684306,176312,321 Site 2 Grade 7 SAT 9 Subsection Scores

6. ReadingMathMath Calculation Math Analytical Non- ELL/SWD Mean45.6349.3049.0948.75 SD21.1020.4720.7819.61 N921791.18984692.50 ELL only Mean20.2636.0039.2033.86 SD16.3918.4821.2516.88 N692687696699 SWD only Mean18.8627.8228.4229.10 SD19.7014.1015.7615.14 N872843883873 ELL/SWD Mean9.7821.3722.7522.87 SD11.5010.7512.9412.06 N93929794 Site 4 Grade 8 Descriptive Statistics for the SAT 9 Test Scores by Strands

7. Reading Science Math MSD M SD M SD Grade 10 SWD only16.412.725.513.322.511.7 LEP only24.016.432.915.336.816.0 LEP & SWD16.311.224.8 9.323.6 9.8 Non-LEP/SWD 38.016.042.617.239.616.9 All students36.016.941.317.538.517.0 Grade 11 SWD Only14.913.221.512.324.313.2 LEP Only22.516.128.414.445.518.2 LEP & SWD15.512.726.120.125.113.0 Non-LEP/SWD 38.418.339.618.845.221.1 All Students36.219.038.218.944.021.2 Reading Science Math MSD M SD M SD Grade 10 SWD only16.412.725.513.322.511.7 LEP only24.016.432.915.336.816.0 LEP & SWD16.311.224.8 9.323.6 9.8 Non-LEP/SWD 38.016.042.617.239.616.9 All students36.016.941.317.538.517.0 Grade 11 SWD Only14.913.221.512.324.313.2 LEP Only22.516.128.414.445.518.2 LEP & SWD15.512.726.120.125.113.0 Non-LEP/SWD 38.418.339.618.845.221.1 All Students36.219.038.218.944.021.2 Normal Curve Equivalent Means & Standard Deviations for Students in Grades 10 and 11, Site 3 School District

8. Accommodations for ELLs and SWDs Can the same accommodations used for students with disabilities be used for ELLs? Can the same accommodations used for ELLs be used for students with disabilities?

9. How Are We Doing in Practice Nationally? Are the states and districts across the nation cognizant of this important principle of using accommodations that are appropriate for a particular subgroup?Are the states and districts across the nation cognizant of this important principle of using accommodations that are appropriate for a particular subgroup? Are there any objective national criteria to help states to select appropriate accommodations for ELLWD students?Are there any objective national criteria to help states to select appropriate accommodations for ELLWD students? Or, is the assignment of accommodations to these students based on temporary and subjective decisions?Or, is the assignment of accommodations to these students based on temporary and subjective decisions?

10. Accommodation in Stanford 9 Testing Used by a District

11. SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies There are 73 accommodations listed: N:Not Related R:Remotely Related M:Moderately Related H:Highly Related From: Rivera (2003) State assessment policies for English language learners. Presented at the 2003 Large-Scale Assessment Conference

12. N1. Test time increased N2. Breaks provided N3. Test schedule extended N4. Subtests flexibly scheduled N5. Test administered at time of day most beneficial to test taker N = not related; R = remotely related; M = moderately related; H = highly related I. Timing/Scheduling (N = 5) SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies

13. N1. Test individually administered N2. Test administered in small group N3. Test administered in location with minimal distraction N4. Test administered in familiar room N5. Test taker in separate location (or carrel) N6. Test administered in ESL/Bilingual classroom N7. Individual administration provided outside school (home, hospital, institution, etc.) N8. Test taker provided preferential seating N9. Increased or decreased opportunity for movement provided II. Setting (N = 17) N10. Teacher faces test taker N11. Special/appropriate lighting provided N12. Adaptive or special furniture provided N13. Adaptive pencils provided N14. Adaptive keyboards provided N15. Person familiar with test taker administers test N16. ESL/bilingual teacher administers test N17. Additional one-to-one support provided during test administration in general education classroom (e.g. instructional assistant, special test administrator, LEP staff, etc.) SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies N = not related; R = remotely related; M = moderately related; H = highly related

14. R1. Directions repeated in English R2. Directions read aloud R3. Audio-taped directions provided in English N4. Key words or phrases highlighted M5. Directions simplified M6. Audio-taped directions provided in native language M7. Directions translated into native language N8. Cues provided to help test taker remain on task M9. Directions explained/clarified in English III. Presentation (N = 32) M10. Directions explained/clarified in native language M11. Both oral and written directions in English provided M12. Both oral and written directions in native language provided M13. Test items read aloud in English H14. Test items read aloud in simplified/sheltered English N15. Audio-taped test items provided in English H16. Test items read aloud in native language H17. Audio-taped test items provided in native language SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies N = not related; R = remotely related; M = moderately related; H = highly related

15. N18. Assistive listening devices, amplifications, noise buffers, appropriate acoustics provided N19. Key words and phrases in test highlighted H20. Words on test clarified (e.g. words defined, explained) H21. Bilingual word lists, customized dictionaries (word-to-word translations) provided N22. Enlarged print, magnifying equipment, Braille provided N23. Memory aids, fact charts, list of formulas and/or research sheets provided N24. Templates, masks or markers provided III. Presentation (N = 32) N25. Cues (e.g. arrows and stop signs) provided on answer form N26. Acetate shield for page provided N27. Colored stickers or highlighters for visual cues provided R28. Augmentive communication systems or strategies provided (e.g. letter boards, picture communication devices, voice output systems, electronic devices) H29. Simplified/sheltered English version of test provided H30. Side-by-side bilingual versions of test provided H31. Translated version of the test provided N32. Test interpreted for the deaf or hearing impaired/use of sign language provided SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies N = not related; R = remotely related; M = moderately related; H = highly related

16. N1. Test taker marks answers in test booklet N2. Test administrator transfers test-taker’s answers N3. Test taker’s transferred responses checked for accurate marking N4. Copying assistance provided between drafts N5. Test taker types or uses a machine to respond (e.g. typewriter/word processor/computer) N6. Test taker indicates answers by pointing or other method N7. Papers secured to work area with tape/magnets N8. Mounting systems, slant boards, easels provided to change position of paper, alter test taker’s position N.9. Physical assistance provided IV. Response (N = 17) N10. Enlarged answer sheets provided R11. Alternative writing systems provided (including portable writing devices, computers and voice-activated technology) R12. Test taker verifies understanding of directions R13. Test taker dictates or uses a scribe to respond in English N14. Test taker responds on audio tape in English H15. Oral response in native language translated into English H16. Written response in native language translated into English H17. Spelling assistance, spelling dictionaries, spell/grammar checker provided SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies N = not related; R = remotely related; M = moderately related; H = highly related

17. N1. Out-of-level testing provided N2. Special test preparation provided V. Other (N = 2) Preliminary Findings: State Assessment Policies for English Language Learners, SY 2000-2001 GW/CEEE, Large-Scale Assessment Conference 2003 Crivera@ceee.gwu.edu SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies N = not related; R = remotely related; M = moderately related; H = highly related

18. There are 73 Accommodations Listed 47 or 64% are not related 7 or 10% are remotely related 8 or 11% are moderately related 11 or 15% are highly related

19. The most important issue is the concern over the validity of accommodation strategies: Research findings suggest that providing accommodations may increase performance of ELLs/SDs, while also benefiting non-ELLs/SDs. There is not enough research support for many of the accommodations currently being used in the national and state assessments. The only way to make judgments about the efficiency and validity of accommodations used by states is to use them in experimentally controlled studies with both ELL/SWD and non-ELL/SWD students.

20. Some forms of accommodation strategies, such as the use of a glossary with extra time, raised the performance of both ELL and non-ELL students (Abedi, Hofstetter, Lord, and Baker, 1998, 2000) ELL students’ performance increased by 13% when they were tested under glossary with extra time accommodation. While this looks promising, it does not present the entire picture. Non-ELL students also benefited from this accommodation, with an increase of 16%. English and bilingual dictionary recipients may be advantaged over those without access to dictionaries. This may jeopardize the validity of assessment. Concern over the validity of accommodation strategies: A research example

21. Linguistic modification of test items is among these accommodations. This accommodation also helped students with learning disabilities. Thus, an accommodation may have the potential to be effective and valid for both LEP and SD, consequently relevant for LEPWD. There are, however, some accommodations that help ELL students with their English language needs without compromising the validity of assessment. Concerns over the validity of accommodation strategies: A research finding

22. Strategies that are expensive, impractical, or logistically complicated are unlikely to be widely accepted. Validity: The goal of accommodations is to level the playing field for ELL/SWD, not to alter the construct under measurement. Consequently, if an accommodation affects the performance of non-ELL/SWD, the validity of the accommodation could be questionable. Feasibility: For an accommodation strategy to be useful, its implementation must be possible in large-scale assessments. Concern over the validity of accommodation strategies

23. How validity of accommodations can be examined? Only through experimentally controlled research where: ELL/SWD and non-ELL/SWD students are randomly assigned to experimental and control groups Both ELL/SWD and non-ELL/SWD students are observed under accommodated and non-accommodated assessments

24. Can existing data (from national and state assessments) be used for examining the effectiveness and validity of accommodations? Effectiveness of accommodations can be examined only if ELL/SWD students are randomly assigned to the accommodated and non-accommodated conditions Validity of accommodations can be examined only if non-ELL/SWD students are randomly assigned to the accommodated and non- accommodated conditions

25. How the validity of accommodations can be examined? Using existing data? Through experimentally controlled field study?

26. LEP Students Assessed With and Without Accommodations, 1998 NAEP Writing Assessment: National Sample, Public and Nonpublic Schools Combined

28. Research findings Pitoniak, M., Lutkus, A., Cahalan-Laitusis, C., Cook, L. & Abedi, J. (2005). Are Inclusion Policies and Practices for State Assessment Systems and NAEP State Assessments Aligned?Pitoniak, M., Lutkus, A., Cahalan-Laitusis, C., Cook, L. & Abedi, J. (2005). Are Inclusion Policies and Practices for State Assessment Systems and NAEP State Assessments Aligned? Sireci, S. G., Li, S. & Scarpati, S. (2003). The effects of test accommodation on test performance: A review of the literature (Center for Educational Assessment Research Report No. 485). Amherst: University of Massachusetts.Sireci, S. G., Li, S. & Scarpati, S. (2003). The effects of test accommodation on test performance: A review of the literature (Center for Educational Assessment Research Report No. 485). Amherst: University of Massachusetts. Thompson, S., Blount, A., Thurlow, M. (2002). A summary of research on the effects of test accommodations: 1999 through 2001 (Technical Report 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes.Thompson, S., Blount, A., Thurlow, M. (2002). A summary of research on the effects of test accommodations: 1999 through 2001 (Technical Report 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., & Elliott, J. L. (2000). Assessment accommodations research: Considerations for design and analysis (NCEO Technical Report 26). Minneapolis: University of Minnesota, National Center on Educational Outcomes.Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., & Elliott, J. L. (2000). Assessment accommodations research: Considerations for design and analysis (NCEO Technical Report 26). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Tindal, G., & Fuchs, L. (2000). A summary of research on test changes: An empirical basis for defining accommodations. Lexington, KY: University of Kentucky, Mid-South Regional Resource Center.Tindal, G., & Fuchs, L. (2000). A summary of research on test changes: An empirical basis for defining accommodations. Lexington, KY: University of Kentucky, Mid-South Regional Resource Center. Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28

29. How the validity of accommodations can be tested in an experimentally controlled condition? LEP/SWD Status/ Accommodation Accommodated Non- Accommodation LEP/SWD Non-LEP/SWD

30. Conclusions and Recommendation Accommodations: Must be relevant to the subgroups of students Must be effective in reducing the performance gap between accommodated and non-accommodated students Must be valid, that is, accommodations should not alter the construct being measured The results could be combined with the assessments under standard conditions Must be feasible in the national and state assessments

31. Conclusion There is not enough research support for many of the accommodations that are currently used in national and state assessments. The only way to make judgments about the efficiency and validity of these accommodations is to use them in experimentally controlled situations with both ELL/SWD and non-ELL /SWD students and examine their validity and effectiveness under a solid experimental condition. The results of CRESST studies along with other studies nationwide have provided support for some of the accommodations used for ELL students.

32. Conclusion cont. Providing a customized dictionary is a viable alternative to providing traditional dictionaries. The linguistic modification of test items that reduce unnecessary linguistic burdens on students is among the accommodations that help ELL students without affecting the validity of assessments. Computer testing with added extra time and glossary was shown to be a very effective, yet valid accommodation (Abedi, Courtney, Leon and Goldberg, 2003) Examples of research-supported accommodations:

33. Conclusion cont. Without information on important aspects of accommodations such as validity, it would be extremely difficult to make an informed decision on what accommodation to use and how to report the accommodated and non- accommodated results. It is thus imperative to examine different forms of accommodations before using them in state and/or national assessments.

36. For more information, please contact Jamal Abedi at: (530) 754-9150 orjabedi@ucdavis.edu