1. Assessment and Accountability Issues for English Language Learners and Students With Disabilities Oregon Department of Education October 4, 2007 Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (UCLA/CRESST)

2. Accountability Questions Are there specific ELL/SWD subgroup features that affect the accountability system? Yes No Could the current accountability system for ELLs/SWDs be improved? Yes No Do research findings help inform assessment & accountability systems for these students? Yes No

3.  Should Schools Test Children with Disabilities? Assessment outcomes may not be valid because their disabilities interfere with content knowledge performance Test results affect decisions regarding promotion or graduation They may be inappropriately placed into special educational programs where they receive inappropriate instruction SWD students may not have received the same curriculum which is assumed for the test  Yes Students with disabilities (SWD) can be placed at a disadvantage because:

4.  Should Schools Test English Language Learners?  Yes Assessment outcomes may not be valid because their low level of English proficiency interferes with content knowledge performance Test results affect decisions regarding promotion or graduation They may be inappropriately placed into special educational programs where they receive inappropriate instruction ELL students may not have received the same curriculum which is assumed for the test English language learners (ELLs) can be placed at a disadvantage because:

5.  Problems Due to the powerful impact of assessment on instruction, ELL and SWD students’ quality of instruction may be affected If excluded, they will be dropped out of the accountability picture Institutions will not be held responsible for their performance in school They will not be included in state or federal policy decision Their academic progress, skills, and needs may not be appropriately assessed  Should Schools Test English Language Learners and Students with Disabilities?  No

6. Problems in AYP Reporting for ELL Students 1.Problems in classification/ reclassification of ELL students (moving target subgroup) 2.Measurement quality 3.Low baseline 4.Instability of the ELL subgroup 5.Sparse ELL population 6.ELL cutoff points (Conjunctive vs. Compensatory model)

7. Percent of ELL student in 2000- 2001 (Kindler, 2002) California1,511,64625.0% New Mexico 63,75519.9% Arizona 135,24815.4% Texas 570,02214.0% Nevada 40,13111.8% Florida 254,51710.7% Utah 44,030 9.3% Oregon 47,382 8.7%

8. Composition of SWD Population: National Statistics Grade 4 11% Nationally Ranging between 4% to 17% by different states Grade 8 10% Nationally Ranging between 6% to 14% by different states

9. Some recent statistics In 2005-2006 there were a total of 559,215 students were enrolled in the K-12 public schools in Oregon Of the total, 11.7% (65,239) were ELLs (as compared with 8.7% in 2000-2001) and 12.8% (71,517) were Special Education students. Of the 65,239 ELL students, (76.9%) had Spanish as their language of origin which is about 9% of the total students enrolled

10. Population Change Race/ Ethnicity 1992-19932005-20061992-3 to 2005-6 Number of Students Percent of All Students Number of Students Percent of all Students Change in Number Percent Change White 446,25187.5401,08671.7-45,165-10.1 African- American 12,2202.416,7423.0+4,522+37.0 Hispanic 27,1155.384,24415.1+57,129+210.7 Asian/ Pacific Islander 15,3603.025,2044.5+9,844+64.1 Native American 9,1761.811,6782.1+2,502+27.3 TOTAL 510,122100.0538,95496.4+28,832+9.6

11. Special Education 1992-19931997-19982001-20022005-2006 Special Education 54,95263,09770,90271,517 Total Enrollment 510,122540,359551,679559,215 % of Total Enrollment 10.811.712.912.8

12. Language Diversity Language of Origin Number of Students Percent of Students Percent of All Students 559,215 Spanish 50,14376.9%9.0% Russian 3,5585.5%0.6% Vietnamese 1,9873.0%0.4% Ukrainian 9571.5%0.2% Korean 7191.1%0.1% Chinese, Yue 6391.0%0.1% Other 7,23611%1.3% TOTAL65,239100.0%11.7%

13. A Point for Discussion : o This large majority of ELL students speaking the same language makes Oregon one of the most eligible state to use native language testing (in Spanish). o However, the decision is not that simple. Why not, what are the issues? o Alignment of language of assessment and language of instruction o Technical issues in translation or trans- adaptation o Comparability Issues

14. Students with Disabilities and Statewide Testing in Oregon Students with Disabilities being served with active Individual Education Plans (IEP) or 504 plans have a set of choices for participation in Oregon’s Statewide Assessment System. Implementing accommodations and/or modifications Out of level testing (no longer an option in Oregon) Alternative Assessments for those with profound cognitive disabilities

15. Alternative Assessments (Oregon) The Alternate Assessment is comprised of tasks designed to measure basic skills which are anchored to the Oregon standards. The Alternate Assessment is administered for the same age groups (in the Spring). Tasks can be administered in a variety of ways and students can respond with considerable latitude (e.g., pointing, sign language).

16. Challenges in Statewide Testing of ELLs & Students with Disabilities Providing effective and valid accommodations Developing reliable and valid alternate assessment Most importantly, comparability issues

17.  Why Should English Language Learners be Accommodated? Their possible English language limitations may interfere with their content knowledge performance. Assessment tools may be culturally and linguistically biased for these students. Linguistic complexity of the assessment tools may be a source of measurement error. Language factors may be a source of construct-irrelevant variance.

18. Why Should Students with Disabilities be Accommodated? Their disabilities put them at disadvantage Accommodations must be provided to level the playing field

19. Means and Standard Deviations of Total Correct Scores for 3 rd Grades Reading SAT 9 (30 items)Reading DL (40 items) MeanSDMeanSD Gender Females19.55.932.611.3 Males19.66.132.811.5 LEP Non-LEP19.65.932.911.4 LEP12.66.622.110.9 SES No21.15.635.910.6 Yes17.05.827.610.8 Title 1 No19.96.033.511.4 Yes17.45.628.310.4 SWD Non-SWD20.35.634.310.7 SWD15.26.124.411.5 Performance Difference Between SWD and Non-SWD

20. ReadingMathMath Calculation Math Analytical Non-LEP-SWD Mean45.6349.3049.0948.75 SD21.1020.4720.7819.61 N921791.18984692.50 LEP only Mean20.2636.0039.2033.86 SD16.3918.4821.2516.88 N692687696699 SWD only Mean18.8627.8228.4229.10 SD19.7014.1015.7615.14 N872843883873 LEP/SWD Mean9.7821.3722.7522.87 SD11.5010.7512.9412.06 N93929794 Site 4 Grade 8 Descriptive Statistics for the SAT 9 Test Scores by Strands

21. Reading Science Math MSD M SD M SD Grade 10 SWD only16.412.725.513.322.511.7 ELL only24.016.432.915.336.816.0 ELL & SWD16.311.224.8 9.323.6 9.8 Non-ELL/SWD 38.016.042.617.239.616.9 All students36.016.941.317.538.517.0 Grade 11 SWD Only14.913.221.512.324.313.2 ELL Only22.516.128.414.445.518.2 ELL & SWD15.512.726.120.125.113.0 Non-ELL/SWD 38.418.339.618.845.221.1 All Students36.219.038.218.944.021.2 Reading Science Math MSD M SD M SD Grade 10 SWD only16.412.725.513.322.511.7 ELL only24.016.432.915.336.816.0 ELL & SWD16.311.224.8 9.323.6 9.8 Non-ELL/SWD 38.016.042.617.239.616.9 All students36.016.941.317.538.517.0 Grade 11 SWD Only14.913.221.512.324.313.2 ELL Only22.516.128.414.445.518.2 ELL & SWD15.512.726.120.125.113.0 Non-ELL/SWD 38.418.339.618.845.221.1 All Students36.219.038.218.944.021.2 Normal Curve Equivalent Means & Standard Deviations for Students in Grades 10 and 11, Site 3 School District

22. Stanford 9 Sub-scale Reliabilities (Alpha), Grade 9 Non-LEP Students Sub-scale (Items)Hi SESLow SES English Only FEPRFEPLEP Reading, N=205,09235,855181,20237,87621,86952,720 -Vocabulary (30).828.781.835.814.759.666 -Reading Comp (54).912.893.916.903.877.833 Average Reliability.870.837.876.859.818.750 Math, N=207,15536,588183,26238,32922,15254,815 -Total (48).899.853.898.876.802 Language, N=204,57135,866180,74337,86221,85252,863 -Mechanics (24).801.759.803.802.755.686 -Expression (24).818.779.812.804.757.680 Average Reliability.810.769.813.803.756.683 Science, N=163,96028,377144,82129,94617,57040,255 -Total (40).800.723.805.778.716.597 Social Science, N=204,96536,132181,07838,05221,96753,925 -Total (40).803.702.805.784.722.530

23. Grade 11 Stanford 9 Reading and Science Structural Modeling Results, Site 3 All Cases (N=7,176) Even Cases (N=3,588) Odd Cases (N=3,588) Non-LEP (N=6,932) LEP (N=244) Goodness of Fit Chi Square1786943870167581 NFI.931.926.934.932.877 NNFI.898.891.904.900.862 CFI.932.928.936.933.908 Factor Loadings Reading Composite 1.733.720.745.723.761 Composite 2.735.730.741.727.713 Composite 3.784.779.789.778.782 Composite 4.817.722.712.716.730 Composite 5.633.622.644.636.435 Math Variables Composite 1.712.719705709660 Composite 2.695.696.695.701.581 Composite 3.641.628.654.644.492 Composite 4.450.428.470.455.257 Factor Corr Reading/ Math.796.795.797.791

24. Fundamental Questions Why should ELLs and SWDs be accommodated? (to level the playing field) What are the major characteristics of accommodations that would level the playing field? Do the most commonly used accommodations possess those characteristics

25. The major characteristics of accommodations that would level the playing field Effective Valid Consistent with students’ background Feasible Relevant From now on our focus will be on ELL students

26. Problems with Accommodation Usage Accommodations for English language learners are often selected based on feedback from teachers and bilingual coordinators without enough influence from research findings Several studies have identified some accommodations that may provide unfair advantage to the recipients of the accommodations and may render invalid results

27. Examples of Accommodations for ELL Students that May Alter the Construct Providing an English dictionary (Abedi, Courtney, & Leon, 2003; Abedi, Lord, Boscardin, & Miyoshi, 2000) Providing extra or extended time (Abedi, Lord, Hofstetter, & Baker, 2000; Hafner, 2001; Thurlow, 2001) Translating tests into students’ native language (Abedi, Lord, Hofstetter, & Baker, 2000) By gaining access to definition of content-related terms, recipients of a dictionary may be advantaged over those who did not have access to the dictionaries. This may compromise the validity of assessment (Abedi, Courtney, Mirocha, Leon, & Goldberg, 2005)

28. Native language testing  Problems in creating parallel forms of the test  Translation issue  Alignment of language of assessment and language of instruction

29. Glossary with extra time raised the performance of both ELL and non-ELL students (Abedi, Hofstetter, Lord, and Baker, 1998, 2000) ELL students’ performance increased by 13% when they were tested under glossary with extra time accommodation. While this looks promising, it does not present the entire picture. Non-ELL students also benefited from this accommodation, with an increase of 16%. English and bilingual dictionaries recipients may be advantaged over those without access to dictionaries. This may jeopardize the validity of assessment. Glossary Plus Extra Time

30. How can accommodations be examined for validity? Only through experimentally-controlled research where: ELL and non-ELL students are randomly assigned to experimental and control groups Both ELL and non-ELL students are observed under accommodated and non- accommodated assessments Using existing data?

31. How the validity of accommodations can be tested in an experimentally controlled condition? AccommodatedNon- Accommodated ELL Non-ELL

32. Characteristics of ELL students ELL students constitute a very diverse and heterogeneous population (SES, cultural and linguistic backgrounds). They can be vastly different in their level of proficiency in their native or home language They are quite different in their level of proficiency in English Studies show that the level of English language proficiency of these students range from high (even higher than some native English speakers) to very low

33. The Most Commonly Used Accommodations for ELL Students Include: (Rivera, 2003) Extended time (42 of the 48 states) Use of glossary (26 states) Use of an English dictionary (33 states) Use of a bilingual dictionary (22 states) Linguistically-simplified test items (12 states)

34. The Most Commonly Used Accommodations for Students With Disabilities Include: (Thurlow, et al, 2001; Tindal et al, 2000) Braille (allowed by 33 of the 48 states studied) Computerized assessment (34 states) Dictation of responses to a scribe (32 states) Extended time (37 states) Translation of instructions (40 states) Allowance for marking answers in the test booklets (33 states) Test items read aloud (34 states) Simplified test directions (31 states) Test breaks (33 states).

35. Do we have enough evidence on the following characteristics of these most commonly used accommodation? Effective Valid Consistent with students’ background Feasible Relevant

36. Accommodations for ELLs and Students with Disabilities Can the same accommodations used for students with disabilities be used for ELLs? Can the same accommodations used for ELLs be used for students with disabilities?

37. SY 2000-2001 Accommodations Designated for ELLs Cited in States’ Policies There are 73 accommodations listed: N:Not Related R:Remotely Related M:Moderately Related H:Highly Related From: Rivera (2003) State assessment policies for English language learners. Presented at the 2003 Large-Scale Assessment Conference

38. There are 73 Accommodations Listed 47 or 64% are not related 7 or 10% are remotely related 8 or 11% are moderately related 11 or 15% are highly related

39. Samples accommodations used for ELL students  Test-taker marks answers in test booklet  Copying assistance provided between drafts  Test-taker indicates answers by pointing or other method  Paper secured to work area with tape/magnet  Physical assistance provided

40. Samples accommodations used for ELL students  Enlarged answer sheets provided  Breaks provided  Test individually administered  Test administered in small group  Test administered in location with minimal distraction

41. Assessment Options for all ELLs in Oregon Students may take the test under standard administration with or without accommodations Students may take side-by-side English/Spanish, English/Russian versions where provided. Eligible students may respond on the Writing Assessment in Spanish. Students may take the test under modified conditions. Students in grade 3 may take the Aprenda (Spanish reading test at grade 3). Students may use the Juried Assessment process for reading or writing in another language.

42. Test Decisions for ELLs (Oregon) A teacher and instructional team who know the student make the decision to test under standard conditions or modify test Consult parent/guardian Each student must be considered individually for each assessment Best interest of student Not on participation in a program nor identification as an ELL How about research evidence?

43. Accommodations Tables: Changes in Timing or Scheduling Knowledge & Skills Test Writing TestELPA Test Extended Time Frequent Breaks Divide into sessions Most beneficial time of day Extended Time Frequent Breaks Most beneficial time of day Extended Time Frequent Breaks Divide into sessions Most beneficial time of day

44. Accommodations Tables: Changes in Test Directions Knowledge & Skills TestWriting TestELPA Read or reread Sign Translate orally Provide written version of oral directions Simplify language Clarify Highlight words Auditory amplification devices Read or reread Sign Translate orally Provide written version of oral directions Simplify language Clarify Highlight words Auditory amplification devices Read, reread or repeat Sign Translate orally Provide written version of oral directions Clarify Auditory amplification devices

45. Accommodations Tables: Changes in How the Test Questions are Presented Knowledge & Skills TestWriting TestELPA Large print version Braille version Read aloud (not reading test) Student reads test aloud or sub- vocalizes Proctor highlights vowel combinations (not reading test) Student highlights vowel combinations Visual magnification devices Large print version Braille version Read aloud Sign Student reads test aloud or sub- vocalizes Electronic word-for- word, text-to-voice scanning Proctor highlights vowel combinations Visual magnification devices Student reads test aloud or sub- vocalizes Visual magnification devices

46. Accommodation Tables: Changes in How the Student Responds Knowledge & Skills TestWriting TestELPA Answers marked in test booklet or recorded & transcribed onto answer sheet Use of assistive technology device that serves as primary communication mode Point to or dictate responses to a scribe Student retells story Student vocalize first Use a recording device to record/playback Students who require increased spacing can do so Respond in Braille Use of assistive technology device that serves as primary communication mode (with certain computer features disengaged) Function keys may not be used Student vocalize first Student retells story to proctor before responding to questions Point to or dictate responses to a scribe (in English or native language) Use of assistive technology device that serves as primary communication mode (with certain computer features disengaged) Function keys may not be used

47. Accommodation Tables: Changes in Test Setting Knowledge & Skills TestWriting TestELPA Test student in a separate location Test small group of students in separate location Minimize distractions Encourage student’s work habits during test Stabilize test material/papers Use sensory processing techniques Use adaptive furniture or positioning Test student in a separate location Test small group of students in separate location Minimize distractions Encourage student’s work habits during test Stabilize test material/papers Use sensory processing techniques Use adaptive furniture or positioning Physical setting Use of physical assistance or devices Test student in a separate location Test small group of students in separate location Minimize distractions Physical setting (seating, lighting) Stabilize test material/papers Use sensory processing techniques Use adaptive furniture or positioning

48. How are we doing in practice nationally?  Are states and districts across the nation cognizant of this important principle of using accommodations that are appropriate for ELLs?  Are there any objective national criteria to help states to select appropriate accommodations for ELL students?  Or, is the assignment of accommodations to these students based on temporary and subjective decisions?

49. Comparability Issues If accommodated assessment is not valid then the outcome may not be comparable with the non-accommodated assessment (a major peer- review concern) To report AYP for ELL students, it is imperative to establish the validity of accommodated assessment Construct-irrelevant sources such as linguistic and cultural biases should be controlled before reporting AYP based on accommodated assessments

50. Are there accommodations that would benefit both ELLs and SWDs? Assessment can be designed in a way to be accessible to both groups. For example, long tests, crowded pages, tables and texts would create frustration and anxiety for everyone, particularly for ELLs and SWDs. Complex linguistic structure of assessment would be a major nuisance variable for ELLs and SWDs ( Tindal, G., Anderson, L., Helwig, R., Miller, S., & Glasgow, A. (2000). Accommodating students with learning disabilities on math tests using language simplification. Eugene: University of Oregon, RCTP)

51. 3 rd Graders Frequencies of Special Education Groupings Overall %Non-LEP %LEP % Regular Education84.184.093.9 Educable Mentally Handicapped1.1 0.0 Seriously Emotionally Disturbed0.4 0.0 Learning Disability10.010.15.1 Trainable Mentally Handicapped0.0 Severely Mentally Handicapped0.0 Physically Impaired0.7 0.0 Hard of Hearing – Partially Deaf0.2 0.0 Blind0.0 Partially Sighted0.0 Autistic0.0 Deaf and Blind0.0 Speech3.4 1.0 Total100.0 A Sample Representing a Subgroup of Students with Disabilities

52. 8TH Graders Frequencies of Special Education Groupings Overall %Non-LEP %LEP % Regular Education86.886.792.3 Educable Mentally Handicapped1.31.23.6 Seriously Emotionally Disturbed0.7 0.0 Learning Disability10.010.13.6 Trainable Mentally Handicapped0.1 0.0 Severely Mentally Handicapped0.1 0.0 Physically Impaired.05 0.0 Hard of Hearing – Partially Deaf0.1 0.0 Blind0.0 Partially Sighted0.0 Autistic0.0 Deaf and Blind0.0 Speech0.3 0.0 Total100.0 A Sample Representing a Subgroup of Students with Disabilities

53. A Clear Language of Instruction and Assessment Works for ELLs, SWDs, and Everyone What is language modification of test items?

54. Learning Disability & Language of Assessment Students in the Learning Disability category may have difficulty processing complex language in assessment Simplifying the language of test items will help students with disabilities, particularly those with learning disabilities As the sample page suggests, a large majority of students with disabilities are in the Learning Disability category

55. Examining Complex Linguistic Features in Content-Based Test Items

56. Familiarity/frequency of non-math vocabulary: unfamiliar or infrequent words changed census > video game A certain reference file > Mack ’ s company Length of nominals: long nominals shortened last year ’ s class vice president > vice president the pattern of puppy ’ s weight gain > the pattern above Question phrases: complex question phrases changed to simple question words At which of the following times > When which is best approximation of the number > approximately how many Linguistic Modification Concerns

57. Conditional clauses: conditionals either replaced with separate sentences or order of conditional and main clause changed If Lee delivers x newspapers > Lee delivers x newspapers If two batteries in the sample were found to be dead > he found three broken pencils in the sample Relative clauses: relative clauses either removed or re-cast A report that contains 64 sheets of paper > He needs 64 sheets of paper for each report Voice of verb phrase: passive verb forms changed to active The weights of 3 objects were compared > Sandra compared the weights of 3 rabbits If a marble is taken from the bag > if you take a marble from the bag Linguistic Modification cont.

58. Harriet, Jim, Roberto, Maria, and Willie are in the same eighth grade class. One of them is this year ’ s class president. Based on the following information, who is the class president? The class president was last year ’ s vice president and lives on Vince Street. Willie is this year ’ s class vice president. Jim and Maria live on Cypress Street. Roberto was not last year ’ s vice president. A. Jim B. Harriet C. Roberto D. Maria E. Willie Harriet, Jim, Roberto, Maria, and Willie are in the same eighth grade class. One of them is this year ’ s class president. Based on the following information, who is the class president? The class president was last year ’ s vice president and lives on Vince Street. Willie is this year ’ s class vice president. Jim and Maria live on Cypress Street. Roberto was not last year ’ s vice president. A. Jim B. Harriet C. Roberto D. Maria E. Willie Original Item

59. Modified Item Harriet, Jim, Roberto, Maria, and Willie ran for president of their eight-grade class. One of them won. Who is president? The president now was vice president last year and lives on Vince Street. Willie is vice president now. Jim and Maria live on Cypress Street. Roberto was not vice president last year. A. Jim B. Harriet C. Roberto D. Maria E. Willie

60. Original: The census showed that three hundred fifty-six thousand, ninety-seven people lived in Middletown. Written as a number, that is: A. 350,697 B. 356,097 C. 356,907 D. 356,970 Modified: Janet played a video game. Her score was three hundred fifty-six thousand, ninety-seven. Written as number, that is: A. 350,697 B. 356,097 C. 356,907 D. 356,970

61. Interview Study Table 1. Student Perceptions Study: First Set (N=19) Item #Original item chosenRevised item chosen 1316 2415 3109 4118 Table 2. Student Perceptions Study: Second Set (N=17) Item #Original item chosenRevised item chosen 5314 64.5a12.5 7215 8215

62. Many students indicated that the language in the revised item was easier: “Well, it makes more sense.” “It explains better.” “Because that one’s more confusing.” “It seems simpler. You get a clear idea of what they want you to do.”

63. “It’s easier to read, and it gets to the point, so you won’t have to waste time.” “I might have a faster time completing that one ’cause there’s less reading.” “Less reading; then I might be able to get to the other one in time to finish both of them.” “’Cause it’s, like, a little bit less writing.” The revised items need less time for response:

64. Conclusions and Recommendations Accommodations: Must be relevant in addressing assessment issues for ELL students Must be effective in reducing the performance gap between accommodated and non-accommodated students Should not alter the construct being measured The accommodated results can be aggregated with the assessments under standard conditions Must be feasible in national and state assessments

65. Conclusions and Recommendations Providing a customized dictionary is a viable alternative to providing traditional dictionaries. The linguistic modification of test items that reduce unnecessary linguistic burdens on students is among the accommodations that help ELL students without affecting the validity of assessments. Computer testing with added extra time and glossary was shown to be a very effective, yet valid accommodation (Abedi, Courtney, Leon, and Goldberg, 2003) Examples of research-supported accommodations:

66. Conclusions and Recommendations Without information on important aspects of accommodations such as validity, it would be extremely difficult to make an informed decision on what accommodations to use and how to report the accommodated and non- accommodated results. It is thus imperative to examine different forms of accommodations before using them in state and/or national assessments.

67. Accountability Questions Are there specific ELL/SWD subgroup features that affect the accountability system? Yes No Could the current accountability system for ELLs/SWDs be improved? Yes No Do research findings help inform assessment & accountability systems for these students? Yes No

68. For more information, please contact Jamal Abedi at UC Davis/CRESST: (530) 754-9150 or jabedi@ucdavis.edu