1. Book 3: Use of Accommodations in Large-Scale Assessments Practical Guidelines for the Education of English Language Learners

2. The Center on Instruction is operated by RMC Research Corporation in partnership with the Florida Center for Reading Research at Florida State University; Horizon Research, Inc.; RG Research Group; the Texas Institute for Measurement, Evaluation, and Statistics at the University of Houston; and the Vaughn Gross Center for Reading and Language Arts at the University of Texas at Austin. The contents of this PowerPoint were developed under cooperative agreement S283B050034 with the U.S. Department of Education. However, these contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. 2007 The Center on Instruction requests that no changes be made to the content or appearance of this product. To download a copy of this document, visit www.centeroninstruction.org

3. Practical Guidelines for the Education of English Language Learners Authors David J. Francis, Mabel Rivera Center on Instruction English Language Learners Strand Texas Institute for Measurement, Evaluation, and Statistics University of Houston Nonie K. Lesaux, Michael J. Kieffer Graduate School of Education Harvard University Héctor Rivera Center on Instruction English Language Learners Strand Texas Institute for Measurement, Evaluation, and Statistics University of Houston

4. Research-based Recommendations for Instruction and Academic Interventions Research-based Recommendations for Serving Adolescent Newcomers Research-based Recommendations for the Use of Accommodations in Large-scale Assessments Practical Guidelines for the Education of English Language Learners

5. Book 3: Use of Accommodations in Large-Scale Assessments Foreword Overview Accommodations and Review of State Policies Conceptual Framework Use of Accommodations Meta-Analysis on Effectiveness of Accommodations in Assessment Selected studies for analysis Methods Results Conclusions Technical Appendices

6. Seminal Research Reviews Rivera, C., Collum, E., Shafner Willner, L., & Sia, J. K. (2006). An analysis of state assessment policies regarding the accommodation of English language learners. In C. Rivera and E. Collum (Eds.), State assessment policy and practice for English language learners: A national perspective (pp. 1-173). Mahwah, NJ: Lawrence Erlbaum. Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28. Sireci, S., Li, S., & Scarpati, S. (2003). The effect of test accommodation on test performance: A review of the literature (Research Report No. 495). Amherst, MA: University of Massachusetts School of Education, Center for Educational Assessment.

7. Demographics

8. Who are English Language Learners? National-origin-minority students with limited proficiency of English; Heterogeneous; Membership defined by limited proficiency in English language use, which directly affects learning and assessment; Membership is expected to be temporary.

9. Demographics One of the fastest- growing groups among school-aged children in U.S. More than 9 million students, roughly 5.5 million of whom are classified as Limited English Proficient (LEP) Large population More than 400 different home languages; Spanish predominant (70%) Spanish predominant Largest and fastest growing ELL groups: Students who immigrated before Kindergarten U.S. born children of immigrants Largest growth By 2015, second-generation children of immigrants are expected to be 30% of the school-aged population

10. Learning challenges ELLs face unique learning challenges: to develop the content-related knowledge and skills defined by state standards; while simultaneously acquiring a second (or third) language ; at a time when their first language is not fully developed (e.g., young children); to demonstrate their learning on assessments in English, their second language.

11. ELL Performance Outcomes Some states have begun to look at the performance of ELLs on state tests after they have gained proficiency in English. Although some reclassified ELLs do well, many students who have lost the formal LEP designation continue to struggle with: academic text; content-area knowledge; and oral language skills.

12. Current Policy and Academic Achievement

13. English Language Learners and the No Child Left Behind Act ELLs present unique challenges to: Teachers, Administrators, Assessment systems, and Accountability systems.

14. NCLB: all High standards of learning and instruction for all students; English Language Learners one of five areas of concentration to advance student achievement; Increased awareness of the academic needs and achievement of ELLs; and Schools, districts, and states held accountable for teaching English and content knowledge to ELLs. English Language Learners and the No Child Left Behind Act

15. Under NCLB, state education agencies are held accountable for the progress of ELLs in two ways: Adequate Yearly Progress (AYP) expectations for reading and mathematics under Title I, and Annual Measurable Achievement Objectives (AMAO) under Title III, demonstrating satisfactory progress in learning English and attaining English proficiency.

16. Academic Performance Indicators for ELLs On 4th grade National Assessment of Educational Progress (NAEP), ELLs were: only1/4 as likely to score proficient or above in Reading as their native English speaking peers, and only 1/3 as likely to score proficient or above in Math as their native English-speaking peers. Compared with native English-speaking peers, ELLs with a formal LEP designation are less likely to score “proficient” on state tests.

17. ELLs Taking Large-Scale Assessments: Participation and Valid Assessment

18. ELL Participation Rates NCLB recognizes the importance of high participation to gain information about achievement and proficiency rates for this subgroup of learners. ELLs are participating in large-scale assessments at much higher rates than in the past. This raises awareness of the academic needs and academic achievement of ELLs as a distinct population. Assessment results can be vital indicators of school performance and efficacy in serving specific subgroups.

19. ELLs & Large-Scale Assessments It is not enough simply to have ELLs participate in large-scale assessments; participation must lead to valid inferences about their achievement and the effectiveness of educational programming. Valid assessments are needed to inform instruction and program design: They provide detailed knowledge of ELL students’ weaknesses and strengths.

20. Large-Scale Assessments Used with ELLs National Assessment of Educational Progress (NAEP); State assessments of academic content; State assessments of language proficiency.

21. NAEP Assessment The only “nationally representative and continuing assessment” of subject area knowledge of U.S. students; Does not score individual students or schools, but now allows some state-by-state comparison; NCLB requires some schools to participate in NAEP grade 4 and 8 reading and math. (National Center for Educational Statistics, 2007)

22. State Assessments of Language Proficiency NCLB requires annual assessment of ELLs’ English language proficiency with assessments aligned to K-12 ELP standards. These must include the 4 language domains: Speaking and listening; Reading and writing. State ELP standards must align to the academic content standards and the challenging academic achievement standards set by the state for all students. (General Accounting Office, 2006)

23. State Assessments of Academic Content Based on federally-mandated, state-defined standards; Selected or developed by each state, with guidance from the U.S. Department of Education, Office of Elementary and Secondary Education. (August & Piche, 2002; General Accounting Office, 2006)

24. Content Knowledge & Language Proficiency ELLs must direct more cognitive resources to processing the language of the test compared with fully English proficient students. Assessments with the most linguistically challenging content show the largest performance gaps between ELLs and native English speakers. This is true for all domains, including math. ELLs may also lack background knowledge that is central to understanding content, but may not be related to actual content knowledge. Effective accommodations allow students to demonstrate their knowledge of a concept by minimizing language obstacles without invalidating the assessment.

25. State Policies Educational agencies across the nation provide accommodations to ELLs as needed; The criteria for selection and strategies for implementation vary by state, according to many factors.

26. State Policies Rivera and colleagues (2006) A comprehensive taxonomy for thinking about accommodations, both direct and indirect linguistic support accommodations.

27. States Must Improve Accountability, Assessment, and Alignment for Education of ELLs States Must Improve Accountability, Assessment, and Alignment for Education of ELLs Accountability Schools are accountable for educational quality, both to those who fund them and to those whom they serve. Assessment External assessment (state and district tests) and classroom assessment are intended to both reflect educational quality and enhance it. Alignment There should be a match between external standards for quality and curricula and classroom practices (Menken, 2000).

28. Validity of Academic Knowledge Assessments for ELLs ELLs must direct more cognitive resources to processing the language of the test compared with fully English proficient students; ELLs may also lack relevant background knowledge that is central to understanding content, but may not be related to actual content knowledge.

29. Validity of Academic Knowledge Assessments for ELLs Using accommodations during testing may improve validity. These would control for language and cultural factors. Valid assessments are also useful for instruction and program design. They provide detailed knowledge of ELL students’ weaknesses and strengths. (Francis et al., 2006)

30. Consequences of Invalid Assessment Does not inform the design of effective programs and curricula;. May undermine quality and appropriateness of individual placement and instruction; May limit a student’s academic achievement.

31. Possible Consequences of High Stakes Assessment Possible Consequences of High Stakes Assessment For schools performing poorly: Federal funds withheld; Schoolwide restructuring; Student transfers allowed; Teachers or administrators unrewarded. For students performing poorly: Additional resources, tutoring, supplemental services and school choice options. (August & Piche, 2002; Coltrane, 2002; Abedi et al., 2003; Francis, 2006)

32. Effective Accommodations for ELLs Taking Large-Scale Assessments

33. What are accommodations? Encompass changes to standard test administration procedures, including: How the assessment is presented to the student; How the student is allowed to respond; Any equipment or materials to be used; Extent of time allowed to complete the test; and Changes to the environment in which the student takes the test.

34. Test Accommodations for ELLs Use of accommodations during testing may improve validity of inferences based on test results. For ELLs participating in large-scale tests, accommodations are modifications to the test or testing conditions designed to reduce the impact of limited English proficiency on the assessment of the target academic skill.

35. Test Accommodations for ELLs Criteria for selection and strategies for implementation vary by state. Appropriate accommodations will address ELLs’ linguistic needs either: directly (e.g., language in test) or indirectly (e.g., increased time to process language).

36. Types of Accommodations for ELLs Modification of the testing conditions: Special scheduling, setting, or extended time; Use of tools such as bilingual or English dictionaries or glossaries.* Modification of the test: Directions or items may be read aloud in English or in native language; may be bilingual, native language, or simplified English version of test;* Response options: response in native language; dictated response.* *Targets language

37. Characteristics of Appropriate Accommodations Take the student’s background into account: Native language literacy, if test is to be written in native language; Familiarity with the accommodation from classroom use, as with prior use of dictionaries; Language of assessment and language of instruction need to match. Their use in combination must be guided by a specific rationale: For example, use of dictionaries and extended time are a reasonable combination.

38. Partial List of Accommodations Responsive to Needs of ELLs Accommodations of Testing Conditions Extended time* Breaks offered between sessions Bilingual glossaries* Bilingual dictionaries* English glossaries* English dictionaries* Accommodations as Test Modifications Directions read in English Directions read in native language Directions translated into native language Simplified English* Side-by-side bilingual version of the test* Native language test* Dictation of answers or use of a scribe Test taker responds in native language * Denotes the accommodations examined in the meta-analysis

39. Effective Accommodations for ELLs: Results of a Meta-Analysis Research Findings

40. What is a Meta-Analysis? A meta-analytic review is a specific approach to research synthesis that attempts to quantify the effect of an intervention from a set of comparable studies.

41. Preliminary Research We examined the effect of 8 types of accommodations which, alone or combined, affected the performance of ELLs in large-scale assessments of achievement.

42. Most Common Accommodations for ELLs (GAO, 2006) Most Common Accommodations for ELLs (GAO, 2006) AccommodationNumber of states Bilingual dictionary32 Reading items aloud in English32 Small-group administration29 Extra time27 Individual administration27 Separate location25 Extra breaks25 Directions in student’s native language24

43. Effective Accommodations for ELLs: Results of a Meta-Analysis 11 studies in total: Each study randomly assigned ELLs and non- ELLs to testing conditions with and without accommodations; Involved 37 different samples of students; Reported 37 different tests of the effectiveness of accommodations for ELLs.

44. Study Descriptions Grades included: 11 studies with 4th graders; 22 studies with 8th graders; 2 studies with 5th graders; 2 studies with 6th graders. Subject Areas: 17 studies tested math skills; 19 studies tested science skills; 1 study tested reading skills.

45. Study Descriptions (cont’d) Type of test 22 studies examined items from the National Assessment of Educational Performance (NAEP); 6 studies examined items from the NAEP and the Trends in International Mathematics and Science Study (TIMSS); 9 studies examined items from State Accountability Assessments (two different states).

46. Seven Accommodations Studied Simplified English (15 studies) English dictionary/glossary (11 studies) Bilingual dictionary/glossary (5 studies) Extra time (2 studies) Spanish language test (2 studies) Dual language questions (1 study) Dual language booklet (1 study)

47. Accommodations Simplified English Involves linguistic changes in the vocabulary and grammar of test items to eliminate irrelevant complexity while keeping the content the same. Some changes may be effected by eliminating non- content related vocabulary, shortening sentences, simplifying sentence structures where possible, using familiar or frequently used words, active instead of passive voice, and using present verb tense where possible.

48. Accommodations English dictionary/glossary Involves adding definitions or simple paraphrases for potentially unfamiliar or difficult words in test booklets (usually on the margins). A variation on this accommodation is to provide computerized tests with built-in English glossaries. Typically, this latter variation involves a computer program that provides a simple and item-appropriate synonym for each difficult non-content word in a test.

49. Accommodations Bilingual dictionary/glossary ELLs are given access to dictionaries, glossaries, and marginal glossaries with words written in English and the student’s native language. Another version of this accommodation uses computerized tests with built-in bilingual glossaries.

50. Accommodations Extra time Providing more time than usual to complete test sections is among the most frequently used accommodations. This accommodation does not change the test itself, but the testing conditions. Extended time is usually provided in combination with other accommodations. The rationale is to allow ELLs extra time to process the language of the test, or when bundling extra time with another accommodation, such as an English language dictionary, to allow time to use the other accommodation.

51. Accommodations Native language test Typically, text is not translated, but adapted to preserve the meaning of the original text. The most preferred method of adapting a test to another language is to use back translation: First a proficient speaker, reader, and writer of both languages translates the original language of the test into the native language; An independent, bilingually proficient person translates the adapted test back into the original language and compares the two original language tests for equivalence; If the two original language versions are deemed to be different, the process is repeated, focusing on correcting those areas of the test which were not successfully adapted.

52. Accommodations Dual-language questions/Dual-language booklet: Changes the format of test booklets; Booklets have English items on one page and a translation into the learner’s first language on the facing page.

53. Criteria for Evaluating Accommodations Effectiveness The extent to which the accommodation leads to improved test scores for the student. Validity Controls for the student’s English and for cultural differences, but does not alter the target skill being measured; Accommodation only affects the performance of students who need it. Practicality Cost and effort involved in implementing the accommodation.

54. Do Accommodations Affect Assessment Validity? Francis et al. (2006) reviewed the research on the effectiveness of accommodations for ELLs in large-scale achievement tests. Method: Using meta-analysis, Francis et al. quantified the effects of seven different types of accommodations on the performance of ELLs (compared with non-ELLs) in tests of math, science, and reading.

55. Do Accommodations Affect Assessment Validity? Francis, et al. (2006) Results: Of the 7 types of accommodations, only English language dictionaries and glossaries had overall positive effects on ELL outcomes; Native language tests and bilingual glossaries were sometimes effective, but not always; There are too few research studies to draw firm conclusions. The effectiveness of a particular accommodation likely depends on how it relates to student backgrounds and characteristics, which other accommodations were offered, etc. Still, the characteristics of good accommodations reviewed above appear valid.

56. ELL TEST OUTCOMES ENGLISH DICT./ GLOSSARY BILINGUAL DICT/ GLOSSARY SIMPLIFIED ENGLISH DUAL LANGUAGE TEST DUAL LANGUAGE QUESTIONS EXTRA TIME SPANISH TEST Effective Accommodations

57. English dictionaries/glossaries Of the seven types of accommodations used, only one had an overall positive effect on ELL outcomes: English language dictionaries and glossaries Produced an average effect—positive and statistically different from zero; No indication that this effect varied across the conditions studied.

58. Providing English Dictionaries & Glossaries Full dictionaries are not typically used; Glosses for specific words from the test appear on the test page; Separate, abbreviated, alphabetical dictionaries are tailored to the test vocabulary. (Abedi et al., 2001)

59. Bilingual Dictionaries and L1Tests Bilingual dictionary/glossary and Spanish language test did not show overall positive effects, but showed varying effects for different groups of students; Results suggest that L1 accommodations might be effective only when ELLs: had received L1 instruction; or are literate in their L1; However, more research is necessary.

60. Findings Effectiveness of native language tests and bilingual glossaries varied; Too few studies to say conclusively why they are not effective, but some possible reasons are: Language of assessment and language of instruction do not match; Students are not literate in L1.

61. Findings (cont’d) Results for Simplified English (one of the most widely used accommodations) were less promising than might have been expected; Lack of effects for Simplified English is not an indictment of universal design.

62. Other Accommodations Based on findings, Simplified English would not be judged as an effective accommodation. However, studies were narrowly focused in terms of grades, content areas, and type of assessment. There were two few studies of Extra Time and Dual Language Tests to draw any final conclusion. More research is needed to further explore additional accommodations in varying conditions.

63. Limitations Research base is limited in important ways: Few studies involve State accountability tests; Few studies in reading and language arts; No accommodation has been studied definitively; Many more accommodations need to be studied; Effects of accommodations need to be studied in relation to different conditions, e.g., content areas, grade levels, test types, students’ characteristics.

64. Important reminders ELLs cannot be assumed to be literate in L1. L1 assessments cannot be assumed to offer linguistically appropriate accommodations. More cannot be assumed to be better. An explicit rationale is needed for combining specific accommodations.

65. Implications of Preliminary Research Alignment of curriculum, instruction, and assessment is crucial to the academic success of all students; Accommodations alone are not effective in creating valid, effective assessment conditions for ELLs; For any accommodation to be successful in the testing situation, students must have experience with it during regular instruction.

66. Implications of Preliminary Research (cont’d) Teachers can incorporate the most effective language accommodation, the use of English dictionaries and glossaries, into classroom vocabulary-learning activities.

67. Academic Language and the Importance of Vocabulary Development

68. The Role of Teachers in Using Test Accommodations for ELLs For an accommodation to be effective in a testing setting, ELLs must be familiar with its use; Teachers can incorporate the most effective language accommodation, English dictionaries and glossaries, into classroom vocabulary-learning activities; Teaching academic English vocabulary is important in all classes at all levels. (Francis et al., 2006)

69. The Role of English Vocabulary in Academic Learning Vocabulary in academic texts and classroom tasks differs from conversational vocabulary; Academic vocabulary is key to learning higher-level content and to performing well on achievement tests; ELLs are learning academic English and academic subject content simultaneously. (Scarcella, 2003; Short & Fitzsimmons, 2006)

70. What is Academic Language? Vocabulary knowledge: Breadth—knowing the meanings of many words, including many words for the same, or related, concepts; Depth—knowing many meanings, both common and uncommon, for a given word; Understanding complex sentence structures and syntax; Written vocabulary as distinct from oral vocabulary; Understanding the structure of argument, academic discourse, and expository texts.

71. What is Academic Language? Other aspects of academic language relate to the text or word problem that is central to the assessment: Organization of expository paragraphs; Function of connectives such as therefore and in contrast; Wide range of vocabulary that appears far more often in text than in oral conversation; Specific academic vocabulary—the words necessary to learn and talk about academic subjects (analyze, abstract, estimate, observe).

72. Academic Language: The Key to Academic Success Developing academic language— fundamental to academic success in all domains; a primary source of ELLs’ difficulties with academic content across grades; often a challenge after students achieve proficiency on state language proficiency tests; influences ELLs performance on large-scale assessments.

73. Academic Language: The Key to Academic Success Good conversational English skills may be accompanied by limited academic language skills: Many elementary and middle school students—ELLs, reclassified ELLs, and native English speakers—in urban schools have academic vocabulary scores around the 20 th percentile. Scores below the 20 th percentile are not uncommon.

74. What Words Should be Taught to ELLs? Frequently used English words: may be difficult for ELLs, or ELLs may not know all of their meanings; Words that students will encounter in reading and writing across disciplines; Important content-area vocabulary; English-Spanish cognates, for Spanish- speaking ELLs. (Stahl & Nagy, 2006)

75. Three Principles of Vocabulary Instruction Present definitional information as well as many examples of use in context; Actively involve students in word learning; Provide multiple exposures to meaningful information about words. (Stahl & Nagy, 2006)

76. Core Aspects of Vocabulary Teaching Strategies Using definitions, discussing usage contexts, and analyzing word parts are basic, complementary strategies; Preparing students to use these sources of information: Develop an awareness of words and their complexity; Learn to recognize one’s own comprehension problems, along with strategies to attack them. (Stahl & Nagy, 2006)

77. Effective Dictionary Instruction for Different Literacy Levels For beginning literacy levels A picture dictionary teaches alphabetization and parts of the dictionary (Echevarria et al., 2004). For advanced English learners Use of bilingual dictionaries while reading increases vocabulary knowledge (Luppescu & Day, 1995). Learning to identify key content vocabulary teaches word and comprehension awareness (Echevarria et al, 2004). For all learners Academic vocabulary should be explicitly taught in all classes, combining dictionary use and other techniques (Francis et al., 2006; Stahl & Nagy, 2006).

78. T eaching the Structure of Dictionary Definitions Begin with words students already know. Show that definitions usually designate: the category (“genus”) of the word (e.g. a lecture is a type of speech, a whale is a type of mammal); how the word differs from others in the category (e.g., lectures are primarily educational, unlike comedy routines). Present word part analysis and contextual information along with the definition. Have students actively engage with the word’s meaning. (Stahl & Nagy, 2006)

79. Learning to Analyze Word Parts Understanding the meanings of English affixes boosts vocabulary learning (White et al., 1989): Eleven prefixes are present in 81% of prefixed words; Six suffixes are present in 80% of suffixed words; Understanding the meaning of common Latin- and Greek-derived roots boosts academic vocabulary learning (Hiebert & Kamil, 2005); Exercises with dictionary definitions can incorporate discussion of word parts.

80. Active Learning Strategies for Analyzing Definitions Use concept of definition maps (Schwartz & Raphael, 1985); Use group discussion as well as whole-class discussion; Have children rewrite definitions in their own words; Once children understand dictionaries, have them use dictionaries while reading. (Stahl & Nagy, 2006)

81. Learning Strategies Using Contextual Information and Clues Have students create sentences or whole stories using the new word and compare with those of others; Discuss the meaning of the same word in different sentences; Expose students to contexts that use different senses of the word. (Stahl & Nagy, 2006; Sternberg & Powell,1983; Nist & Olejnik, 1995)

82. Other Active Vocabulary Learning Strategies Personal dictionaries of unknown words; Word study books arranged by morphology (e.g., suffixes) (Buehl, 1995); A classroom Word Wall of key content vocabulary; Maps of conceptual knowledge ranging beyond definitional information (Buehl, 1995); Word sorts by morphology (e.g., affix types) or meaning; Whole-class word-generation games (e.g., “How many words contain port, and what do they mean?”). (Echevarria et al., 2004)

83. The Future of ELL Assessment Developing a stronger research base may lead to better test design or better guidance for using specific accommodations. Native language and alternate assessments are challenging to develop and implement, but may be useful in some situations. The academic achievement of ELLs remains a strong national priority, and assessment has an important role to play in today’s accountability agenda. (G.A.O., 2006)