1 Assessment, Research, & Accountability Update David Abrams Assistant Commissioner for Standards, Assessment, and Reporting 2008 Teachers’ Institute

2 Performance of ELL Students on 2008 Grade 3-8 ELA Tests  This analysis summarizes the performance of NYS public and charter school ELLs on the Grades 3-8 English Language Arts tests in The ELA performance of former ELLs who exited in 2006 and 2007 is also presented.  The analyses are based on 72,698 ELLs who took the State Grades 3-8 ELA tests in January 2008 and the NYSESLAT in May The analyses of the former ELLs are based on the 30,564 ELLs who exited in 2006 and 2007 and whose 2008 Grades 3-8 ELA scores are available.  The ELL population has significantly improved their performance on the Grades 3-8 ELA tests between 2007 and The percentage of ELLs scoring at Level 3 and above has increased and the percentage of ELLs showing serious academic deficiencies has decreased.  Former ELLs had comparable chance to score proficient (Level 3 or above) on the Grades 3-8 ELA tests as the English proficient students.

3 Number of ELLs Tested On Grades 3-8 ELA Tests

4 Performance of ELLs On 2008 Grades 3-8 ELA Tests

5 The Mean Scale Score increased for ELLs in every grade, though the smallest increase is seen in Grade 8

6 There was an Increase in the Percentage of ELLs Scoring At Levels 3 and 4 except for Grade 8

7 There was an Increase in Percentage of ELLs Scoring At Level 2 and Above

8 There was a Noticeable Decrease in Percentage of ELLs Scoring at Level 1

9 There was an Increase in Percentage of Proficient ELLs Scoring at Levels 3 and 4, but no gain in Grade 8

10 Performance of ELLs on Grades 3-8 ELA Tests by NYSESLAT Proficiency Level: As Student Performance on NYSESLAT increases, there is a greater chance that students will score at Levels 2 and/or Levels 3 & 4

11 Grade 3 ELL Performance By NYSESLAT Proficiency Level

12 Grade 4 ELL Performance By NYSESLAT Proficiency Level

13 Grade 5 ELL Performance By NYSESLAT Proficiency Level

14 Grade 6 ELL Performance By NYSESLAT Proficiency Level

15 Grade 7 ELL Performance By NYSESLAT Proficiency Level

16 Grade 8 ELL Performance By NYSESLAT Proficiency Level

17 ELL Performance By Number of Years of ESL Services Duratio nTotal < , , , , ,478 68,250 75,016 83,256 91,

18 ELL Performance By Need/Resource Category

19 ELL Performance By Major Home Language Group

20 Performance of ELLs With Disabilities

21 Performance of ELLs Without Disabilities

22 Performance of ELLs With Disabilities By Disability Type

23 Percentage of ELLs at Level 3-4 on by Disability Status

24 Percentage of ELLs scoring at Level 1 by Disability Status

25 Comparison of Non-ELL, ELL and Former ELL Students’ Performance on Grade 3-8 ELA Tests

26 Comparison of Non-ELL, ELL and Former ELL Students’ Performance: Percentage of Students at Level 3 & 4

27 Percentage of ELLs Scoring at Each of the NYSESLAT Proficiency Levels Passing/Failing the Regents English Exam in Grade 11

28 Percentage of ELLs Scoring at Each of the NYSESLAT Proficiency Levels Passing/Failing the Regents English Exam in Grade 12

29 NYSESLAT Scores Predict ELA Performance Well NYSESLAT scores explained 46% to 53% of the variance in ELA performance across grades in Multiple Regression Model Summary GradeCorrelation Coefficient Grade Grade Grade Grade Grade Grade *Dependent Variable: ELA scale score *Predictors: NYSESLAT Listening, Speaking, Reading, Writing scores

30 Summary  In 2008, the percentage of ELLs meeting the ELA standards increased from 18% in 2007 to 25% in 2008; the percentage of ELLs showing serious academic deficiencies in ELA deceased from 29% in 2007 to 18% in  NYSESLAT scores are good predictors of the ELA performance, accounting for 43% to 53% of the variance in ELA scale scores across grades. ELLs who scored at the proficient level on the NYSESLAT had a much better chance to meet the ELA standards than those who scored below proficient.

31 Summary  Former ELLs in grades 3 to 5 had comparable chance as English proficient students to meet the ELA standards, the percentage of former ELLs in grades 6 to 8 ranged from 43% to 62%.  ELLs in the elementary grades, with 3-6 years of services, and those from schools outside the Big 5 cities were more likely than their counterparts to score proficient on the Grade 3-8 ELA tests.  ELLs who scored at Level 1 on the 3-8 ELA tests were more likely to be in the middle school grades, with less than 3 years services.

32 SED Academic Language Analysis Research Project  Department is working with Charlene Rivera from the George Washington University: Center for Equity and Excellence in Education (GW-CEEE).  Purpose is to provide a description of academic language associated with the Living Environment Regents Exam and Commencement Level Core Curriculum.  Goal is to pilot an approach for describing the academic language demands.  GW-CEE staff involved in 2 year project with a selected NYS Analysis Team.

33 SED Academic Language Analysis Research Project Goals of NYS Analysis Team: 1.Describe and articulate the vocabulary, grammar, and academic language functions associated with NYS Living Environment course; 2.Articulate their expectations for the Living Environment Academic Language Framework; & 3.Review and make recommendations to improve the usability of the Content Standards Language Analysis Tool.

34 SED Academic Language Analysis Research Project Major Outcomes Include: 1.Draft list and categorization of academic vocabulary associated with classroom texts and Living Environment Regents Exam; 2.Draft list, examples, and explanations of grammatical structures that cause difficulty in comprehending Living Environment texts & assessment items; 3.Preliminary identification and descriptions of the most prevalent academic language functions for Core Curriculum; 4.Draft outline for the Living Environment Academic Language Framework; & 5.Design Professional development modules for teachers.

35 SED Academic Language Analysis Research Project  Academic Language: language used in the learning of academic subject matter in formal schooling context; aspects of language strongly associated with literacy and academic achievement, including specific academic terms or technical language, & speech registers related to each field of study (TESOL ESL Standards 4/97).  Academic Vocabulary: includes both specialized academic vocabulary, (e.g. organism, linear equation) and general academic vocabulary (e.g. produce, examine, & specify). Specialized academic vocabulary is associated with concepts of a discipline and general academic vocabulary cuts across academic disciplines.

36 SED Academic Language Analysis Research Project  Grammatical Structures: refers to the structure and arrangement of words in phrases and sentences with in written discourse. The following were selected for this project: compound & complex sentences; nominalization, long noun phrases, passive voice, and long or multiple prepositional phrases.  Academic Language Functions: language performances expected or realized in the doing of an academic tasks either through the production and/or comprehension of written or oral texts (Bailey et al, 2007).

37 SED Academic Language Analysis Research Project Next Steps: 1.Complete analysis, identification, and refinement of the framework; 2.Revise Content Standards Language Analysis Tool to provide user-friendly tool for the field; 3.Develop a draft Living Environment Academic Language Framework 4.Gather input/feedback; 5.Finalize Framework; & 6.Develop professional development modules.

38 New York State NCLB Title III Accountability: Proposed Revisions to Title III AMAOs David Abrams Office of Standards, Assessment & Reporting Dr. Pedro J. Ruiz Office of Bilingual Education and Foreign Language Studies

39 NCLB Title III Requirements  Standards for English Language Proficiency  Conduct an annual, standards-based assessment of English Language Proficiency: New York State English as a Second Language Achievement Test (NYSESLAT)  Define Annual Measurable Achievement Objectives (AMAOs) for increasing percentage of LEP/ELLs progressing toward and attaining English Language proficiency and for meeting academic achievement standards  Hold Local Educational Agencies (LEAs) and Consortia accountable for meeting the AMAOs Source: No Child Left Behind Act (NCLB) of 2001, Title III: Language Instruction for Limited English Proficient and Immigrant Students. Public Law , January 8, 2002.

40 Accountability Issues Regarding Title III Consortia  A Title III Consortium consists of a group of LEAs that join together as one eligible entity in order to qualify for the $10,000 minimum requirement and jointly apply to the State for a Title III subgrant.  Component districts that form one consortium are held accountable as one entity for meeting all three AMAOs.  AMAO determinations are made for the consortium as a whole by aggregating the data from the component districts to the consortium level (rather than the district level).

41 Title III Annual Measurable Achievement Objectives (AMAOs): Definitions  AMAO 1: Annual increases in the number or percentage of LEP/ELLs making progress in learning English  AMAO 2: Annual increases in the number or percentage of LEP/ELLs attaining English language proficiency  AMAO 3: Adequate yearly progress (AYP) for LEP/ELL subgroup in meeting grade-level academic achievement standards in English Language Arts (ELA) and mathematics Source: No Child Left Behind Act of 2001, Title III: Language Instruction for Limited English Proficient and Immigrant Students. Public Law , January 8, 2002.

42 NYS Title III AMAO 1: Making Progress  Proposed Revised Definition: 1) Advance one overall proficiency level on the NYSESLAT between two consecutive years; 2) Maintain the same proficiency level and make a total scale score gain of 43 points between two years; or 3) Score at Level 2 (Intermediate Level) or above on the NYSESLAT for ELLs with one data point only.  Unit of Accountability: Title III LEA and Consortia  ELLs Included in AMAO 1 Determination: All identified ELLs in the current school year

43 NYS Title III AMAO 2: Attaining English Proficiency  No proposed change from previous definition  Definition: Scoring at the proficient level (Level 4) on both Listening & Speaking (L/S) and Reading & Writing (R/W) modality combinations of the NYSESLAT  Unit of Accountability: Title III LEA and Consortia  ELLs Included in AMAO 2 Determination: All identified ELLs in the current school year

44 Title III AMAO 3: Making AYP  Definition: LEP/ELL subgroup must make AYP at the district level in meeting grade- level academic achievement standards in ELA and mathematics.  Unit of Accountability: Title III LEA and Consortia  Data source: Title I District AYP determination for the LEP/ELL subgroup; aggregated district data for Consortia.

45 Final Determination of Title III LEA AMAO Status For Title III accountability purposes, all Title III LEAs, including Consortia, must meet all three AMAO targets each year to be considered making AMAOs.

46 Accountability Count of Title III LEAs and Consortia in NYS  , N=184  , N=191  , N=208 (including 10 Consortia)

47 Current Title III AMAO Targets e.g. For , 60 percent of LEP/ELLs in each LEA must make AMAO 1 and 10 percent of LEP/ELLs must make AMAO 2.

48 Percent of Title III LEAs Meeting Current AMAOs Projected

49 Rationale for Revising Title III AMAOs  The current AMAO targets and increments were established in 2003 on estimated projections; longitudinal empirical data were not then available.  The current AMAO targets and annual increments were set with the assumption that 100 percent of LEP/ELLs must meet the AMAO 1 and AMAO 2 targets by as per Title I accountability requirements.

50 Rationale for Revising Title III AMAOs (Cont.)  Based on technical assistance from USED, that assumption has been revised. Although annual increases in AMAO targets are required, 100 percent attainment by is not.  The revised NYSESLAT (2005) differs from the original NYSESLAT in number of items and total score points available. The original AMAOs require revision due to change in test design.

51 Empirical Method for Setting Proposed Revised AMAO 1 Targets and Annual Growth Increments 1. Determine adequate annual scale score gain as 43 scale score points (40 th percentile gain) based on the matching data from 2006 and 2007 NYSESLAT administrations. 2. Determine individual ELL’s AMAO 1 status using the three criteria: 1) advancing one overall proficiency level; 2) making 43 total score gain; or 3) score at Level 2 and above on the current year NYSESLAT for ELLs with one data point. 3. Calculate the percentage of ELLs who made AMAO 1 in each district by dividing the sum of all students who made AMAO 1 by the total number of identified ELLs in the district for the current school year.

52 Empirical Method for Setting Proposed Revised AMAO 1 Targets and Annual Growth Increments (Cont.) 4. Rank order districts with 30 or more ELLs from low to high on percentage of students making AMAO Select the 25 th percentile LEA performance on AMAO 1 measure as the starting target for , and the 65 th percentile district performance as the ending point for Set annual growth increments as the equal interval between the starting and ending targets.

53 Comparison of Current and Proposed Revised AMAO 1 Targets and Annual Growth Increments

54 Percentage of Title III LEAs Meeting Proposed Revised AMAO 1 Targets (Percentage of LEAs projected to meet AMAO 1 based on simulations using matching NYSESLAT data from 2006 and 2007)

55 Implications of the Proposed Revised AMAO 1 Targets  In , 75 percent of the Title III LEAs will meet the AMAO 1 target and 25 percent will not.  The proposed starting AMAO 1 target, or 25 th percentile, for means that 58.9 percent of all LEP/ELLs in each Title III LEA are expected to make AMAO 1 by one of the three criteria.  The proposed ending AMAO 1 target, or 65 th percentile, for means that 69.6 percent of the LEP/ELLs in each Title III LEA must make AMAO 1.

56 Implications of the Proposed Revised AMAO 1 Targets (Cont.)  The ending target, or 65 th percentile, for also means that all Title III LEAs must perform at the same level as the top 35 percent of the LEAs performed in  As the AMAO 1 target increases by one percentage point each year, all LEAs are expected to increase the number and percent of ELLs making AMAO 1 each year. Without making the expected annual improvement, only 33 percent of the Title III LEAs would meet the AMAO 1 target by

57 Empirical Method for Setting Proposed Revised AMAO 2 Targets and Annual Growth Increments 1.The same empirical method used to revise the AMAO 1 targets and annual increments was used to revise the AMAO 2 targets. 2.Designate individual ELLs as meeting AMAO 2 if they scored at the proficient level (Level 4) on both the L/S and RW components of the 2007 NYSESLAT. 3.Calculate the percentage of students making AMAO 2 in each district by dividing the number of ELLs who made AMAO 2 by the total number of identified ELLs in the district in the current school year.

58 Empirical Method for Setting Proposed Revised AMAO 2 Targets and Annual Growth Increments (Cont.) 4. Rank the order of all districts with 30 or more LEP/ELLs from low to high on percentage of LEP/ELLs making AMAO Select the 25 th percentile of LEA’s performance on the AMAO 2 measure as the starting point for school year, and the 65 th percentile as the ending target for the Set annual growth increments as equal interval between the starting and ending targets.

59 Comparison of Current and Proposed Revised AMAO 2 Targets and Annual Growth Increments

60 Percentage of Title III LEAs Projected to Meet the Proposed Revised AMAO 2 Targets (Based on simulations using 2007 NYSESLAT Data)

61 Implications of the Proposed Revised AMAO 2 Targets  In , 75 percent of the Title III LEAs will meet the AMAO 2 target and 25 percent will not.  The proposed starting AMAO 2 target, or 25 th percentile, for means that 9.9 percent of all LEP/ELLs in each Title III LEA are expected to score at the proficient level on the NYSSELAT.  The proposed ending AMAO 2 target, or 65 th percentile, for means that 16.3 percent of the LEP/ELLs in each Title III LEA must score at the proficient level.

62 Implications of the Proposed Revised AMAO 2 Targets  The ending target for also means that all Title III LEAs must perform at the same level as the top 35 percent of the LEAs performed in  As the AMAO 2 target increases each year, all LEAs are expected to increase the number and percent of ELLs making AMAO 2. Without making the expected annual improvement, only 35 percent of the Title III LEAs would meet the AMAO 2 target by