1. C R E S S T / CU University of Colorado at Boulder National Center for Research on Evaluation, Standards, and Student Testing Measuring Adequate Yearly Progress Robert L. Linn CRESST Conference, UCLA, September 10-11, 2002

2. C R E S S T / CU Adequate Yearly Progress (AYP)  Central to the Accountability System of the No Child Left Behind (NCLB) Act of 2001  States required to define AYP for the State, school districts, and schools in a way that enables all children to meet the States student achievement standards by 2014

3. C R E S S T / CU Some Key Criteria for State’s AYP Definitions  Same high standards of academic achievement to all public elementary school and secondary school students in the State  Statistically valid and reliable  Results in continuous and substantial academic improvement for all students

4. C R E S S T / CU Some Key Criteria for State’s AYP Definitions (Continued)  Annual measurable AYP Mathematics Reading/language arts Substantial and continual progress  All students considered as a whole  Subgroups of Students

5. C R E S S T / CU Subgroups of Students Identified for AYP  Economically disadvantaged students  Major racial and ethnic groups  Students with disabilities  Students with limited English proficiency

6. C R E S S T / CU AYP Starting Point  Starting point defined in 2001-2002  The larger of either 1. The percentage of students in the lowest scoring subgroup who achieve the proficient level or higher, or 2. The percentage proficient or higher in the school at the 20 th percentile, based on enrollment, among all schools ranked by the percentage of students at the proficient level or higher.

7. C R E S S T / CU Annual Measurable Objectives  Separate for mathematics and reading/language arts  Straight line increase from starting points in 2221-2002 to 100% in 2013-2014  May combine across grade levels within a subject for school, district, or State

8. C R E S S T / CU Illustrations of AYP Targets  Starting points 52% proficient or above in reading/language arts and 40% proficient or above in mathematics  Annual gains required Reading/language arts: 4% = (100% - 52%)/12 Mathematics: 5% = (100% - 40%)/12  2003-2004 targets: Reading/language arts: 60% = 52% + 2(4%) Mathematics: 50% = 40% + 2(5%)

9. C R E S S T / CU Requirements for School to Meet AYP Target  Meet measurable objectives in reading/language arts and mathematics for all students considered as a whole  Meet objects for subgroups: Economically disadvantaged Major race and ethnic groups Students with disabilities Students with limited English proficiency

10. C R E S S T / CU “Safe Harbor” Exception  If one subgroup fall short of AYP target school can still avoid being placed in needs improvement category if 1. The percentage of students who score below the proficient level is decreased by at least 10% from the year before, and 2. There is improvement for that subgroup on other indicators

11. C R E S S T / CU Statistically Reliable Results Disaggregated reporting is required only if 1. Results are statistically reliable, and 2. The identity of individual students will not be revealed Statistical reliability requirement leads to need to establish minimum number of students for subgroup reporting

12. C R E S S T / CU Minimum Number of Students  There is no number below which results have zero statistical reliability and above which the statistical reliability is good  Statistical reliability increases as a function of the square root of the number of students  Relevant statistic is the standard error of the difference between percentages for two independent samples

13. C R E S S T / CU The Standard Error of the Difference Between Percentages for Two Independent Samples as a Function of the Number of Students in Each Sample When the Average Percentage is 50 Number of Students in Each Sample Standard Error of Difference in Percentages 1022.4 2015.8 3012.9 4011.2 5010.0

14. C R E S S T / CU Minimum Number of Students: Implications è The number of students needed each year to have a standard error as small as 10% is 5O.  Even with 50 students in a category each year about 1 time in 6 the percentage of students in the sample who are proficient would be no larger in year 2 than in year 1 event when the instruction had improved enough to increase the percentage proficient for an indefinitely large number of students by 10%.

15. C R E S S T / CU Minimum Number of Students: Implications (Continued) If the minimum number of students in a category were set at 50, the number of groups that would qualify for disaggregated reporting would be relatively small at most schools.

16. C R E S S T / CU Minimum Number of Students: Implications (Continued) Tradeoff between competing goals 1. More disaggregated reporting, and 2. Improved statistical reliability Compromise between competing goals may be best solution, e.g., minimum n of, say, 25, rather than 10 or 50

17. C R E S S T / CU Variability in Stringency of Progress Requirements for Different Schools: 2003- 2004 AYP Target is 50% Percentage of students proficient or above in 2001-2002  School A: 30%  School B: 45%  School C: 75% Needed Increases in percent proficient or above by 2003-2004  School A: 20%  School B: 5%  School C: could decline by up to 25%

18. C R E S S T / CU Scatterplot of Percent Proficient or Above for Schools on Colorado Grade 4 Reading Assessments in 1997 and 1999

19. C R E S S T / CU Cross Tabs of Schools Percent Proficient or Above Standing in 1997 vs. 1999 Column 1999 Row 1997 0 to 49.99%50 to 100%Total 0 to 39.99%126 (84.5%) 23 (15.5%) 148 40 to 100%90 (15.2% 504 (84.8) 594 Total215527742

20. C R E S S T / CU Index Scores  Proficient level vs. below only credits changes across proficient level cut score  Improvements in partially proficient region not recognized by dichotomous system  Index scores can credit improvements below proficient level and still be compatible with NCLB goals

21. C R E S S T / CU Definitions of AYP Based on Longitudinal Data Advantages:  Individual student growth used as basis of measuring progress  Past achievement taken into account without assumptions needed in successive groups approach  Only students who attend school a full year contribute to school accountability measure

22. C R E S S T / CU Definitions of AYP Based on Longitudinal Data (Continued) Challenges and disadvantages:  Matched student records more difficult to track for mobile students than less mobile students  Need to take account of students with only a single assessment result for the the year AYP is assessed  Quasi-longitudinal approach as an alternative

23. C R E S S T / CU Secretary Paige “The purpose of the statute, for both assessments and accountability, is to build on high quality accountability systems that States already have in place, not to require every State to start from scratch. Therefore, I want to assure you that the Department will work with States so that they nave the tools they need to implement definitions of AYP that meet the requirements of the statute and maintain high standards” (Secretary Paige, www.ed.gov/News/Letters/020724.html). www.ed.gov/News/Letters/020724.html