1. Venessa A. Keesler, Ph.D. Bureau of Assessment and Accountability Michigan Department of Education Presentation to MASFPS Fall Directors’ Institute October 4, 2012

2. #1: NO ONE (and I do mean no one) likes accountability. #2: If you are the one who is holding people accountable, you are not going to be the most popular person in the room #3: See point number 1 Things I’ve learned about accountability since taking this job

3. Three myths; one reality Myth #1: To drive reform Myth #2: To create education policy Myth #3: Because we are gluttons for punishment Reality: Accountability metrics/systems are quantitative articulations of the core policy beliefs of the education system They help us measure our progress in meeting those core policy goals They are the measure, not the purpose or the goal So why do we do accountability?

4. A new era of accountability Switching from a purely criterion-based system to a normative system Criterion-based systems: Set average proficiency targets for schools. Normative system: identifies the “worst” or “best” or “lowest” or “highest” Accountability Landscape: 2012

5. Policy imperative for NCLB: all students CAN and SHOULD demonstrate proficiency  criterion- system with proficiency targets for all schools and subgroups 10 years later: our average achievement is increasing, but we still have students and schools lagging behind New policy imperative (ESEA Flex): we must target our lowest performing schools AND our lowest performing students more specifically and strategically Why the change?

6. “Never trust an average” Averages mask low performance Example: Proficiency target is 50% Sunshine School has 20 students, 10 of whom are proficient and 10 of whom are not. Sunshine School meets it’s target; hooray! BUT  those 10 not proficient students are possibly left behind The problem with average proficiency rates at a school or district level

7. Top to Bottom Ranking Priority Schools (bottom 5%) Focus Schools (largest achievement gaps) Reward Schools (high performing, high progress, beating the odds) Accountability Scorecard Proficiency targets for all schools Michigan’s Accountability System

8. We believe: Accountability only measures the core policy beliefs The data in the accountability metrics is useful and necessary for schools and districts to understand their performance Understanding where you are as a school/district is the first step toward moving forward Working smarter, not just harder Moving Beyond the Label

9. Three main components by subject: Achievement Improvement in achievement over time The largest achievement gap between two subgroups calculated based on the top scoring 30% of students versus the bottom scoring 30% of students Each component tells schools something about their overall performance and can be used for diagnostics Using the Top to Bottom Ranking

10. Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation Application  Some schools do not receive a ranking if they: Have too few FAY students Only have one year of data Who receives a ranking?

11. Reading and Mathematics: Grades 3-8 and 11 In grades 3-8, testing every year allows us to figure out student performance level change (our current “growth” metric) in reading and math Students can either significantly improve, improve, maintain, decline or significantly decline Writing: Grades 4 & 7 Science: Grades 5 & 8 Social Studies: Grades 6 & 9 Tested Grades and Subjects

12. A school must change by four or more grades in order to get a new code Example: A K-2 building becoming a K-6 building. New codes NOT granted when a school is reopened as a charter, for example If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder school There is no “phase reset” like there was in AYP If school population changed by 51%, could request a phase reset—still got AYP calculations, but sanctions delayed Under Priority/Focus interventions, would simply have a customized intervention. What about Reconfigured schools?

13. Quick Reference for Z-Scores

14. Z-scores are a standardized measure that helps you compare individual student (or school) data to the state average data (average scores across populations). Z-scores allow us to “level the playing field” across grade levels and subjects Each Z-score corresponds to a value in a normal distribution. A Z- Score will describe how much a value deviates from the mean. What do you need to know: Z-scores are used throughout the ranking to compare a school’s value on a certain component to the average value across all schools. Why do We Use Z Scores?

15. Z-scores are centered around zero Positive numbers mean the student or school is above the state average Negative numbers mean the student or school is below the state average What is a Z-Score? 0 1 23 -2-3 State Average Better than state average….…Worse than state average

16. Your school has a z-score of 1.5. You are better than the state average. Z-Score Examples 0 1 23 -2-3 State Average Better than state average….…Worse than state average Z-score of 1.5

17. Your school has a z-score of.2. You are better than the state average, but not by a lot. Z-Score Examples 0 1 23 -2-3 State Average Better than state average….…Worse than state average Z-score of 1.5 Z-score of 0.2

18. Your school has a z-score of -2.0. You are very far below state average. Z-Score Examples 0 1 23 -2-3 State Average Better than state average….…Worse than state average Z-score of 1.5 Z-score of 0.2Z-score of -2.0

19. How Is the Top to Bottom Ranking Calculated Two-Year Average Standardized Student Scale (Z) Score Improvement Metric (Performance Level Change OR Four Year Improvement Slope) Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Improvement Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score

20. How Is the Top to Bottom Ranking Calculated Two-Year Average Standardized Student Scale (Z) Score Improvement Metric (Performance Level Change OR Four Year Improvement Slope) Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Improvement Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score IMPORTANT PART RIGHT HERE!!!

21. Step #1: Take each student’s score on the test they took and compare that score to the statewide average for students who took that same test in the same grade and year This creates a student-level z-score for each student in each content area Compare MEAP to MEAP MEAP-Access to MEAP-Access MME to MME MI-Access Participation to Participation Supported Independence to Supported Independence Functional Independence to Functional Independence How do we get Standardized Scale Scores for Each Student?

22. Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school. Z-scores will have come from different tests, and compare students to statewide average for that grade, test, and subject But they can now be combined for the school Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score. Step #4: Define the top and bottom 30% subgroups, based on that rank ordering. What do we do with those standardized scores?

23. StudentTest TakenZ-score TommyMi-Access, Participation2.5 SallyMEAP2.0 MauraMI-Access, SI1.9 FredMEAP1.5 IchabodMEAP-Access1.0 FreudMEAP0.8 MaybelleMI-Access, FI0.7 DestinyMEAP0.5 HaroldMEAP-0.2 BickfordMI-Access, FI-0.5 TalledagaMEAP-Access-0.7 FrancineMEAP-1.2 JoeyMEAP-1.9 WilliamMEAP-2.2

24. StudentTest TakenZ-score TommyMi-Access, Participation2.5 SallyMEAP2.0 MauraMI-Access, SI1.9 FredMEAP1.5 IchabodMEAP-Access1.0 FreudMEAP0.8 MaybelleMI-Access, FI0.7 DestinyMEAP0.5 HaroldMEAP-0.2 BickfordMI-Access, FI-0.5 TalledagaMEAP-Access-0.7 FrancineMEAP-1.2 JoeyMEAP-1.9 WilliamMEAP-2.2 Average Z-score (average standardized student scale score): 0.28 (sum all z-scores, divide by 15)

25. StudentTest TakenZ-score TommyMi-Access, Participation2.5 SallyMEAP2.0 MauraMI-Access, SI1.9 FredMEAP1.5 IchabodMEAP-Access1.0 FreudMEAP0.8 MaybelleMI-Access, FI0.7 DestinyMEAP0.5 HaroldMEAP-0.2 BickfordMI-Access, FI-0.5 TalledagaMEAP-Access-0.7 FrancineMEAP-1.2 JoeyMEAP-1.9 WilliamMEAP-2.2 Top 30% Bottom 30%

26. What is important to show Schools? Two-Year Average Standardized Student Scale (Z) Score Two-Year Average Performance Level Change Index Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Performance Level Change Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score Step #1: Achievement How well did the school do in that subject? Positive number = better than average Near zero = average Negative number = worse than average

27. For grade 3-8 reading and mathematics What is important to show Schools? Two-Year Average Standardized Student Scale (Z) Score Improvement Score Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Improvement Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score Step #2: Improvement Is the school improving in that subject? Positive number = greater rate of improvement than average Near zero = average improvement Negative = slower rate of improvement than average; can also mean they are declining

28. For grade 3-8 reading and mathematics What is important to show Schools? Two-Year Average Standardized Student Scale (Z) Score Improvement Value Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Performance Level Change Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score Step #3: Achievement Gap Is the gap in that subject between top 30% and bottom 30%: (positive number) = smaller gap than average (negative number) = larger gap than average (near zero) = average gap

29. What’s the overall pattern? Low achievement? Declining achievement? Large gaps? Where are the actionable areas? Which subjects need the most attention? Is everyone doing poorly (small gap, low achievement) or are some students doing well and others falling behind (decent achievement, but large gap) Once they have looked at each component, Discuss:

30. For grade 3-8 reading and mathematics What is important to show Schools? Two-Year Average Standardized Student Scale (Z) Score Two-Year Average Performance Level Change Index Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Z-Score School Performance Level Change Z-Score School Achievement Gap Z-Score School Content Area Index 1/2 1/4 Content Index Z- score Focus Schools Reward Schools (for improvement)

31. My daughter’s elementary school K-4 building Go here to get this tool: www.mi.gov/ttb An example from data:

34. Accountability data does not tell ALL “Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.” Moving beyond the label only takes capacity at every level of the field Final Point

35. www.mi.gov/ttb Individual school lookup tool Diagnostic Worksheet www.mi.gov/priorityschools www.mi.gov/focusschools www.mi.gov/rewardschools Resources

36. Venessa A. Keesler keeslerv@michigan.govkeeslerv@michigan.gov (personal email) mde-accountability@michigan.gov 877-560-8378, option 6 Contact Information