Commissioner’s Performance-Based Accountability Task Force: A Proposal for a Multi-level System Deb Wiswell & Scott Marion February 19, 2010

Key Decisions  We need to decide by the end of today’s meeting… 1. Should we continue to pursue this “shared” two-level system as the framework for the performance system? 2. Should we require all schools to participate in both levels of the performance system?  If “no” to #2, which schools should be required to participate in either Level 1 or Level 2? 3. Should we phase-in Level 2 after Level 1 is established?  If so, how long should before Level 2 gets phased in? Commissioner's Performance Based Accountability System Task Force. 2/19/10 2

A Multi-Level Performance Accountability System for NH Commissioner's Performance Based Accountability System Task Force. 2/19/10  Level One  A very limited set of common (across the state) indicators and metrics  Applied consistently across all schools in the state  Focused on unarguable outcomes, e.g., NECAP, graduation rate, postsecondary assessments, attendance  Level Two  Locally determined goals, targets, and indicators 3

Level One: Indicators  K-8  Inclusion Factors  Status Measures  Growth Measures  Gap Analyses  High School  Inclusion Factors  Status Measures  Gap Analyses  Postsecondary Indicators Commissioner's Performance Based Accountability System Task Force. 2/19/10 4

Level 1: K-8— Inclusion  Test Participation—required level of participation (e.g., 95%) to meet performance requirements  Weighted average across all state tests given in the school  School must average 95% participation to meet adequacy standard, otherwise the school will be deemed as not providing an opportunity for an adequate education  Do we agree with this stance?  Attendance/Truancy—percent of students absent fewer than 15 days  Actual percentage will get folded into the overall adequacy determination—recommend that this indicator count no more than 5-10% of total weight Commissioner's Performance Based Accountability System Task Force. 2/19/10 5

Level 1: K-8— Status  NECAP index scores (status)  Reading  Math  Science  Writing  Average index score—weighted by the number of tests in the school (i.e., writing and science will count less than reading and math)—will be the status score  Should be weighted at least 20% and no more than 30% Commissioner's Performance Based Accountability System Task Force. 2/19/10 6

Level 1: K-8— Growth  NECAP results for grades 4-8  Reading  Math  We will use the Student Growth Percentile methodology that combines both criterion and normative information to evaluate the degree to which students at a school are making adequate growth  Suggest that growth count at least twice as much as status Commissioner's Performance Based Accountability System Task Force. 2/19/10 7

Level 1: K-8— Gaps  Status Gaps—effect size differences-NECAP scale scores)  Reading, Mathematics. Science, Writing  Weighted average (by # students tested)  Key gap comparisons  FRL with those not receiving FRL  Whites and “non-white”  Special education and non-special education  Growth gaps  Reading and math only  Groups need to be of appropriate size (FRL vs non-FRL)  Specific details being worked out by AYP task force  Suggest that growth gaps count at least half or more of the “Gap weight” and that Gaps count at least 25% of the system Commissioner's Performance Based Accountability System Task Force. 2/19/10 8

Level 1: K-8 Weighting Summary  Participation—must pass  Inclusion-no more than 10%  Status—20-30%  Growth—40-60%  Gaps—25-30%  Suggestion:  Attendance/Inclusion=10%  Status=20%  Growth=40%  Gaps=30% Commissioner's Performance Based Accountability System Task Force. 2/19/10 9

Level 1: HS— Inclusion  Test Participation—required level of participation (e.g., 95%) to meet performance requirements  Weighted average across all state tests given in the school  School must average 95% participation to meet adequacy standard, otherwise the school will be deemed as not providing an opportunity for an adequate education  Do we agree with this stance?  Attendance/Truancy—percent of students absent fewer than 15 days  Actual percentage will get folded into the overall adequacy determination—recommend that this indicator count no more than 5% of total weight Commissioner's Performance Based Accountability System Task Force. 2/19/10 10

Level 1: HS— Status  NECAP index scores (status)  Reading  Math  Science  Writing  Average index score—weighted by the number of tests in the school (i.e., writing and science will count less than reading and math)—will be the status score  Should be weighted at least 20% and no more than 30% Commissioner's Performance Based Accountability System Task Force. 2/19/10 11

Level 1: HS--Postsecondary Readiness  Graduation rate—Using the new federally required graduation rate  Dropout rate—Using NH method for calculating dropout  Other postsecondary measures???  Are we willing to require census administration of SAT/ACT?  Suggest weighting graduation rate 3 times as much as dropout rate and together postsecondary readiness should count at least 40% of high school rating Commissioner's Performance Based Accountability System Task Force. 2/19/10 12

Level 1: HS— Gaps  Status Gaps—effect size differences-NECAP scale scores)  Reading, Mathematics. Science, Writing  Weighted average (by # students tested)  Key gap comparisons  FRL with those not receiving FRL  Whites and “non-white”  Special education and non-special education  Postsecondary gaps  Graduation rate only or both graduation and dropout rates?  We need to decide upon the appropriate statistic (simply finding the differences in % might not be appropriate)  Suggest that postsecondary gaps count at least half of the “Gap weight” and that Gaps count at least 25% of the HS system Commissioner's Performance Based Accountability System Task Force. 2/19/10 13

Level 1: HS Weighting Summary  Participation—must pass  Inclusion-no more than 5%  Status—20-30%  Postsecondary—40-50%  Gaps—25-30%  Suggestion:  Attendance/Inclusion=5%  Status=20%  Postsecondary=50%  Gaps=25% Commissioner's Performance Based Accountability System Task Force. 2/19/10 14

Level 1: Putting it together  Aggregate indicators within major classifications, e.g., status, growth, postsecondary  Determine “adequate” for each class of indicators  This allows for more informative feedback  Aggregate these “adequate” determinations across major classifications using a weighting scheme as described on previous slides  Use a profile or related approach to determine overall adequacy for Level One Commissioner's Performance Based Accountability System Task Force. 2/19/10 15

Level 2: Locally-determined system Commissioner's Performance Based Accountability System Task Force. 2/19/10  A very limited set (e.g., 2-5) of district/school-determined goals, targets, and indicators  For example, “increase the % of students achieving their NWEA growth targets to 90% by 2015”  The school results related to such goals and targets would count in the performance-based accountability system  The following set of slides describes a simple proposal based on our previous discussions 16

Level 2: Phase In  We suggest phasing in Level 2 over 2-3 years.  Year 1—Level 1 only  Year 2—Level 2, only 1 goal required  Year 3—Level 2, 2-5 goals  The following slides focus on Year 2 for example purposes only Commissioner's Performance Based Accountability System Task Force. 2/19/10 17

Level 2: Establishing the Goals & Targets  Schools will be required to identify at least one goal for which they want to be held accountable  Goals must relate to the opportunity for an adequate education  Goals must be tied to identifiable measurement approaches  Goals could (should) stretch over several years, but yearly measureable targets must be established  Committee could require that the goals could be:  Academic only  A mix of academic, social, emotional, and physical, but requiring at least one academic  Any goals that the school chooses that can be measured  Recommend that for the phase-in, the single goal should be academic Commissioner's Performance Based Accountability System Task Force. 2/19/10 18

Level 2: Justifying the Goal  The school must provide a rationale for selecting the goal  This rationale should be constructed in terms of a “theory of action,” i.e., a logical flow that describes how focusing (and measuring) this particular goal and associated targets will lead to the ultimate goal of improving student achievement  For example….  Goal: Increase the number/percent of students taking and passing (earning a 3) AP exams  Target: In 2010, we will increase by 5% the percentage of seniors who have taken at least one AP course with no drop in the percentage of students earning a 3 or better Commissioner's Performance Based Accountability System Task Force. 2/19/10 19

Level 2: For example (continued)…  Rationale: Our school team has determined that one way to ensure the postsecondary preparation of students leaving our school is to increase the rigor of our HS courses. The AP program provides a vehicle for accomplishing and measuring our goals  Theory of action:  The focus on AP, will lead our school to ensure that our prerequisite course are aligned with AP expectations.  This, in turn, will provide students with more preparation and interest in the particular subjects to increase their likelihood of enrolling in AP.  Counselors ensure that essentially all students are steered toward this academic pathway  The district will support this goal by creating an “AP fund” to pay for tests for students unable to pay Commissioner's Performance Based Accountability System Task Force. 2/19/10 20

Level 2: Approving the goals  The school will be required to have the goals approved by the district leadership and the board  The school will be required to publish the goals, targets, indicators, and results  NH DOE will have to approve the goals (or just the process by which the goals were established?) and the plans for determining targets and measures  Should NH DOE be the entity to approve:  The goals?  The process for setting the goals?  The appropriateness of the measures?  The appropriateness (rigor) of the yearly targets? Commissioner's Performance Based Accountability System Task Force. 2/19/10 21

Level 2: Measuring the goals and targets  NH DOE Guidance and Review May Ask…  Targets  Are the targets ambitious enough?  How were the targets established?  What data were used?  What is the rationale to suggest that the targets are ambitious?  Was the target-setting process appropriate and inclusive?  Measures  Are the measures/indicators appropriate for the purpose?  How/why were these measures chosen?  Are the measures of appropriate technical quality?  E.g., is the measurement error larger than the yearly target? Commissioner's Performance Based Accountability System Task Force. 2/19/10 22

Level 2: Evaluating the results  The local school, with district sign-off will have to document whether or not the school has met or exceeded their goal(s)  The school will then have to provide a summary of this evaluation to NH DOE  We will design a process for simplifying this summary  NH DOE will review the summary and determine if it provides credible evidence that the school met/did not meet its goal(s)  We will develop publically shared criteria for these judgments  NH DOE will respond to the school that it agrees with the school’s judgment, disagrees, or needs more information Commissioner's Performance Based Accountability System Task Force. 2/19/10 23

Integrating Levels 1 & 2 Commissioner's Performance Based Accountability System Task Force. 2/19/10  Combining evaluations from Level One and Two?  If both levels “point” in the same direction—easy decision  If Level 2 points in an opposite direction from Level 1, how should the school be evaluated?  Does Level 2 have the credibility/validity/etc to “overrule” Level 1?  Should Level 2 be used as “extra credit” or “tie-breaker”?  Should ratings from both Levels somehow be averaged?  Level Two can be a powerful vehicle and incentive for having schools engage in systematic school improvement efforts  Other considerations, concerns, questions, etc? 24

Decision Time 1. Should we continue to pursue this “shared” two-level system as the framework for the performance system? 2. Should we require all schools to participate in both levels of the performance system?  If “no” to #2, which schools should be required to participate in either Level 1 or Level 2?  If yes, do we have to worry about unfunded mandate issues? 3. Should we phase-in Level 2 after Level 1 is established?  If so, how long should before Level 2 gets phased in?  Remember, if we do not use something like Level 2, the entire decision rests on Level One (unlikely to be able to require additional data collection) Commissioner's Performance Based Accountability System Task Force. 2/19/10 25