Evaluation, Government Performance and Results Act and Performance Reports FY06 ERCM Initial Grantee Meeting December 7, 2006, San Antonio, TX U.S. Department of Education, Office of Safe and Drug-Free Schools 400 Maryland Avenue, SW / Washington, DC Tara Hill Michelle Sinkgraven Sara Strizzi Program Analysts, U.S. Department of Education, Office of Safe and Drug-Free Schools

2 Session Objectives Review the importance of evaluation Discuss key evaluation components Discuss creating performance measures and an evaluation plan Highlight the Emergency Response and Crisis Management (ERCM) Government Performance and Results Act (GPRA) measures for FY 2006 grantees Explain how to collect data and report on the GPRA measures on the ED 524B form

3 Why Evaluate ERCM Projects? Because evaluation…. Is a way to gather school-based and community data to help improve emergency management plans Uncovers new information or consequences that were not anticipated Involves multiple stakeholders in the process and creates buy-in for emergency management planning

4 What Evaluation Does… Determines if a grant project is accomplishing the performance objectives Supports decision-making in the district and community Provides data for communicating to stakeholders the benefits and purpose of the emergency management plan Builds a knowledge base about what does and does not work in an emergency management plan Helps the U.S. Department of Education report on progress

5 What Evaluation Does NOT Do… Attempt to judge a program, project, or school system subjectively Aim to represent a false picture of program success for administrators or funders

6 Creating an ERCM Evaluation Plan: Key Components Key questions to consider Who will conduct your evaluation? Who are the key stakeholders in your grant project and your overall emergency management efforts? Have you captured the appropriate data at the beginning of your project? What are your key project objectives?

7 Reviewing Project Objectives ERCM grant objectives Project-specific objectives and performance measures Drawn from needs assessment As reflected in grant applications Based on individual school and district issues Government Performance and Results Act measures

8 Project-Specific Objectives & Performance Measures Project Objective: A specific, measurable statement about what the project will achieve. Hint: Start the objective with the word "To" followed by a verb. For example, "To increase the number of bus drivers trained in CPR from 5 bus drivers to 25 bus drivers.

9 Project-Specific Objectives & Performance Measures (continued) Performance Measure: A value or characteristic that can be used to determine the extent to which the objective has been achieved. For example, the # of bus drivers trained in CPR between September 1, 2006-March 31, 2008.

10 Government Performance and Results Act (GPRA) What is GPRA? Purpose of the GPRA ED's response to GPRA GPRA measures for the ERCM Program Measure 1 Measure 2 Measure 3

11 What is GPRA? The Government Performance and Results Act of 1993 requires all federal agencies to manage activities with attention to outcomes. Agencies must: State intended accomplishments Identify the resources required Periodically report to Congress

12 Purpose of the GPRA Continuous monitoring of program outcomes results in: Improved accountability for expenditures using public funds Informed Congressional decision making through use of objective information Government focus on results

13 U.S. Department of Education's Response to GPRA Develop a strategic plan for Plan reflects the Department's goals and priorities Department goals: Goal 1: Create a Culture of Achievement Goal 2: Improve Student Achievement Goal 3: Develop Safe Schools and Strong Character Goal 4: Transform Education into an Evidence-based Field Goal 5: Enhance the Quality of and Access to Postsecondary and Adult Education Goal 6: Establish Management Excellence

14 What are the ERCM GPRA Measures? Measure 1: Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan. Measure 2: Demonstration of improved response time and quality of response to practice drills and simulated crises. Measure 3: A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

15 Measure 1 Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan.

16 Measure 1: Increased Number of Hazards Definitions: Hazards: Broadly defined as potential threats to the school / school district whose date or time of occurrence cannot be predicted. Examples include: Terrorist attack Tornado Earthquake Chemical spill Unlawful entry by unlawful person Criminal activity Addressed: Written plan is in place that responds to hazards Baseline plan: Emergency plan in place prior to grant award date

17 Data Needed for Measure 1 Number of hazards addressed by the improved school emergency plan Number of hazards addressed by the baseline plan Measure can include information on number of hazards at the district level and at the school level

18 Completing the 524B Sample scenario: ABC School District had a basic emergency plan in place prior to the grant that addressed fires, tornadoes, and unlawful entry. At the end of the grant, the district addressed three additional hazards: terrorist attack, bomb threat, and active shooter.

19 Completing the 524B Box 1.a. Performance Measure: Number of hazards addressed by the improved school emergency response plan as compared to the baseline plan. Measure Type: GPRA Quantitative Data: Step 1: Enter 999 in all data fields Step 2: Provide numerical answers to each of the following in the "Explanation of Progress" section:  Number of hazards addressed in original (baseline) plan  Number of hazards addressed in final plan  Number of additional hazards addressed by final plan  Formula: # of hazards in final plan - # of hazards in baseline plan

20 Completing the 524B Form Additional information to include in the "Explanation of Progress" section: Provide a narrative that explains what the numbers mean and how they were calculated. Be specific and list additional hazards that have been addressed. Describe any barriers that prevented you from meeting your goal, if applicable.

21 FAQs Can I count facilities improvements as hazards that my school/school district has addressed? What if our district did not include funds to address hazards that were uncovered in our vulnerability assessment? Do I need to report school-by-school data on the 524B? My application is for a consortium of school districts. How should I report data for this measure—do I aggregate or report on a district- by-district basis?

22 Measure 2 Demonstration of improved response time and quality of response to practice drills and simulated crises.

23 Measure 2: Improved Response Time and Quality of Response Definitions: Response Time: The number of minutes and seconds it takes to respond to a practice drill or simulated crisis. The type of response depends on the type of drill staged. Drills include, but are not limited to: Fire drill Lockdown Community-wide emergency response drill Hazardous weather drill Active shooter drill Shelter-in-place drill Evacuation

24 Measure 2: Improved Response Time and Quality of Response Definitions, continued: Demonstration of improved response time: A quantifiable decrease in the amount of time it took to respond to a practice drill or simulated crisis that was staged at the beginning of the performance period and the end of the performance period. Quality of response: To respond to this portion of the measure, attach a copy of the after-action report for each drill and clearly state what has been done to make improvements in quality.

25 Data Needed for Measure 2 Response time: Baseline (Drill #1): Response time to drill conducted at the beginning of the performance period or before the grant was awarded. Drill #2: Response time (to same type of drill) at the end of the performance period. Quality of response: A thorough narrative description of the drills documenting lessons learned and a plan for addressing any shortcomings. Attach an after-action report for each drill.

26 Completing the 524B Sample scenario: ABC School District conducted a tornado drill. An after-action report highlighted confusion about designated safe areas. In response to the report, students were briefed and shelter-in-place signs were hung around school. A few months later, a second tornado drill was conducted. The after-action report attributed the 3 minute, 16 second reduced response time to the changes implemented from the lessons learned in the first (baseline) drill.

27 Completing the 524B Box 1.a. Performance Measure: Demonstration of improved response time and quality of response to practice drills and simulated crises. Measure Type: GPRA Quantitative Data: Step 1: Enter 999 in all data fields Step 2: Provide numerical answers to each of the following in the "Explanation of Progress" section:  Actual time for drill #1 (baseline drill)  Goal (target) time for drill #2  Actual time for drill #2  Difference between drill time #1 and drill time #2

28 Completing the 524B Additional information to include in the "Explanation of Progress" section: Provide a narrative for the data provided, explaining what the numbers mean and how they were calculated Be specific, include information about what went right and wrong and how times were calculated Provide the rationale for the drill #2 goal/target and when it was established (I.e. upon completion of drill #1) Describe any barriers that prevented you from meeting the goal for drill #2, if applicable Include a qualitative analysis of each drill, such as an after- action report

29 FAQs Do I need to compare similar types of drills for reporting purposes? Does my first (baseline) drill need to occur during the grant performance period or can we use data from a prior drill or simulation? What if our grant application specifies running a variety of drills only once? Can I report on community-wide crisis drills?

30 Measure 3 A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

31 Measure 3: Plan for and Commitment to Sustainability and Continuous Improvement Definitions: Plans: The emergency management plan developed under the ERCM grant.

32 Data Needed for Measure 3 Provide the answer to the following question: "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?" A simple "yes" or "no" is not sufficient. A copy of the plan should be provided with your final report. In addition, an objective analysis of whether or not the plan meets this measure should be provided.

33 Completing the 524B Box 1.a. Performance Measure: A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance. Measure Type: GPRA Quantitative Data: Target, Raw Number: YES Actual, Raw Number: YES, if your district has met this standard. If not, NO.

34 Completing the 524B Explanation of Progress: Provide a detailed answer to the question, "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?" Provide proof of your plan. This could include, but is not limited to, the following:  Copies of updated Partner Agreements  Timelines/agenda for upcoming community partner meetings/drills after the grant period is over  Your district's completed emergency plan

35 FAQs Do I need to submit a copy of my district's completed emergency plan with my final report? What are some ways a district might demonstrate sustainability? Do I need to submit signed copies of all Memorandums of Understanding (MOU) at the conclusion of the grant period?

36 QUESTIONS