325D: PREPARATION OF LEADERSHIP PERSONNEL Webinar on Project Objective and Performance Measures for the Annual Performance Report for Continuation Funding Office of Special Education Programs U.S. Department of Education 1

ANNUAL GRANT PERFORMANCE REPORT Provides an annual report of your performance on (a) meeting the approved objectives of the project, and (b) financial management of federal funds Required for all active grants, including those in no cost extension (NCE) Used by OSEP to determine if substantial progress has been made in order to grant continued funding or a NCE 2

OVERVIEW 3 Recognize strong project objectives that can be associated with high-quality performance measures Develop relevant, measurable, outcome-oriented performance measures that maximize the potential for meaningful data Project Objectives Performance Measures

WHY IS THIS IMPORTANT? 4 Make it easier for you to measure your progress Establish targets (both short-term/annual and long- term) Allow you to report progress easily and quantitatively HIGH-QUALITY OBJECTIVES & MEASURES: Allow OSEP staff to gather evidence of program effectiveness Taken from the Center for Evaluation and Education Policy (CEEP) presentation

GOAL - OBJECTIVES - MEASURES 5 Program Goal Project Objectives: What your project is doing to support the overall program goal (Found in your application) Performance Measures: How you measure your progress toward meeting your objectives (Program/ GPRA, Project) Taken from the Center for Evaluation and Education Policy (CEEP) presentation

PROJECT OBJECTIVES 6 Preferred format for objectives: Begin the objective with a verb and define a desired outcome or condition WHAT ARE YOU TRYING TO ACCOMPLISH?

HIGH QUALITY PROJECT OBJECTIVES 7 How relevant is the project objective to the overall goal of the program and/or the goal of your project? How focused is the project objective? How applicable is the project objective to the specific activities that are being conducted through your particular project? Are there concepts in the project objective that lend themselves to measurement? If so, is measurement feasible? RELEVANCEAPPLICABILITY FOCUSMEASURABILITY Taken from the Center for Evaluation and Education Policy (CEEP) presentation

PROJECT OBJECTIVES: EXAMPLES 8 Recruit and retain high-quality scholars Implement a high-quality personnel preparation program that will promote competencies in using evidence-based practices Prepare high-quality personnel with the knowledge and skills to use evidence-based instruction and intervention practices to improve the outcomes of children with disabilities Ensure that scholars are knowledgeable of and can implement evidence-based practices to improve outcomes for children with disabilities

PERFORMANCE MEASURES How are you measuring your progress in meeting your objectives? 9

PERFORMANCE MEASURES ARE: 10 How will progress be assessed? Measurable indicators used to determine how well objectives are being met. How much progress will constitute success? How will it be known if an objective or part of an objective has been achieved? Taken from the Center for Evaluation and Education Policy (CEEP) presentation

PERFORMANCE MEASURES 11 Performance Measure 1a Project Objective 1 Performance Measure 1b Performance Measure 1c Taken from the Center for Evaluation and Education Policy (CEEP) presentation

TYPES OF PERFORMANCE MEASURES 12 Program Measures established by OSEP for the Personnel Development program. These include measures established for reporting to Congress under the Government Performance and Results Act of 1993.

Personnel Development PROGRAM Performance Measures The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. (not for 325D): The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion, and who are fully qualified under IDEA. The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. (not for 325D): The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years, and who are fully qualified under IDEA. The Federal cost per fully qualified degree/certification recipient.

Additional Personnel Development Performance Measures 14 In addition, the Department will be gathering information on the following outcome measures: □The number and percentage of degree/certification recipients who are employed in high-need districts; □The number and percentage of degree/certification recipients who are employed in the field of special education for at least two years; and □The number and percentage of degree/certification recipients who are rated as effective by their employers.

PROGRAM PERFORMANCE MEASURES 15 □Grantees should propose a project performance measure that addresses each program performance measure reported for the 325D program. The focus of the project measure must be clearly aligned with the program measure. □ 325D grantees must develop project measures that align with program measures 1, 2, 3, 4 and 7. Program measures 5 and 6 do not apply to 325D.

PROGRAM PERFORMANCE MEASURE 16 Example of related project performance measure: Annually, one selected course syllabus will achieve a score of 90 or higher on its integration of evidence-based practice as measured by an expert panel using a rubric designed by the project. #1. The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. (Process Measure)

PROGRAM PERFORMANCE MEASURE (2) 17 Example of related project performance measure: By year 5 of the project, 90% of scholars will demonstrate knowledge and skills in research competencies at an “acceptable” level or higher using a rubric designed by the project. #2. The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. (Outcome Measure)

PROGRAM PERFORMANCE MEASURE (3) 18 Example of related project performance measure: As measured annually, no scholars (0%) will have exited the program due to poor academic or field- based performance. #3. The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. (Process Measure)

PROGRAM PERFORMANCE MEASURE (4) 19 Example of related project performance measure: By the end of Year 5 of the project, 90% of scholars will be working in the area for which they were trained following program completion. #4. The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. (Intermediate Outcome Measure)

PROGRAM PERFORMANCE MEASURE (5) 20 Example of related project performance measure: The Federal cost is measured in Year 5 by dividing the total grant award (e.g., $250,000 x 5) by the number of scholars. #7. The Federal cost per fully qualified degree/certification recipient. (Efficiency Measure)

TYPES OF PERFORMANCE MEASURES (6) 21 □Measures that the grantee establishes to meet their project objectives □Project performance measures can address both the process of working towards an objective and the outcome related to meeting the objective □Ensure a mix of both process and outcome measures Project Measures

HIGH-QUALITY PERFORMANCE MEASURES 22 High-quality performance measures show: □What will change. □How much change you expect. □Who will achieve the change. □When the change will take place.

PROJECT PERFORMANCE MEASURE EXAMPLES 23 Process Measure (e.g.,) Project staff (who) will implement 4 (how much) targeted recruitment activities (what ) during the first year of the grant (when). Outcome measure (e.g.,) At the end of Year 2 (when), inquiries about the program (what) from potential candidates from underrepresented groups (who) will increase 20% (how much) from previous recruitment efforts during the first year of the grant. Outcome measure (e.g.,) At the end of each semester (when), 100% of scholars (who) will rate their mentoring support (what) as 5 or higher on a 7-point scale (how much) using a survey developed for the project. Process Measure (e.g.,) At the end of the project (when), 100% of students (who) will produce at minimum of 3 scholarly products (e.g., manuscripts, presentations, preliminary exams) (what) that are rated by a panel of 3 faculty as showing “strong and substantial application” (how much) of evidence based practice in language and literacy.

PROJECT PERFORMANCE MEASURE EXAMPLES (2) 24 Outcome measure (e.g.,) At the end of each course taught (when), 90% of scholars (who) will have their teaching competence of evidence based practices (what) rated by 2 faculty as 6 or higher (how much) on a 7-point scale using a standard instrument developed by the project. Outcome measure (e.g.,) By the end of Year 3 of their program (when), 100% of scholars (who) will serve on at least one (how much) MA or undergraduate research committee (what). Outcome measure (e.g.,) By the end of their program (when), 100% of scholars (who) will attain at least 90% (how much) of required competencies (what) as measured by faculty evaluations of a web-based portfolio.

COMMON PROBLEMS 25 □Activities are NOT performance measures □If the best response is “Yes, we did that,” it is likely an activity (not a performance measure) □Examples of activities: □Scholars complete required coursework □Hold an Advisory Board meeting quarterly □Establish a mentoring program □Evaluate the project □Performance measures need to be measurable □Examples with measurement problems: □Maintain meaningful collaborative partnerships □Provide mentors to socialize scholars to the program □Provide academic support in areas of concern to scholars Taken from the Center for Evaluation and Education Policy (CEEP) presentation

Need additional information on writing performance measures? 26 Please contact your project officer for additional resources on developing performance measures and logic models

SUMMARY 27 □Projects should have a few clear objectives that explain what the project is doing to support their overall goal(s) □Each objective should have a few, specific performance measures to demonstrate how progress toward meeting the objective is being measured

COMPLETING ED GRANT PERFORMANCE REPORT 28 Review the recorded webinar on Completing the ED Grant Performance Report (ED 524B) for the Annual Performance Report for Continuation Funding

Contact your OSEP Project Officer with any questions! 29