325D Preparation of Leadership Personnel Webinar on the Annual Performance Report for Continuation Funding Presenter: Patricia Gonzalez, Ph.D. Office of Special Education Programs US Department of Education 202-245-7355; patricia.gonzalez@ed.gov

Annual Grant Performance Report An annual report of your performance in meeting the approved objectives of the project Required for all active grants, including those in no cost extension (NCE) OSEP reviews the report to determine if substantial progress has been made in order to grant continued funding or an NCE

Overview Recognize strong project objectives that can be associated with high quality performance measures Develop relevant, measurable, outcome- oriented performance measures that maximize the potential for meaningful data Complete the ED Grant Performance Report (ED 524B)

Why Is This Important? High quality objectives and measures … Make it easier for you to measure your progress Allow you to report progress easily and quantitatively Establish targets (both short-term/annual & long- term) Allow OSEP staff to gather evidence of program effectiveness Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Goal – Objectives - Measures Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

What are you trying to accomplish? Preferred format for objectives: Project Objectives What are you trying to accomplish? Preferred format for objectives: Begin the objective with a verb and define a desired outcome or condition Good objectives facilitate all aspects of program implementation and evaluation

High Quality Project Objectives Relevance How relevant is the project objective to the overall goal of the program and/or the goal of your project? Applicability How applicable is the project objective to the specific activities that are being conducted through your particular project? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

High Quality Project Objectives Focus How focused is the project objective? Measurability Are there concepts in the project objective that lend themselves to measurement? If so, is measurement feasible? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Project Objectives -- Examples Recruit high quality scholars Provide ongoing support to retain scholars in the program Provide learning opportunities that result in scholars attaining jobs as faculty in higher education (e.g., knowledge of evidence-based practice, writing/research, teaching, and cultural competence Develop and offer courses in assistive technology that are evidence-based

How are you measuring your progress in meeting your objectives? Performance Measures How are you measuring your progress in meeting your objectives?

Performance Measures Measurable indicators used to determine how well objectives are being met. How will progress be assessed? How much progress will constitute success? How will it be known if an objective or part of an objective has been achieved? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Performance Measures Taken from the Center for Evaluation & Educational Policy (CEEP) at Indiana University

Types of Performance Measures Program Measures established by OSEP for the Personnel Development program. These include measures established for reporting to Congress under the Government Performance and Results Act of 1993.

Personnel Development PROGRAM Performance Measures #1: The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. #2: The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. #3: The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. #4: The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. #5 (not for 325D): The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion and who are fully qualified under IDEA. #6 (not for 325D): The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years and who are fully qualified under IDEA. #7: The Federal cost per fully qualified degree/certification recipient.

Program Performance Measures Grantees should propose a project performance measure that addresses each program performance measure reported for the 325D program. The focus of the project measure must be the same as and clearly aligned with the program measure. 325D grantees must develop project measures that align with program measures 1, 2, 3, 4 and 7. Measures 5 and 6 do not apply to 325D. See the examples below.

Program Performance Measure #1. The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. (Process Measure) Example: Annually, one selected course syllabus will achieve a score of 90 or higher on its integration of evidence- based practice as measured by an expert panel using a rubric designed by the project. 16

Program Performance Measure #2. The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. (Outcome Measure) Example: By year 5 of the project, 90% of scholars will demonstrate knowledge and skills in all research competencies at an “acceptable” level or higher using a rubric designed by the project 17

Program Performance Measure #3. The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. (Process Measure) Example: As measured annually, no scholars (0%) will have exited the program due to poor academic or field-based performances. 18

Program Performance Measure #4. The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. (Intermediate Outcome Measure) Example: By the end of year 5 of the project, 90% of scholars will be working in the area for which they were trained following program completion. 19

Program Performance Measure #7. The Federal cost per fully qualified degree/certification recipient. (Efficiency Measure) Example: The Federal cost is measured in Year 5 by dividing the total grant award (e.g., $250,000 x 5) by the number of scholars counted for Performance Measure #2. 20

Types of Performance Measures Project Measures that the grantee establishes to meet their project objectives Project performance measures can address both the process of working towards an objective and the outcome related to meeting the objective Ensure a mix of both process and outcome measures Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

High Quality Performance Measures High quality performance measures show What will change How much change you expect Who will achieve the change When the change will take place Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Project Performance Measure Examples Process measure (e.g.) - Project staff (who) will implement 4 (how much) targeted recruitment activities (what ) during the first year of the grant (when)

Project Performance Measure Examples Outcome measure (e.g.) – Inquiries from potential candidates (who) about the program (what) will increase 20% from previous recruitment efforts (how much) during the first year of the grant (when)

Project Performance Measure Examples Outcome measure (e.g.) – At the end of each semester (when), 100% of scholars (who) will rate their academic/psycho- social support (what) as 5 or higher on a 7- point scale (how much) using a standard instrument developed for the project

Project Performance Measure Examples Outcome measure (e.g.) – At the end of the project (when), 100% of students (who) will produce at minimum 3 scholarly products (e.g., manuscripts, presentations, preliminary exams) (what) that are rated by a panel of 3 faculty as showing “strong and substantial application” (how much) of evidence based practice in language and literacy

Project Performance Measure Examples Outcome measure (e.g.) – At the end of each course taught (when), 90% of scholars (who) will have their teaching competence of evidence based practices (what) rated by 2 faculty as 6 or higher (how much) on a 7-point scale using a standard instrument developed by the project

Project Performance Measure Examples Outcome measure (e.g.) – By the end of Year 3 of their program (when), 100% of scholars (who) will serve on at least one (how much) MA or undergraduate research committee (what)

Project Performance Measure Examples Outcome measure (e.g.) – By the end of their program (when), 100% of scholars (who) will attain at least 90% (how much) of required research competencies (what) as measured by faculty evaluations of a web-based portfolio

Common Problems Activities are NOT performance measures If the best response is “Yes, we did that,” it is likely an activity (not a performance measure) Examples of activities: Scholars complete required coursework Faculty meet with scholars on a monthly basis Evaluate the project Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Common Problems Performance measures need to be measurable Examples with measurement problems: Maintain meaningful collaborative partnerships Provide mentors to socialize scholars to the program Provide academic support in areas of concern to scholars Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Need additional information on writing performance measures? All grantees are strongly encouraged to seek training on writing performance measures. For further information on developing performance measures and logic models, see - http://www.tadnet.org/model_and_performance

Summary Projects should have a few clear objectives that explain what the project is doing to support their overall goal(s) Each objective should have a few, specific performance measures to demonstrate how progress toward meeting the objective is being measured Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

Completing the ED 524B The ED 524B is a required annual and final reporting form with specific instructions. The form is used by all ED grants and has been approved by the Office of Management and Budget (OMB). Project Directors must follow the directions listed in the Dear Colleague letter and ED 524B Instructions provided by OSEP Word or PDF versions of the forms are available at http://www2.ed.gov/fund/grant/apply/appforms/appforms.html

X x

ANNUAL PERFORMANCE REPORTS Reporting Period: For first year grants, the date is the beginning of the project through February 29th. For grants in years 2-5, it is the date from the end of the previous reporting period through February 29th. 02 29 20--__

ANNUAL PERFORMANCE REPORTS Budget Expenditures: Must be data or information from the business or grants office. Report expenditures through the end of the “Reporting Period” .

ANNUAL PERFORMANCE REPORTS Performance Measure Status: This will be checked “No” for all performance reports. The APR contains data for the reporting period. The date entered here will be the due date for your Final Performance Report, which is 90 days after the end of the grant. Signatory must have authority to sign on behalf of the institution since the grant is from the Department to the institution and not to an individual. (This person is typically not the Project Director).

EXECUTIVE SUMMARY SHEET OMB No. 1894-0003 Exp. 02/28/2011 OMB No. 1894-0003 Exp. 02/28/2011 EXECUTIVE SUMMARY SHEET H325 - - - - - - *** Provide highlights of the project's activities and the extent to which the expected outcomes and performance measures were achieved during the reporting period. Do NOT include the project abstract.

PROJECT STATUS CHART H325 - - - - - -

PROJECT STATUS CHART H325 - - - - - - Enter one of the project’s objectives; on subsequent pages, you will enter additional project objectives as submitted in your grant application.

PROJECT STATUS CHART Enter the PROGRAM performance measure(s) that align with the objective immediately followed by any PROJECT measure(s) developed to address that program measure. Enter any additional PROJECT performance measures that show you are measuring progress toward meeting the objective.

PROJECT STATUS CHART Here you identify if the performance measure is a program measure, “PRGM,” or a project measure, “PROJ.” PRGM PROJ Note: Program measure refers to one of OSEP’s 7 performance measures for the Professional Development Program. Project measures are unique to your grant.

PROJECT STATUS CHART QUANTITATIVE DATA Depending on your measure, enter either a raw number or a ratio and percentage. Enter the target number identified in the performance measure and then the actual data for this year. If complete data are not available for the measure, enter “999” (if no baseline) or “NA” in the “Raw Number” or “% “column (as appropriate) and provide an explanation at the bottom of the page. Measure #1 8 6 Measure #2 10 10 100 5 10 50

Information to Include in the Explanation of Progress Section Describe the data provided (e.g., what data collection methods were used, when were the data collected, how was a sample drawn, are there missing/incomplete data, what was the response rate, was a reliability measure taken). Your Project Officer should be able to understand and interpret the number in the chart from your description in this section. What changes in the data occurred since last APR (i.e., trend)? What activities were undertaken to achieve the targets? If targets were not met, what are possible reasons? How will activities that failed to meet targets be improved?

PROJECT STATUS CHART QUALITATIVE DATA If the measure requires the collection of qualitative data, then leave the “Quantitative Data” section blank. In the Explanation of Progress section of the page, referencing the performance measure by number, report applicable qualitative data along with other information about how these data were collected, targets and activities –refer to previous slide for additional content requirements.

H325 - - - - - - Provide learning opportunities that result in scholars attaining jobs as faculty in higher education The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curriculum. PRGM Annually, one selected course syllabus will achieve a score of 90 or higher on its integration of evidence-based practice as measured by an expert panel using a rubric designed by the project. PROJ 90 90 No data are reported on program measures in the APR (change in 2011). The project measures must align to the program measure. Both are aligned to the project objective. Upon completion of a seminar on universal design for instruction, the mean rating across scholars on the usefulness of the seminar will be 4.5 or greater on a 5-point scale. PROJ 4.5 4.0 47

H325 - - - - - - Provide learning opportunities that allow scholars to succeed as faculty in higher education. Explanation of Progress (Include Qualitative Data and Data Collection Information) (1b) We have attached the instrument developed for this purpose to Section C. There are 10 items in the rubric; each can achieve a score of 0-10, so a score of 90 reflects a reduction of 10 points across the items. Using this tool, the number in the chart is the mean rating by 2 faculty members at another campus of our university. The course selected for review was a course on Issues in Special Education. Based on their feedback, we updated references on progress monitoring. (1c) Since this grant is designed with a focus on universal designs for instruction, a required seminar on this topic was identified for more in-depth evaluation. The seminar was offered during the Fall 2010 semester. All students in the course completed the rating. Given that the target was not met, students will be interviewed to see how the seminar might be improved for future scholars, with the goal of meeting the “usefulness” score target of 4.5 the next time the class is offered and evaluated. 48

Final Page of the Report Section B: Refer to the instructions for Section B with the ED 524B Section C: Refer to the instructions for Section C with the ED 524B—put attachments here (e.g., recruitment material, evaluation instruments, journal articles)

Section B – Budget Information This section is never blank! Actual expenditures for reporting period Provide an explanation if you did not expend funds at the expected rate. Describe any significant changes to your budget resulting from modifications to project activities. Describe any changes to your budget that affected your ability to achieve your approved project activities and/or project objectives.

Section B – Budget Information Do you expect to have any unexpended funds at the end of the current budget period? (Explain why, provide an estimate, and indicate how you plan to use the unexpended funds [carryover] in the next budget period.) Describe any anticipated changes in your budget for the next budget period that require prior approval from the Department.

Section C – Additional Information If applicable, please provide a list of current partners on your grant and indicate if any partners changed during the reporting period. Please indicate if you anticipate any change in partners during the next budget period. If any of your partners changed during the reporting period, please describe whether this influenced your ability to achieve your approved project objectives and/or project activities. Describe any changes that you wish to make in the grant’s activities for the next budget period that are consistent with the scope and objectives of your approved application.

Section C – Additional Information If you are requesting changes to the approved Project Director and/or to other approved key personnel with a proposed effective date during the remainder of the current or the next budget period, please indicate the name, title and percentage of time of the requested key personnel. Attach a resume or curriculum vitae for the proposed key personnel when you submit your performance report. Note: Do not report on any key personnel changes that were already made during the current or previous budget period(s). Departmental approval must be requested and received prior to making key personnel changes. Provide any other information about your project including unanticipated outcomes or benefits

Submitting the ED 524B Submit the ED 524B at http://www.g5.gov/ (there are instructions for using G5 in the continuation packet) A signed ED 524B Cover Sheet must be scanned and emailed in PDF format to your Project Officer. Special cases require regular email submission of the 524B and the signed cover sheet in PDF format to your Project Officer rather than submission through G5 – Final reports and APRs for grants in their last performance period or in no-cost extension Grants that were front-loaded (forward-funded) last year –ALL 2011 325D GRANTS WERE FRONT LOADED LAST YEAR

Contact your OSEP Project Officer with any questions!