Educational Technology, Media, and Materials Educational Technology, Media, and Materials Program (ETechM2) APR Reporting FY 2016 Office of Special Education Programs Research to Practice Division Discretionary Grants

Overview Provide guidance in…. Strengthening project and OSEP ETechM2 program performance measures to maximize the potential for meaningful Educational Technology, Media, and Materials (ETechM2) program data. Completing the U.S. Department (ED) Grants Performance Report form - (ED 524B). On December 9, 2012, OSEP officially changed the name of the Technology and Media Services Program to the Educational Technology, Media, and Materials Program. The program purpose was amended to include "...(4) provide accessible educational materials to children with disabilities in a timely manner."

Goals – Objective - Measures ETechM2 Program Goals Project Objectives: The objectives listed in grant proposal and how project will support the overall OSEP program goal. Project Performance Measures: The quantitative and qualitative steps on the progress toward meeting project and program objectives.

ETechM2 Program Goals To improve results for children with disabilities by… Promoting the development, demonstration, and use of technology; Supporting educational media activities designed to be of educational value in the classroom; Providing support for captioning and video description that is appropriate for use in the classroom; and Providing accessible educational materials to students with disabilities in a timely manner

Project Objectives Project objectives should be: Clear about what your project is trying to accomplish and support the overall program goal; and High quality.

Why High Quality is Important? High quality objectives and measures… Makes it easier for grantees to measure progress Allows grantees to report progress easily and quantitatively Establishes targets (both short- [annual] and long-term) Allows OSEP to justify continued grantee funding and gather evidence of program effectiveness.

Criteria for High Quality Project Objectives Relevance How relevant are the project objectives to the overall goals of the program and/or the goal of your project? Applicability How applicable are the project objectives to the specific activities that are being conducted through your particular project?

Criteria for High Quality Project Objectives (cont'd) Focus How focused are the project objectives? Measurability Are there concepts in the project objectives that lend themselves to measurement? If so, is measurement feasible?

ETechM2 Program Performance Measures All OSEP funded projects must respond to the quality, relevance and usefulness (QRU) program measures annually.

ETechM2 Program Performance Measures Measure 1: The percentage of ETechM2 Program products and services judged to be of high quality by an independent review panel of experts qualified to review the substantive content of the products and services. Measure 2: The percentage of ETechM2 Program products and services judged by an independent review panel of qualified experts to be useful in improving results for infants, toddlers, children and youth with disabilities.     Measure 3: The percentage of ETechM2 Program products and services judged by an independent review panel of qualified experts to be of high relevance to improving outcomes of infants, toddlers, children and youth with disabilities.   

Definitions Quality: The degree to which the product content or content delivered through the service is consistent with current research or policy. Relevance: The degree to which the product content or content delivered through the service addresses current educational problems or issues. Usefulness: the degree to which the product content or content delivered through the service can be readily and successfully used by consumers.

Definitions (cont’d) A product is a piece of work, in tangible or electronic form developed and disseminated by an ETechM2 project to contribute to the improvement of outcomes for children with disabilities. A service is work performed by an ETechM2 project to provide information to a specific audience relevant to the improvement of outcomes for children with disabilities.

ETechM2 Program Performance Measures (cont'd) The federal cost per unit of products/services developed/rendered by the ETechM2 projects. All Stepping up Technology Implementation (327S) projects will need to report on this cost measure for years 4 and 5. Other projects may also need to respond to this measure, so please speak with your Project Officer.

Tips for Writing Project & Program Measures Report on ETechM2 program performance measures. Align each program performance measure under the appropriate project objective for which it might be a reasonable measure of progress. Describe under "Explanation of Progress" section additional information about progress achieved or not achieved toward project and program measures. Project measures should clearly address how and what outcomes are being accomplished to help meet the project objective.

Tips for Writing Project & Program Measures (cont’d) When writing the program performance measures, first write the measure verbatim. Then write the measure specifically for how your project will measure quality, relevance and usefulness.

Example OSEP Program Measure: The percentage of Educational Technology, Media, and Materials Program products and services judged to be of high quality by an independent review panel of experts qualified to review the substantial content of the products and services. Project Measure: Beginning in year 2, 80% of participating educators will rate the technology resources as high quality (4.5 or higher on a 5-point likert scale) for improving reading outcomes for students with high incidence disabilities.

Tips for Writing Project & Program Measures (cont'd) Project measures should be measurable and have established targets and benchmarks. Project measures that are qualitative, describe targets and outcomes under “explanation of progress” Your statements of program performance measures should reflect the formative and summative evaluation data from advisory committees, user groups, field tests, needs assessment surveys, etc. Work with your PO when writing project objectives and performance measures.

Tips for Writing Project & Program Measures (cont'd) Do not put all program measures (QRU) under one project objective. If the best response is “Yes, we did that,” it is likely an activity (not a performance measure) Listing meetings and phone calls are not project measures, but a process.

Tips for Writing Project & Program Measures (cont'd) Projects should have a few clear objectives that explain what the project is doing to support their overall project goal(s). Each objective should have a few, specific project performance measures to demonstrate how progress toward meeting the objective is being measured Work with your Project Officer.

High Quality Project Performance Measures High quality project performance measures show— What will change How much change you expect Who will achieve the change When the change will take place

Performance Measures All grantees are strongly encouraged to seek training on writing performance measures. Webinar adapted from information on developing performance measures provided by Center for Evaluation and Education Policy, see - http://www.tadnet.org/pages/589 Work with your Project Officer

Part II Completing ED Grant Performance Report - ED 524B Form

Completing the ED 524B The ED 524B is a required annual and final reporting form with specific instructions-- The form is used by all ED grants and has been approved by the Office of Management and Budget (OMB). Project Directors must follow the directions listed in the Dear Colleague letter and ED 524B Instructions provided by OSEP http://www2.ed.gov/fund/grant/apply/osep/funding.html

Annual Performance Reports Submission date: …by May 5, 2017 Reporting Period: For first year grants, the date is the beginning of the project through February 28th. For grants in years 2-5, it is the date from the end of the previous reporting period through February 28th.

Annual Performance Reports Budget Expenditures: Must be data or information from the business or grants office. Report actual expenditures for the “Reporting Period” (i.e., Budget Period and Reporting Period are the same)

Annual Performance Reports Performance Measure Status: This will be checked “NO” for all performance reports. The APR contains data for the reporting period (see Inst #7), not the budget period. The data entered here will be the date for your Final Performance Report, which is 90 days after the end of the grant. Signatory must have authority to sign on behalf of the institution since the grant is from ED to the institution. Submit a pdf copy of the signed Cover Sheet to your Project Officer if requested. Upload signed cover sheet to G5 along with the package.

Annual Performance Reports Executive Summary Sheet: Provide highlights of the project’s activities and the extent to which the expected outcomes and performance measures were achieved during the reporting period. DO NOT include the project abstract. Insert the PR/Award # (on each page thereafter)

Annual Performance Reports Project Status Chart: Enter one of the project’s objectives; on subsequent pages, you will enter additional project objectives as submitted in your grant application Enter PR/Award #

Annual Performance Reports Project Status Chart: Enter the PROGRAM (PRGM) performance measure(s) that align with the objective immediately followed by any PROJECT (PROJ) measure(s) developed to address that program measure Enter any additional PROJECT performance measures that show you are measuring progress toward meeting the objective

Annual Performance Reports Project Status Chart: Quantitative Data Depending on your measure, enter either a raw number or a ratio and percentage. Enter the target number identified in the performance measure and then the actual data for this year. If complete data are not available for the measure, enter “999” (if no baseline) or “NA” in the “Raw Number” or “% “column (as appropriate) and provide an explanation at the bottom of the page.

Annual Performance Reports Project Status Chart: Qualitative Data If the measure requires the collection of qualitative data, then leave the “Quantitative Data” section blank. In the Explanation of Progress section, referencing the performance measures by number, report applicable qualitative data along with other information about how these data were collected, targets and activities. Ensure additional content requirements are provided.

Annual Performance Reports Project Status Chart: Explanation of Progress Provide any additional information that clarifies the data provided (e.g., what data collection methods were used, when were the data collected, how was a sample drawn, are there missing/incomplete data, what was the response rate, was a reliability measure taken). Your Project Officer should be able to understand and interpret the numbers in the chart from your description in this section. What changes in the data occurred since last APR (i.e., trend)?

What activities were undertaken to achieve the targets? Project Status Chart : Explanation of Progress What activities were undertaken to achieve the targets? If targets were not met, what were some of the possible reasons? How will activities that failed to meet targets be improved? This year the package is set up to give you the opportunity to upload a document with additional text for Section A (specify the performance measure being addressed).

Annual Performance Reports Project Status Chart: Final Page of the Report Section B - Refer to the Instructions for Section B with the ED 524B Section C - Refer to the Instructions for Section C with the ED 524B

Section B - Budget Information Annual and Final Performance Reports This section is never blank! Actual expenditures for reporting period – not allocation amount Provide an explanation if you did not expend funds at the expected rate. Describe any significant changes to your budget resulting from modifications to project activities. Describe any changes to your budget that affected your ability to achieve your approved project activities and/or project objectives.

Section B - (Cont'd) If you have any... Annual Performance Reports If you have any... Unexpended funds at the end of the current budget period explain why, provide an estimate, and indicate how you plan to use the unexpended funds [carryover] in the next budget period. Anticipated changes in your budget for the next budget period that require prior approval from the Department.

Section C - Additional Information Annual Performance Reports – A list of all new products that were developed in the past year (i.e., between March 1, 2016 and February 28, 2017) Examples of Products: Software or hardware products Journal or informational articles Digital materials Websites Manuals Research reports Learning modules

Section C - Additional Information (cont’d) Projects must include— A list of all new services that were rendered in FY 2016 (between October 1, 2015 and September 30, 2016) * Examples of Services Conducting training sessions Providing captioning, video description, Braille, or other accessible formatting of text or media Leading and convening informational meetings Responding to inquires from a targeted population

Section C - Additional Information (cont’d) If applicable, provide a list of current partners on your grant and indicate any partners that changed during the reporting period. If any of your partners changed, describe whether this influenced your ability to achieve your approved project objectives and/or project activities. Describe any changes that you wish to make in the grant’s activities for the next budget period that are consistent with the scope and objectives of your approved application.

Section C - Additional Information (cont’d) If you are requesting changes to the approved Project Director and/or to other key personnel contact your project officer. Provide any other information about your project including unanticipated outcomes or benefits.

Submitting the ED 524B APRs must be submitted in G5 at: http://www.g5.gov/ (there are instructions for using G5 in the continuation packet) Upload a signed ED 524B Cover Sheet in PDF format to G5. Project’s in a No Cost must still submit an APR in G5. If you have questions about this, please contact your Project Officer.

Questions. Thank you. Terry L. Jackson, Ed. D Questions? Thank you. Terry L. Jackson, Ed.D. OSEP ETechM2 Program Lead terry.jackson@ed.gov