1. OSEP TA&D Program Performance Measurement
GPRA Data Collection Involving Centers
May 8th, 2017

2. Webinar Objectives
Enhance awareness of the Program GPRA Performance Measures requirements
Provide overview of procedures related to OSEP TA&D Program GPRA performance data--
Collection and information gathering
Analysis
Reporting
Respond to grantee questions

3. Performance Measures
The Government Performance and Results Act (GPRA) of 1993 required all federal agencies to establish and report annually on goals, measures, and targets
The law was reauthorized in 2010 as “GPRA Modernization Act of 2010”
GPRA requires performance assessments of Government programs for purposes of assessing agency performance and improvement
The Office of Management and Budget, together with the Federal agencies, determines how programs will be assessed
Congress uses program performance assessment data to justify program funding

4. Performance Measures
OSEP reports progress on program performance measures annually to OMB and Congress
This provides an aggregate picture of performance
Their review of our performance data affects the continued funding of IDEA programs
OSEP uses the data to improve the TA&D Program’s performance

5. Purpose of the Program
The purpose of the Technical Assistance and Dissemination to Improve Services and Results for Children with Disabilities program is to promote academic achievement and to improve results for children with disabilities by providing technical assistance (TA), supporting model demonstration projects, disseminating useful information, and implementing activities that are supported by scientifically based research

6. TA&D Program Performance Measures
Annual
Quality of New Service or New Product
Relevance of New Service or New Product
Usefulness of New Service or New Product
Efficiency of the TA&D program (includes milestones and costs)
Long-term
Implementation of evidence-based practices
Promotion of effective models
For Annual Measures, quality, relevance, and usefulness (QRU) involve collecting data from you in addition to what you provide to OSEP in your APR.
You already provide the milestone data for the efficiency measure to us in your APR.
For Long-term Measures, data collection occurs only every two years. This year, we are collecting data on the implementation of evidence-based practices.

7. How is Quality Rated by the Panel?
Percentage of Special Education Technical Assistance and Dissemination (TA&D) Products and services deemed to be of high quality by an independent (Science or Stakeholder) review panel
Substance: Does the product content or the content delivered through the service reflect evidence of conceptual soundness and quality, grounded in recent scientific evidence, legislation, policy, or accepted professional practice?
Communication: Is the product content or the content delivered through the service presented in such a way so as to be clearly understood, as evidenced by being well-organized, free of editorial errors and appropriately formatted?

8. How is Relevance Rated by the Panel?
Percentage of Special Education Technical Assistance and Dissemination (TA&D) Products and services deemed to be of high relevance by an independent (Stakeholder) review panel
Need: Does the product content or the content delivered through the service attempt to solve an important problem or deal with a critical issue?
Pertinence: Does the product content or the content delivered through the service address a problem or issue recognized as important by the target audience(s)?
Reach: To what extent is the product content or the content delivered through the service applicable to diverse segments of the target audience(s)?

9. How is Usefulness Rated by the Panel?
Percentage of Special Education Technical Assistance and Dissemination (TA&D) Products and services deemed to be useful by an independent (Stakeholder) review panel
Ease – Does the product content or the content delivered through the service address a problem or issue in an easily understood way, with directions or guidance regarding how the content can be used to address the problem or issue?
Suitability - Does the product or service provide the target audience(s) with information or resources that can be used again or in different ways to address the problem or issue?

10. How is Efficiency Reviewed?
The cost efficiency of the technical assistance and dissemination program includes the percentage of milestones achieved in the current APR period and the percentage of funds spent during the current fiscal year
You submitted milestone performance as part of your APR
We consider allocations and drawdowns from G5

11. Who Participates?
All Centers in their 2nd-5th year of funding in FY2016
All Centers operating under a no cost extension during FY2016
Centers operating in their first year of funding during FY2016 are not included.

12. Data Collection for Annual Measures
New Products and Services
Projects generate a list of new products/services developed or delivered during FY2016
TSG randomly selects one product and one service from the list
Projects provide a description of the selected new product and new service
Descriptions and products are sent to a science expert panel (for evidenced-based products or services) to rate quality and a stakeholder expert panel to rate the quality (for policy-related products and services) and relevance and usefulness (for all products and services)
Product: A product is a piece of work, in text or electronic form developed and disseminated by an OSEP-funded project to inform a specific audience on a topic relevant to the improvement of outcomes for infants, toddlers, children, and youth with disabilities.
Examples of products include: software or hardware products, journal or informational articles, research reports, booklets, pamphlets (not marketing brochures about your project), web-based instructional materials, manuals, DVDs, CDs, multi-media kits or modules, and PowerPoint presentations.
Examples of items that are not considered products include: Databases, project websites, meeting agendas or materials developed for one time use by an individual, archived chat, conference proceedings, contact lists, progress/performance/continuation/evaluation reports.
Service: A service is work performed by an OSEP-funded project to provide information or assistance to a specific audience on a topic relevant to the improvement of outcomes for infants, toddlers, children, and youth with disabilities.
Examples of services include conducting training or technical assistance, providing captioning, video description, Braille, or other accessible formatting of text or media, leading and convening informational meetings, responding to inquiries from a targeted population.
Note: For the purpose of this performance measurement review process, maintaining a website is not considered to be either a product or a service.

13. Data Collection for Annual Measures
A product is a piece of work (in text or electronic form) that informs a specific audience on a topic relevant to outcomes for children with disabilities
Examples of Products
Journal or informational articles
Booklets
Pamphlets
Manuals
DVDs or CDs
Multimedia kits or modules
PowerPoint Presentations

14. Data Collection for Annual Measures
A service is work that provides information to a specific audience relevant to outcomes for children with disabilities
OSEP recognizes three levels of TA Services
General/Universal
Targeted/Specific
Intensive/Sustained
For the purpose of this performance measurement review process, maintaining a website is not considered to be either a product or a service.

15. Data Collection for Annual Measures
Examples of Services
Conducting training
Providing captioning, video description, Braille, or other accessible formatting of text or media
Leading and convening informational meetings
Responding to inquires from a targeted population

16. New Product/Service Description Guides
Centers develop a New Product and a New Service Description Guide for panel review.
These guides are the primary source of information for the GPRA review
Complete, detailed, and clearly written guides make it easier for panelists to rate product/service QRU accurately
Please consult tips for completing description guides provided by TSG
The New Product, and New Service Description Guides are very important to the GPRA review process. They need your time and attention.
The guides are the primary source of information consulted by the expert review panels in making their QRU ratings.
Although you have the option to submit supporting materials along with each guide, the panelists are not required to read through these materials in their entirety.
Projects that submit complete, detailed, and clear guides make it easier for the expert review panels to rate product and service quality, relevance, and usefulness. If the guides are incomplete, you cannot expect that panelists will spend substantial amounts of time seeking out the information they need from your supporting materials.
TSG has developed a set of tips for completing the guides that can be helpful. You have, or will, receive a copy to consult when developing your guide(s).
Keep in mind that TSG staff are always willing to have a conversation with you about this task and to review a draft of your response.

17. Data Collection for Annual Measures
Submission
Send an electronic copy (in WORD or PDF) of your completed guides and any supporting materials attached to an message to
Be sure that URL addresses where supporting materials for your product and/or service are marked clearly
Deadline is May 25th, 2017

18. Long-term Measure for TA&D
The percentage of states receiving TA&D program services regarding scientifically- or evidence-based practices for infants, toddlers, children, and youth that successfully promote the implementation of those practices in their school districts or service agencies
collected every 2 to 3 years

19. Data Collection for the Long-Term Measure
Selection
Centers who self report that they worked intensively with states to implement evidence-based practices
Data Collection
Centers provide a list of states with which they have worked to promote evidence-based practices to districts and service agencies
TSG randomly selects one state on which to focus data collection and contacts Center staff to learn more about the state and local EBP implementation
Centers are first asked if they have worked intensively in states during the most recent 5 fiscal years.
Those Centers that reported that they had, are then asked to submit a list of states with which they worked. TSG randomly selects one state from each list. The Centers provide a description of the scientifically or evidence-based practice they have promoted to this state along with contact information for state agency personnel most closely engaged with the work.
TSG schedules a telephone interview with Center staff to understand how the EBP has been implemented at the state and local levels.
Centers involved with the measure can expect to hear from TSG later this month.

20. Our Partnership is the Key to Successful Performance