1. Office of Special Education and Rehabilitative Services United States Department of Education Program Review and Best Practices: Who, What, When, Where, Why, How? Jeremy Buzzell, Program Specialist Denver, CO. May 23, 2007

2. 1 Program Review: Why? Section 7(b): Review of Participating Entities The Secretary shall assess the extent to which entities that receive grants under this Act are complying with the applicable requirements of this Act and achieving measurable goals that are consistent with the requirements of the grant programs under which the entities received the grants.

3. 2 Section 7(c): Corrective Action and Sanctions (1) CORRECTIVE ACTION.—If the Secretary determines that an entity that receives a grant under this Act fails to substantially comply with the applicable requirements of this Act, or to make substantial progress toward achieving the measurable goals... the Secretary shall assist the entity, through technical assistance... within 90 days after such determination, to develop a corrective action plan. (2) SANCTIONS.—If the entity fails to develop and comply with a corrective action plan...the entity shall be subject to 1 of the following corrective actions selected by the Secretary: (A) Partial or complete termination of funding under the grant program, until the entity develops and complies with such a plan. (B) Ineligibility to participate in the grant program in the following year. (C) Reduction in the amount of funding that may be used for indirect costs under section 4 for the following year. (D) Required redesignation of the lead agency... or an entity responsible for administering the grant program.

4. 3 Program Review: How? The program review is a one- size fits all protocol.

5. 4 Program Review: How? Proposed Process for Developing Program Review Protocol: 1) Input from NATTAP Annual Conference in Denver 2) Review of existing OSERS protocols 3) Internal development of ideas based on (1) and (2) 4) Share and seek feedback on ideas 5) Refine ideas based on (4) – develop written draft 6) Share and seek feedback on draft 7) Refine draft based on (6) – develop final 8) Internal review of final 9) Roll-out

6. 5 Program Review: What? “... substantial progress toward achieving the measurable goals...” Measurable goals are in State Plan and current data collection requirements (access and acquisition measures) Measurable goals are in State Plan and current data collection requirements (access and acquisition measures) Initially, states set their own targets Initially, states set their own targets Once there is enough data, national targets for performance will be set Once there is enough data, national targets for performance will be set - Example: Range of acceptable percentages - Example: Range of acceptable percentages *NISAT will provide TA to RSA and States*

7. 6 Program Review: What? “... substantial compliance with the applicable requirements...” Applicable requirements: Lead agency and implementing entity Lead agency and implementing entity Advisory council (composition and function) Advisory council (composition and function) State-level activities State-level activities State Leadership activities State Leadership activities Funding allocations, accounting for, and use of funds Funding allocations, accounting for, and use of funds Provision of data Provision of data Statewideness and comprehensiveness Statewideness and comprehensiveness ADA and 508 ADA and 508

8. 7 Program Review: How? What does substantial compliance mean? Can it be measured quantitatively using NISAT data?

9. 8 Program Review: How? What does substantial compliance mean? If it cannot be measured quantitatively, what can/must be measured qualitatively? How is this qualitative measurement done?

10. 9 Program Review: How? Do all 7 state-level and state leadership activities need to be reviewed? Is each activity reviewed differently?

11. 10 Program Review: When? Program reviews to begin in FY 2008 (3 rd year of plan) Limitation: No more than 6 a year given staff time and budget Process must be selective On what basis is a state chosen for review?

12. 11 Program Review: When? On what basis is a state chosen for review? Performance?Random? Alphabetical? Population? Region?Grant size? Lead agency/implementing entity? Program maturity? Other dimension or combination of dimensions?

13. 12 Program Review: Where? Must a program review be done on-site? (On-site review is no longer required by the Act.)

14. 13 Program Review: Where? What does on-site mean? What is “seen” on-site? What can be done off-site?

15. 14 Program Review: Who? Who is involved in the review?

16. 15 Program Review: Who? Who is involved in the review? RSA staff RSA staff External experts? External experts? Internal staff? Internal staff? Consumers/stakeholders? Consumers/stakeholders? Subcontractors? Subcontractors?

17. 16 Program Review: Then What? Program reviews generally conclude with a written report. These reports are public information. What should we be able to say about a program when the review is complete?