1. AmeriCorps State and National Evaluation Requirements August 16, 2013 1

2. Learning Objectives Increased understanding of broader federal policy context Increased understanding of AmeriCorps State and National evaluation requirements 2

3. Federal Policy Context Presidential AdministrationsOffice of Management and Budget Guidance President Clinton (1993 – 2001) Government Performance and Results Act President Bush (2001 – 2009) Program Assessment Rating Tool President Obama (2009 – 2017) Memoranda M-10 -01 Increased Emphasis on Program Evaluation M-12-14 Use of Evidence and Evaluation in the 2014 Budget M-13-17 Next Steps in the Evidence and Innovation Agenda 3

4. Code of Federal Regulations The Code of Federal Regulations, or CFR, is the collection of all administrative rules and regulations from the various departments and agencies of the federal government. It is organized by titles, chapters, and parts that cover specific regulatory areas 4

5. Code of Federal Regulations http://www.gpoaccess.gov/ecfr – Select Agency List from left navigation bar – Scroll down to CNCS, select 45 CFR Chapters XII, XXV – Scroll down to XXV, select 2500-2599 – Select 2522 – Select Subpart E – Evaluation Requirements 5

6. Code of Federal Regulations Primary Focus of Todays Webinar: Evaluating Programs: Requirements and Procedures (§§.700 -.740) CNCS finalized these regulations July 8, 2005 6

7. Definition of Evaluation …evaluation uses scientifically-based research methods to assess the effectiveness of programs by comparing the observed program outcomes with what would have happened in the absence of the program. §2522.700 7

8. Definition of Performance Measurement Performance measurement is the process of systematically and regularly collecting and monitoring data related to the direction of observed changes in communities, participants (members), or end beneficiaries receiving your programs services. It is intended to provide an indication of your programs operations and performance. §2522.700 8

9. Evaluation Requirements National Direct & State Competitive ($500K+) …you must arrange for an independent evaluation of your program, and you must submit the evaluation with any application to the Corporation for competitive funds… § 2522.710 9

10. Strengthening the Evidence Base National Direct & State Competitive ($500K+) Greater investments of public resources warrant more rigorous evidence and evaluation requirements Independent, impact evaluations designed to provide evidence of a causal relationship between program activities and participant outcomes are required 10

11. Evaluation Requirements National Direct & State Competitive ($500K+) Your evaluation must cover a minimum of one year but may cover longer periods. § 2522.720 11

12. Evaluation Requirements National Direct & State Competitive (less than $500K) or Education Award Program Corporation …you must conduct an internal evaluation of your program, and you must submit the evaluation with any application to the Corporation for competitive funds… § 2522.710 12

13. Strengthening the Evidence Base National Direct & State Competitive (less than $500K) or Education Award Program Corporation Encourage use of experimental or quasi- experimental designs but not required Select a study design most appropriate for developmental phase of AmeriCorps program (process, implementation, outcome or impact) 13

14. Evaluation Requirements National Direct & State Competitive (less than $500K) or Education Award Program Corporation Your evaluation must cover a minimum of one year but may cover longer periods. § 2522.720 14

15. Evaluation Requirements State Commission …you must establish and enforce evaluation requirements for your State formula subgrantees, as you deem appropriate. § 2522.710 15

16. Evaluation Requirements State Formula Grantees If you are a State formula grantee, you must conduct an evaluation, as your State commission requires. § 2522.720 16

17. Evaluation Requirements The Corporation may, in its discretion, supersede these requirements with an alternative evaluation approach, including one conducted by the Corporation at the national level. Grantees must cooperate fully with all Corporation evaluation activities. § 2522.710 17

18. Evaluation Requirements What is due when? If you are an existing grantee re-competing for AmeriCorps funds for the first time, you must submit an evaluation plan… as part of your application for funding. If you again compete for AmeriCorps funding after a second three-year grant cycle, you must submit the completed evaluation with your application for funding. § 2522.730 18

19. CNCS Cheat Sheet 19 If you are competing for… You are… Your first AmeriCorps grant Not required to submit an evaluation plan with the application or complete an evaluation during the initial grant period. You are encouraged to begin the evaluation planning process. Your second AmeriCorps grant Required to submit an evaluation plan with the application and complete the evaluation during the next three-year grant period. Your third AmeriCorps grant Required to submit a report on the evaluation conducted during the previous grant period with the application, and submit an evaluation plan for the next grant period with the application. An AmeriCorps grant beyond your third grant award Required to submit an evaluation plan for the next grant period, along with a report that details findings from evaluations conducted in previous grant periods with the application. Evaluation plans should build on previous evaluation efforts. Formula grantees applying for competitive funds Required to submit an evaluation plan for the next grant period.

20. CNCS Priorities Strengthening the evidence base for national service using scientifically-based research methods Supporting grantee efforts to build the evidence base of their AmeriCorps programs 20

21. Strengthening the Evidence Base In order to maximize the impact of the public investment in national service, CNCS will fund programs that can demonstrate community impact and solve community problems using an evidence- based or evidence-informed approach. (2013 ACSN NOFO, page 1) An evidence-based approach is one where the intervention is based on research or backed by statistically significant evaluation findings. (2012 ACSN NOFO, page 19) 21

22. -Gather evidence supporting the intervention- Design/Adopt a strong program -Develop a Logic Model -Create Implementation Materials -Pilot implementation -Document program process(es) -Ensure fidelity in implementation -Evaluate programs quality and efficiency -Establish continuous process improvement protocols [Performance Measures - Outputs] -Develop indicators for measuring outcomes -Conduct pre-/post- intervention evaluation to measure outcomes -Conduct process evaluation [Performance Measures - Outcomes] -Examine linkage between program activities and outcomes -Perform multiple pre- and post-evaluations (time series design) -Conduct independent (unbiased) outcome evaluation(s) -Conduct meta-analysis of various studies -Establish causal linkage between program activities and intended outcomes/impact (e.g. Conduct quasi- experimental evaluation using a comparison group, evaluation with random assignment (RCT), regression analysis, or other appropriate study design) -Conduct Multiple independent evaluations using strong study designs -Measure cost effectiveness compared to other interventions addressing same need Identify a strong program design Attain strong evidence of positive program outcomes Assess programs outcomes Ensure effective implementation Obtain evidence of positive program outcomes Building Evidence of Effectiveness Evidence Informed Evidence Based

23. Supporting Grantees Grantees that have not completed the evaluation requirements in the proper timeframe are strongly encouraged to provide any evaluation findings available and provide an evaluation plan for the next grant period that would fulfill the requirements Evaluations that show null or negative findings will not automatically lead to an unacceptable assessment of your program. CNCS expects such findings to be used for program improvements.

24. Supporting Grantees: Summer 2013 Disseminate Evaluation FAQs Conduct Webinars: – Evaluation Capacity Building – CNCS Evaluation Requirements and FAQs Create central location for evaluation materials on website

25. Supporting Grantees: Fall 2013 Grantee Symposium, Performance Measurement & Evaluation Track – Performance Measurement Pre-Conference Workshop – Developing evaluation plans and reports – Evaluation 101 Initiate individualized technical assistance – 1:1 coaching to help finalize evaluation plans, use performance monitoring and evaluation for program improvement, and develop evaluation reports

26. Supporting Grantees: 2013 - 2014 Individualized technical assistance Evaluation Core Curriculum (webinars, online courses, downloadable resources) – Developing evaluation plans and reports – Evaluation 101 – Logic models – How to report evaluation findings – How to manage an external evaluation – Budgeting for evaluation

27. Questions and Answers We welcome questions at this time. Thank you. 27