1. Rapid cycle evaluation:
An approach to evidence-based decision-making in TANF programs
Presentation at the 2017 National TANF State Directors’ Meeting
Arlington, VA
Tuesday, August 15, 2017
Jonathan McCay  Annalisa Mastri

2. Overview of today’s session
Introduction and overview of evaluation approaches
RCE case study from Larimer County, CO
Project IMPROVE and other resources

3. Traditional program evaluation
Researcher-driven and knowledge-focused
Answers the question: Does “it” work?
Focused on internal validity and can be designed to generalize results to other contexts
Often a multi-year process

4. Rapid cycle evaluation
Focused on program improvement
Therefore, an RCE is implementer-driven
Measures the impact of changes to program operations or services
Answers the question: Can “it” be improved?
Uses rigorous research methods to identify a causal link between the change(s) and the outcome(s)
Tailored to inform decision-making
Relies predominantly on existing program data to measure impacts
Short time horizon
Focus on results that can be observed in a relatively short timeframe

5. Traditional Program Evaluation
A shift in purpose
Traditional Program Evaluation
Rapid Cycle Evaluation
Often…
Uses analytic methods
Implementer-driven
Researcher-driven & knowledge-focused
Can “it” be improved?
Does “it” work?
Builds knowledge & evidence
Tailored to inform decision-making
Generalized
Multi-year
A year or less

6. Situations well-suited to RCE
Strategy to improve engagement and follow-through
Particularly, behavioral “nudges”
Variation in the combination, dosage, or intensity of services
Operational changes to improve efficiency
Different communication and marketing approaches
Effectiveness of an oversubscribed program
Using a random lottery selection process

7. Is RCE feasible? What, specifically, do you hope to achieve?
When do you expect it to affect change?
What data do you have available to measure the change(s)?
Can we find or create a good comparison between those who receive X and those who do not?

8. The RCE process

9. An RCE case study: Larimer County, CO
1 Diagnose the problem
Monthly Time Sheet (MTS) is the required reporting form for Works Program (TANF) participants
Documents hours by activity for the work participation rate
As of December 2015, about 1 of every 2 participants submitted their MTS on time
Considerable amounts of staff time spent on case conciliation due to low submission rate
Families at risk of sanction or case closure
Admittedly, the program did very little to communicate to participants about the MTS upfront

10. An RCE case study: Larimer County, CO
1 Diagnose the problem
Misunderstanding
Procrastination
Forgetting

11. An RCE case study: Larimer County, CO
2 Define the question & population
Does a low-touch or a high-touch behavioral intervention increase on-time MTS submissions?
Low-touch approach with existing TANF participants
High-touch approach with new TANF participants

12. An RCE case study: Larimer County, CO
3 Define the intervention
Intro brochure + Planning + Reminder postcard
+ Prize lottery for on-time submitters
Low-touch approach
Variation in high-touch approach

13. An RCE case study: Larimer County, CO
4 Conduct test
County-based tracking data
MTS submission data
Participant data in the Colorado Benefits Management System
Cloud-based dashboards for random assignment and selecting prize lottery winners

14. An RCE case study: Larimer County, CO
4 Conduct test
Treatment
Receives postcard
221
participants
Random assignment
Low-touch RCE with existing participants
Control
Business as usual
227
participants
Enhanced Treatment
Basic + prize lottery
159
participants
Random assignment
High-touch RCE with new participants
Basic Treatment
Brochure, , postcard
144
participants
Control
Business as usual
147
participants

15. An RCE case study: Larimer County, CO
5 Analyze results: Low-touch strategy
36.3% timely at least half of study period
38.3% timely at least half of study period
Proportion of clients
Total # of timely submissions

16. An RCE case study: Larimer County, CO
5 Analyze results: High-touch strategy
47.9% at least 1 timely month
54.1% at least 1 timely month

17. An RCE case study: Larimer County, CO
Use evidence to inform decision-making
Larimer County has moved to implementing an online form for MTS submission
Interested in testing this with various incentives and/or messaging strategies to improve submission rates

18. Evaluation that serves program improvement
Accessible process
Building local research capacity
Co-creative, collaborative approach between practitioners and researchers
Decision-focused evaluation
Encourages iteration and continuous quality improvement

19. Project IMPROVE: A resource
Project sponsored by the Office of Family Assistance
Goal:
To improve TANF programs’ practices and capacities for using evidence, developing evaluation practices, documenting progress and processes, and revising program components by using evidence and evaluation data
Technical assistance provided by Mathematica Policy Research
Supports up to 5 “sites” per federal fiscal year

20. Project IMPROVE: A resource
Federal fiscal year sites
California
Colorado
Maryland
Washington (State)
Tribal TANF learning community
Federal fiscal year sites
TBD

21. RCE resources to support you
The RCE Coach
A guide to opportunistic experiments Google search: “opportunistic experiments in human services”
Join Project IMPROVE Talk with us after today’s presentation if you’re interested!

22. For more information Annalisa Mastri amastri@mathematica-mpr.com
Jonathan McCay
Scott Baumgartner