Rapid cycle evaluation: An approach to evidence-based decision-making in TANF programs Presentation at the 2017 National TANF State Directors’ Meeting Arlington, VA Tuesday, August 15, 2017 Jonathan McCay  Annalisa Mastri

Overview of today’s session Introduction and overview of evaluation approaches RCE case study from Larimer County, CO Project IMPROVE and other resources

Traditional program evaluation Researcher-driven and knowledge-focused Answers the question: Does “it” work? Focused on internal validity and can be designed to generalize results to other contexts Often a multi-year process

Rapid cycle evaluation Focused on program improvement Therefore, an RCE is implementer-driven Measures the impact of changes to program operations or services Answers the question: Can “it” be improved? Uses rigorous research methods to identify a causal link between the change(s) and the outcome(s) Tailored to inform decision-making Relies predominantly on existing program data to measure impacts Short time horizon Focus on results that can be observed in a relatively short timeframe

Traditional Program Evaluation A shift in purpose Traditional Program Evaluation Rapid Cycle Evaluation Often… Uses analytic methods Implementer-driven Researcher-driven & knowledge-focused Can “it” be improved? Does “it” work? Builds knowledge & evidence Tailored to inform decision-making Generalized Multi-year A year or less

Situations well-suited to RCE Strategy to improve engagement and follow-through Particularly, behavioral “nudges” Variation in the combination, dosage, or intensity of services Operational changes to improve efficiency Different communication and marketing approaches Effectiveness of an oversubscribed program Using a random lottery selection process

Is RCE feasible? What, specifically, do you hope to achieve? When do you expect it to affect change? What data do you have available to measure the change(s)? Can we find or create a good comparison between those who receive X and those who do not?

The RCE process

An RCE case study: Larimer County, CO 1 Diagnose the problem Monthly Time Sheet (MTS) is the required reporting form for Works Program (TANF) participants Documents hours by activity for the work participation rate As of December 2015, about 1 of every 2 participants submitted their MTS on time Considerable amounts of staff time spent on case conciliation due to low submission rate Families at risk of sanction or case closure Admittedly, the program did very little to communicate to participants about the MTS upfront

An RCE case study: Larimer County, CO 1 Diagnose the problem Misunderstanding Procrastination Forgetting

An RCE case study: Larimer County, CO 2 Define the question & population Does a low-touch or a high-touch behavioral intervention increase on-time MTS submissions? Low-touch approach with existing TANF participants High-touch approach with new TANF participants

An RCE case study: Larimer County, CO 3 Define the intervention Intro brochure + Planning email + Reminder postcard + Prize lottery for on-time submitters Low-touch approach Variation in high-touch approach

An RCE case study: Larimer County, CO 4 Conduct test County-based tracking data MTS submission data Participant data in the Colorado Benefits Management System Cloud-based dashboards for random assignment and selecting prize lottery winners

An RCE case study: Larimer County, CO 4 Conduct test Treatment Receives postcard 221 participants Random assignment Low-touch RCE with existing participants Control Business as usual 227 participants Enhanced Treatment Basic + prize lottery 159 participants Random assignment High-touch RCE with new participants Basic Treatment Brochure, email, postcard 144 participants Control Business as usual 147 participants

An RCE case study: Larimer County, CO 5 Analyze results: Low-touch strategy 36.3% timely at least half of study period 38.3% timely at least half of study period Proportion of clients Total # of timely submissions

An RCE case study: Larimer County, CO 5 Analyze results: High-touch strategy 47.9% at least 1 timely month 54.1% at least 1 timely month

An RCE case study: Larimer County, CO Use evidence to inform decision-making Larimer County has moved to implementing an online form for MTS submission Interested in testing this with various incentives and/or messaging strategies to improve submission rates

Evaluation that serves program improvement Accessible process Building local research capacity Co-creative, collaborative approach between practitioners and researchers Decision-focused evaluation Encourages iteration and continuous quality improvement

Project IMPROVE: A resource Project sponsored by the Office of Family Assistance Goal: To improve TANF programs’ practices and capacities for using evidence, developing evaluation practices, documenting progress and processes, and revising program components by using evidence and evaluation data Technical assistance provided by Mathematica Policy Research Supports up to 5 “sites” per federal fiscal year

Project IMPROVE: A resource Federal fiscal year 2016-2017 sites California Colorado Maryland Washington (State) Tribal TANF learning community Federal fiscal year 2017-2018 sites TBD

RCE resources to support you The RCE Coach www.edtechrce.org A guide to opportunistic experiments Google search: “opportunistic experiments in human services” Join Project IMPROVE Talk with us after today’s presentation if you’re interested!

For more information Annalisa Mastri amastri@mathematica-mpr.com Jonathan McCay jmccay@mathematica-mpr.com Scott Baumgartner sbaumgartner@mathematica-mpr.com