Rapid cycle evaluation:

Slides:



Advertisements
Similar presentations
Job Search Assistance Strategies Evaluation Presentation for American Public Human Services Association February 25, 2014.
Advertisements

Decision-Making and Strategic Information Workshop on M&E of PHN Programs July 24-August 11, 2006 Addis Ababa.
Evaluation. Practical Evaluation Michael Quinn Patton.
PHAB Slide Set 2013 The slides in this set are made available for use in presentations and educational sessions by health departments. The information.
Virginia Learning Collaboratives Reducing Family Homelessness in Virginia: A Rapid Re-Housing Approach.
Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7 th – 9 th, 2013.
Client Assessments and the Reemployment of Low-Income Workers: Lessons from the Field OWRA: An Online Tool for Supporting Self-Sufficiency Chicago, Illinois.
Family Service System Reform Grant Application Training Video FY Donna Bostick-Knox, Pennsylvania Department of Public Welfare, Office of Children.
Developing Evidence on “What Works” in Moving TANF Recipients to Work through Job Search Assistance Karin Martinson, Abt Associates February,
The CSEFEL and State Implementation Partnerships An Overview.
Using Logic Models in Program Planning and Grant Proposals The Covenant Foundation.
Mathematics Performance Tasks Applying a Program Logic Model to a Professional Development Series California Educational Research Association December.
1 YOUTHBUILD EVALUATION Building evidence about the effect of YouthBuild on the young people it serves August 17, 2011 Washington, D.C.
Session 7: Planning for Evaluation. Session Overview Key definitions:  monitoring  evaluation Process monitoring and process evaluation Outcome monitoring.
[Presentation location] [Presentation date] (Confirm ABT logo) Building Bridges and Bonds (B3): An introduction.
DAC Data Analytics Using State and Local Data to Improve Results.
Wisconsin Personnel Development System Grant Click on the speaker to listen to each slide. You may wish to follow along in your WPDM Guide.
CIAS Program Level Assessment Office of Educational Effectiveness Assessment September 6, 2016.
PYRAMID PLUS APPROACH: THE “WHAT’S NEW REPORT”FOR EARLY INTERVENTION COLORADO Webinar: March 3 rd, :00-1:00 pm.
Chapter 6 Selecting a Design. Research Design The overall approach to the study that details all the major components describing how the research will.
First Things First Grantee Overview.
Presenter Disclosures
Logic Models How to Integrate Data Collection into your Everyday Work.
AdvancED Accreditation External Review October 23-26, 2016
A Forum at the Center for Improving Research Evidence (CIRE)
School Psychologists in Multi-Tiered Systems of Support: A Discussion of Roles, Challenges, and Successes Elena Diamond, Lewis & Clark College Angela Whalen,
Mason County Schools Policy 5310 August 11, 2016.
Evidence Based Practices in Napa County Probation
Community Project Overview
Supporting Community Priorities and Emphasizing Rigor An Approach to Evaluation Capacity Building with Tribal Home Visiting Programs Kate Lyon, MA Julie.
UMRG 3rd Learning Session: July 19th, 2012
Kate Lyon, MA, James Bell Associates, Inc.
MUHC Innovation Model.
Professional Development BOOT CAMP
Sherry Deren, Sung-Yeon Kang, Milton Mino & Honoria Guarino
Introduction to Program Evaluation
Doing more with less: Evaluation with the Rapid Cycle Evaluation Coach
Zelphine Smith-Dixon, State Director of Special Education
Overview – Guide to Developing Safety Improvement Plan
Transforming Hidalgo County CSCD into an Evidence Based Agency
Overview – Guide to Developing Safety Improvement Plan
America’s Promise Evaluation What is it and what should you expect?
Innovative Recruitment Strategies: Tools for Your Recruitment Toolbox
Agenda Welcome and Introductions Purpose of Investment
Monitoring and Evaluation using the
CHAMPS: CHanging Attitudes and Motivation in ParoleeS
What can we learn from small pilots conducted by school districts?
Differentiated Supports in Special Education
Local Control Accountability Plan Input
Mason County Schools Policy 5310 August 11, 2016.
Response to Intervention (RTI)
Leveraging Evaluation Data: Leading Data-Informed Discussions to Guide SSIP Decisionmaking Welcome Mission of IDC- we provide technical assistance to build.
Primary Prevention Institute
Introductions Introduction
Introduction Introduction
Effectively Training Parents in Behavior Analytic Interventions
A Family Support Approach to Family Support Evaluation
Regulated Health Professions Network Evaluation Framework
Leveraging Performance Management to Support School Priorities
Introductions Introduction
School-Wide Positive Behavioral Interventions and Supports (SWPBIS)
Introductions Introduction
What is new in the Sphere Handbook 2018 and how to get benefit from it
Disproportionality Prevention Support
Using Logic Models in Project Proposals
Introduction Introduction
Some Further Considerations in Combining Single Case and Group Designs
Using State and Local Data to Improve Results
Introductions Introduction
Framing Grants for policy Research
Presentation transcript:

Rapid cycle evaluation: An approach to evidence-based decision-making in TANF programs Presentation at the 2017 National TANF State Directors’ Meeting Arlington, VA Tuesday, August 15, 2017 Jonathan McCay  Annalisa Mastri

Overview of today’s session Introduction and overview of evaluation approaches RCE case study from Larimer County, CO Project IMPROVE and other resources

Traditional program evaluation Researcher-driven and knowledge-focused Answers the question: Does “it” work? Focused on internal validity and can be designed to generalize results to other contexts Often a multi-year process

Rapid cycle evaluation Focused on program improvement Therefore, an RCE is implementer-driven Measures the impact of changes to program operations or services Answers the question: Can “it” be improved? Uses rigorous research methods to identify a causal link between the change(s) and the outcome(s) Tailored to inform decision-making Relies predominantly on existing program data to measure impacts Short time horizon Focus on results that can be observed in a relatively short timeframe

Traditional Program Evaluation A shift in purpose Traditional Program Evaluation Rapid Cycle Evaluation Often… Uses analytic methods Implementer-driven Researcher-driven & knowledge-focused Can “it” be improved? Does “it” work? Builds knowledge & evidence Tailored to inform decision-making Generalized Multi-year A year or less

Situations well-suited to RCE Strategy to improve engagement and follow-through Particularly, behavioral “nudges” Variation in the combination, dosage, or intensity of services Operational changes to improve efficiency Different communication and marketing approaches Effectiveness of an oversubscribed program Using a random lottery selection process

Is RCE feasible? What, specifically, do you hope to achieve? When do you expect it to affect change? What data do you have available to measure the change(s)? Can we find or create a good comparison between those who receive X and those who do not?

The RCE process

An RCE case study: Larimer County, CO 1 Diagnose the problem Monthly Time Sheet (MTS) is the required reporting form for Works Program (TANF) participants Documents hours by activity for the work participation rate As of December 2015, about 1 of every 2 participants submitted their MTS on time Considerable amounts of staff time spent on case conciliation due to low submission rate Families at risk of sanction or case closure Admittedly, the program did very little to communicate to participants about the MTS upfront

An RCE case study: Larimer County, CO 1 Diagnose the problem Misunderstanding Procrastination Forgetting

An RCE case study: Larimer County, CO 2 Define the question & population Does a low-touch or a high-touch behavioral intervention increase on-time MTS submissions? Low-touch approach with existing TANF participants High-touch approach with new TANF participants

An RCE case study: Larimer County, CO 3 Define the intervention Intro brochure + Planning email + Reminder postcard + Prize lottery for on-time submitters Low-touch approach Variation in high-touch approach

An RCE case study: Larimer County, CO 4 Conduct test County-based tracking data MTS submission data Participant data in the Colorado Benefits Management System Cloud-based dashboards for random assignment and selecting prize lottery winners

An RCE case study: Larimer County, CO 4 Conduct test Treatment Receives postcard 221 participants Random assignment Low-touch RCE with existing participants Control Business as usual 227 participants Enhanced Treatment Basic + prize lottery 159 participants Random assignment High-touch RCE with new participants Basic Treatment Brochure, email, postcard 144 participants Control Business as usual 147 participants

An RCE case study: Larimer County, CO 5 Analyze results: Low-touch strategy 36.3% timely at least half of study period 38.3% timely at least half of study period Proportion of clients Total # of timely submissions

An RCE case study: Larimer County, CO 5 Analyze results: High-touch strategy 47.9% at least 1 timely month 54.1% at least 1 timely month

An RCE case study: Larimer County, CO Use evidence to inform decision-making Larimer County has moved to implementing an online form for MTS submission Interested in testing this with various incentives and/or messaging strategies to improve submission rates

Evaluation that serves program improvement Accessible process Building local research capacity Co-creative, collaborative approach between practitioners and researchers Decision-focused evaluation Encourages iteration and continuous quality improvement

Project IMPROVE: A resource Project sponsored by the Office of Family Assistance Goal: To improve TANF programs’ practices and capacities for using evidence, developing evaluation practices, documenting progress and processes, and revising program components by using evidence and evaluation data Technical assistance provided by Mathematica Policy Research Supports up to 5 “sites” per federal fiscal year

Project IMPROVE: A resource Federal fiscal year 2016-2017 sites California Colorado Maryland Washington (State) Tribal TANF learning community Federal fiscal year 2017-2018 sites TBD

RCE resources to support you The RCE Coach www.edtechrce.org A guide to opportunistic experiments Google search: “opportunistic experiments in human services” Join Project IMPROVE Talk with us after today’s presentation if you’re interested!

For more information Annalisa Mastri amastri@mathematica-mpr.com Jonathan McCay jmccay@mathematica-mpr.com Scott Baumgartner sbaumgartner@mathematica-mpr.com