Net Impact Evaluation Framework For Minnesota With preliminary results as of May 2014 A Standardized

Getting Started Nick Maryns Senior Policy Analyst Governor’s Workforce Development Council Raymond Robertson Professor of Economics Macalester College

Motivations, History, Partners Overview and Basic Parameters Pilot ProjectEvaluation Design Standardized Net Impact Evaluation Framework Preliminary Results

Motivations, History, Partners “Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.” -Peter Orszag and John Bridgeland, The Atlantic Monthly, July 2013 Motivations History Advisory Group

Motivations, History, Partners Motivations History Advisory Group The National Conversation Pew-MacArthur Results First Initiative Results for America Social Impact Bonds / Pay for Success

Motivations, History, Partners Motivations History Advisory Group Apples and Oranges Approaches across the State

Motivations, History, Partners Motivations History Advisory Group The UPAM law required the development of uniform ROI measure.

Motivations, History, Partners Motivations History Advisory Group (d) Functions. The State Board shall assist the Governor in— (6) development and continuous improvement of comprehensive State performance measures, including State adjusted levels of performance, to assess the effectiveness of the workforce investment activities in the State as required under section 2871 (b) of this title; -Section 111 of the Workforce Investment Act Subd. 3. Purpose; duties. (c) “Advise the governor on the development and implementation of statewide and local performance standards and measures relating to applicable federal human resource programs and the coordination of performance standards and measures among programs” - Minnesota Statute 116L.665 Subd. 3c The GWDC’s Role

Motivations, History, Partners Motivations History Advisory Group Community Organizations Greater Twin Cities United Way Lukeworks Twin Cities RISE! State Agencies Department of Employment and Economic Development Department of Corrections Department of Education Department of Human Services MN State Colleges and Universities Researchers / Evaluators Anton Economics Invest in Outcomes Macalester College Minneapolis Federal Reserve Bank Wilder Research Local Workforce Boards City of Minneapolis Employment and Training Program Minnesota Workforce Council Association Workforce Development, Inc. Business / Employers Dolphin Group MN Chamber of Commerce

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Goal A framework for measuring and understanding the net impacts and social ROI of all publicly-funded workforce programs that is standardized and credible, and that informs strategy and continuous improvement

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Supportive Policies and Infrastructure Cost-Benefit Analysis Net Impact Analysis Oversight / Management Framework

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Manageable, feasible to administer Useful, relevant, timely Credible, transparent, trusted Adaptable, sensitive to change

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Improving Services, Driving Value “What works, and for whom?” “What disparities exist?” Making Smarter Investments “How do current investments align to what works, and to disparities in our community?” Communicating Value “How do workforce services benefit participants and taxpayers?” Standardizing the Approach Strengthening Transparency/Accountability

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Publicly-administered and funded workforce programs DEED and other state agencies Non-profit passthroughs Public education (elements of K-12 and PS) State and federal competitive grants and special initiatives Programs serving targeted populations (e.g. people with disabilities, veterans) Long-Term Vision Independent nonprofits and education providers Other service areas

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs United Way Wilder Research Invest in Outcomes/ State Pay for Performance National Governors Association

Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs One methodology, many programs Ensuring usefulness to program managers and policy makers (Unintended) incentives created by measure

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Policy Framework Management Framework Cost-Benefit Analysis Net Impact Analysis

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Costs Benefits Break-Even Point ROI = (Benefits – Costs) / Costs Time $ (Return)(Investment)

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Employment Income / Fringe Benefits Taxes Income / Payroll / Sales Public Assistance Savings MFIP / SNAP / UI Healthcare Savings MinnesotaCare / Medical Assistance Incarceration Avoidance

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Program Costs Time-weighted / Service-weighted (where possible) Cost to Participant

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Benefits and Costs to Participants + Benefits and Costs to Taxpayers = Total Social Benefits and Costs

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Some public benefits Subsidized housing costs Prescription Drug Program costs Child Support payments Other important but difficult-to-quantify effects Change in mental and physical health Change in worker productivity Reduction in criminal activity Economic multipliers

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Cloning Randomized Trials Causality / True Attribution Approaching Kernel Density Propensity Score Matching Difference-in-Difference Estimator “As good as random”

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Foundation: administrative data at the individual level Avoid broad assumptions wherever possible Not Used: Self-reported program performance indicators, e.g. entered employment rate six-month retention rate earnings change

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Net Impact Comparison Group Treatment Group Earnings Time Not the relevant comparison

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Accounts for Many Factors Personal Characteristics Geography Local Economic Conditions Services Received Also allows us to analyze performance by these categories

Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Contextualized Performance Goals Adjusted for population served, local conditions Leading Indicators For near-term relevance; based on statistical relationships between near-term indicators and long-term outcomes

Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Purpose Primarily for internal use, to test concept, methodology, data process The pilot evaluation comprises 950,000 individuals and roughly 50 million data points

Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Initial Cohorts ( and ) WIA Adult Program WIA Dislocated Worker Program Twin Cities RISE! New Cohorts ( ) FastTRAC I&B Grantees MFIP / DWP Employment Services Adult Basic Education SNAP Employment and Training

Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Registrants at WorkForce Centers and on MnWorks.net Unemployment Insurance Applicants

Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe

Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe New round of Data Sharing Agreements recently finalized Data are currently coming in Results this Fall

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Preliminary results address earnings and employment impacts across two programs: WIA Adult Dislocated Worker (both WIA and MN) Treatment cohorts are defined as such: Cohort WIA Adult 0708 WIA Adult 0910 DW 0708 DW 0910 Exit Dates July 2007 – June 2008 July 2009 – June 2010 July 2007 – June 2008 July 2009 – June 2010 Additionally, some initial findings on FastTRAC data are also provided. See disclaimer to the left.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next The analysis that has produced the following preliminary results has been guided by the GWDC Net Impact Advisory Group and is still under development. The preliminary results that follow have been reviewed by program directors and relevant staff at DEED, who emphasized the value of the findings and voiced their support for the continuation of the effort.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals. INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals. INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals. INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals. INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 1a: WIA AD 0708 Kernel Density Distribution of Pre Wages Figure 1b: WIA AD 0910 Kernel Density Distribution of Pre Wages INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer. INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 2a: DW 0708 Kernel Density Distribution of Pre Wages Figure 2b: DW 0910 Kernel Density Distribution of Pre Wages INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer. INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: This table tells us the average time in program is between three quarters and a year, with a lot of variation. INTERPRETATION: This table tells us the average time in program is between three quarters and a year, with a lot of variation.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 3: Unmatched Wage Distribution: WIA Adult 0708 INTERPRETATION: In the earnings charts that follow, 0 represents time of enrollment. We worked to match earnings in the pre- period. The net impact on earnings is the average difference in the post-period, specifically quarters 5-8. INTERPRETATION: In the earnings charts that follow, 0 represents time of enrollment. We worked to match earnings in the pre- period. The net impact on earnings is the average difference in the post-period, specifically quarters 5-8. Matched AverageNet Impact

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 3: Unmatched Wage Distribution: WIA Adult 0708 INTERPRETATION: For WIA AD 0708, the results at right translate to a net 30% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress. INTERPRETATION: For WIA AD 0708, the results at right translate to a net 30% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 4: Unmatched Wage Distribution: WIA Adult 0910 INTERPRETATION: For WIA AD 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress. INTERPRETATION: For WIA AD 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 5: Unmatched Wage Distribution: Dislocated Worker 0708 INTERPRETATION: For DW 0708, the results at right translate to a net 52% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For DW 0708, the results at right translate to a net 52% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 6: Unmatched Wage Distribution: Dislocated Worker 0910 INTERPRETATION: For DW 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For DW 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 7: Unmatched Employment Distribution: WIA Adult 0708 INTERPRETATION: For AD 0708, the results at right translate to a net 30% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For AD 0708, the results at right translate to a net 30% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 8: Unmatched Employment Distribution: WIA Adult 0910 INTERPRETATION: For AD 0910, the results at right translate to a net 29% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For AD 0910, the results at right translate to a net 29% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 9: Unmatched Employment Distribution: Dislocated Worker 0708 INTERPRETATION: For DW 0708, the results at right translate to a net 6% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For DW 0708, the results at right translate to a net 6% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 10: Unmatched Employment Distribution: Dislocated Worker 0910 INTERPRETATION: For DW 0910, the results at right translate to a net 5% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant. INTERPRETATION: For DW 0910, the results at right translate to a net 5% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next A statistical analysis of FastTRAC is forthcoming; FastTRAC data have presented a number of challenges, many of which illustrate common data challenges we face. Funded as a pilot project through the Joyce Foundation, the MN FastTRAC model was not initially designed to measure outcomes based on placement, but instead focused on educational attainment among a hard-to-serve population (likely MFIP participants). The model also allowed flexibility across local service providers, which created greater differences in self-reporting outcomes by each provider.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Specifically, gathering data on program participants has presented the following challenges: 1.Participant data did not require entry into one database but relied on local systems. Data entry is now entered into WF1. – Some participants are excluded from data altogether depending on program completion, placement, or continuation of their academic program. – Entrance/exit dates may be defined inconsistently across programs – Program activities/services may be used and/or defined inconsistently (trying to adapt to other programs within WF1) 2.FastTRAC participants are characterized in part by the multiple barriers they face; accordingly, it may be more difficult to find strong control group matches for them. 3.Small sample sizes and variance among FastTRAC participant characteristics make statistically significant results harder to obtain.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next As FastTRAC has evolved, the pilot recognized challenges with collecting data to measure program impacts. Progress is being made to make data collection practices more complete and consistent across FastTRAC programs. Data practices are improving, but it will take time for those changes to be reflected in net impact analyses since those analyses require at least one year of post-enrollment data.

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Further analysis is currently underway. Here’s what to expect: 1.Further refinement of matching and estimation, to improve the statistical significance of the results. 2.Additional net outcomes measured over longer time periods, including usage of public benefits, reincarceration, and associated monetary (ROI) impacts. 3.Additional programs to be analyzed, including FastTRAC, Adult Basic Education, MFIP Employment Services, and SNAP Employment and Training. 4.Results disaggregated by participant characteristics (e.g. race, gender) and other factors.

Wrapping Up Discussion and Questions Raymond Robertson Professor of Economics Macalester College Nick Maryns Senior Policy Analyst Governor’s Workforce Development Council