1. ROI Committee 2 nd Meeting November 1, 2013

2. Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Preliminaries Agenda Meeting Objectives: Review and understand committee process and key decision points Review and approve evaluation design developed by the advisory committee, staff, and evaluator Make initial decisions on ROI reporting/dashboards 9:00Welcome and Introductions 9:10Committee Process and Decision Points 9:20 Evaluation Design 10:00Audiences and Dashboards 10:50 Wrap-Up and Next Steps 11:00Adjourn

3. Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Preliminaries Review and understand work of the ROI Advisory Committee, staff, and evaluator Receive and consider recommendations from the advisory group and staff Make final recommendations to the full council for approval

4. Preliminaries Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Evaluation Design Policy Recommendations Data Sharing, Funding, Incentives, etc. Management Functions How is ROI used and reported? Governance Who administers/audits/certifies/ reports/interprets ROI?

5. Preliminaries Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure October Introduction to ROI November – December Discussion and Decisions January Finalize Findings / Recommendations February Present to Full Council, Discuss, and Vote

6. Preliminaries Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Inez Wildwood, Chair Mo Amundson Rep. Karen Clark Katie Clark Sieben (alternate: Cynthia Bauerly) Steve Ditschler Frank Forsberg (alternate: Brian Paulson) Sen. John Hoffman Eric Kaler (alternate: Mary Nichols) Mary Russell Nicholas Maryns, Staff

7. Preliminaries Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Attend all meetings; listen/watch webinars if a meeting is missed Actively participate, do any reading/assignments between meetings Express your opinions; engage in thoughtful discussion and debate Disagree respectfully Act as an “ambassador” for the ROI Initiative; share what you’ve learned

8. Preliminaries Committee Purpose Key Decision Points Timeline Membership Responsibilities Procedure Meeting Attendance No quorum needed to hold a meeting Voting Quorum needed to hold an in-person/conference call vote; Majority of quorum needed for approval; Votes can be held over email over an established timeframe Others?

9. Overview and Basic Parameters Values Objectives of ROI Scope of Programs Manageable, easy to administer Useful, relevant and timely Understandable, transparent Credible, trusted Adaptable Sensitive to change

10. Overview and Basic Parameters Smarter Investments Improving Services Strengthening Accountability/Transparency Communicating Value Standardizing Results Values Objectives of ROI Scope of Programs

11. Overview and Basic Parameters Publicly-administered and funded workforce programs DEED and other state agencies Non-profit passthroughs Public education (elements of K-12 and PS) State and federal competitive grants and special initiatives Programs serving targeted populations (e.g. people with disabilities, veterans) Long-term Vision Independent nonprofits and education providers Other service areas Values Objectives of ROI Scope of Programs

12. Evaluation Design TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency Policy Framework Management Framework Cost-Benefit Analysis Regression Analysis

13. Evaluation Design Staff Recommendation: Approve of the evaluation design developed by the Advisory Group, evaluator, and staff. TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

14. Evaluation Design Directly Observed Benefits (no assumptions needed) Employment Income / Employment Status Public Assistance Savings MFIP / SNAP / UI TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

15. Evaluation Design Partially Imputed Benefits (requiring some assumptions) Employment Fringe Benefits (healthcare, retirement, vacation): 20% of gross wages Taxes Income Tax: use estimators like TaxSIM and those developed by the MN Dept. of Revenue Payroll Tax: change in earnings * statutory rate Sales Tax: change in earnings * average marginal sales tax rate as provided by DOR Tax Incidence Studies TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

16. Evaluation Design Partially Imputed Benefits (requiring some assumptions) Healthcare Savings MinnesotaCare and Medical Assistance: observe enrollment, use average monthly costs at different eligibility levels provided by DHS Incarceration Avoidance Change in costs derived from differences in incarceration and recidivism rates. Cost per inmate equals length of stay (using averages as needed) * marginal per diem cost per inmate. Note: We currently track felony-level offenses; misdemeanors should be added in a future iteration. TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

17. Evaluation Design Program Costs Time-weighted / Service-weighted Cost to Participant TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

18. Evaluation Design Savings associated with misdemeanor offenses Some public benefits Subsidized housing costs Prescription Drug Program costs Child Support payments Other important but difficult-to-quantify effects Change in mental and physical health Change in worker productivity Reduction in criminal activity Economic multipliers An important corollary: additional types of benefits should be considered for inclusion in future iterations. TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

19. Evaluation Design No economic multiplier No projection of impacts beyond timeframes we observe TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

20. Evaluation Design Benefits and costs are estimated from two distinct perspectives: The Participant The Taxpayer / Government Together, the two add up to total social return on investment TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

21. Evaluation Design Benefits and costs are estimated from two distinct perspectives: The Participant The Taxpayer / Government Together, the two add up to total social return on investment TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

22. Evaluation Design All results are based on a net-impact methodology, i.e. using control groups Difference-in-Difference Kernel Density Propensity Score Matching (“as good as random”) TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

23. Evaluation Design Net Impact Comparison Group Treatment Group Earnings Time Not the relevant comparison TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

24. Evaluation Design Treatment Cohorts Defined by Time of Entrance Control Groups Unemployment Insurance Applicants MnWorks.net and WFC Registrants In Future Iterations MFIP Recipients ABE Enrollees (not participants) TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

25. Evaluation Design Contextualized Performance Targets Adjusted for population served, local conditions Leading Indicators For near-term relevance; based on statistical relationships between near-term and long-term indicators TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

26. Evaluation Design Near Term Average of quarters 2-3 after exit Medium Term Average of quarters 5-6 after exit Long-Term Average of quarters 9-12 after exit TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

27. Evaluation Design Complete methodology and data analysis process should be published with as much transparency as possible, up to current standards in the econometrics literature. TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

28. Evaluation Design All analyses should be accompanied with a full set of caveats with regard to methodology and interpretation TOPICS Cost-Benefit Analysis Benefits Costs What’s Not Included B/C Perspectives Regression Analysis Net Impact Cohorts Other Features Timeframes Transparency

29. ROI in Practice Policy Framework Management Framework Cost-Benefit Analysis Regression Analysis DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

30. ROI in Practice How to make information useful? Which questions & decision points should the dashboard inform? For which audiences? Frequency of reporting/ analysis/review Simplicity vs. Nuance DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

31. ROI in Practice Audiences More FormativeMore Summative Providers “What drives value?” Investors* “What are we getting for our money?” * Broadly defined; can include legislators, taxpayers, and philanthropy DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

32. ROI in Practice Object of Evaluation More SpecificMore Broad Services Funding Streams * Broadly defined; can include legislators, taxpayers, and philanthropy Programs Providers Target Populations DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

33. ROI in Practice Contextual Questions General Performance Questions Questions about Disparities Explanatory Questions Current and Future Performance Methodological Questions DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

34. ROI in Practice Program Over Time Contextualized Targets Similar Programs How direct of a comparison? How is “similar” defined? Population Served Programmatic Goals Intensity of Service (cost, duration, treatment) Training vs. Non-Training High- vs. Low-Intensity Programs DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

35. ROI in Practice Dynamic, Online Dashboard Print Reports Primary Report 1.Performance Across Whole System, Including Major Funding Streams 2.Performance by Service Type (“What works?”) 3.Performance Across Target Populations (“How well are we serving _____?”) Individual Program Reports Technical Appendices DASHBOARD TOPICS Big Questions Audiences Objects of Evaluation Categories of Questions Comparisons Reporting

36. Discussion and Next Steps Upcoming Meetings Thoughts and Questions?

37. Discussion and Next Steps Upcoming Meetings ROI Committee Webinar #3 Friday, November 15 9:00 am – 11:00 am ROI Advisory Group Meeting Wednesday, November 6 1:00 pm – 3:00 pm DEED

38. Motivations, History, Partners Overview and Basic Parameters Pilot ProjectROI in PracticeEvaluation Design Preliminaries Next Steps The Return on Investment Initiative