1. United States Department of Labor Employment & Training Administration EVALUATING TAACCCT Kristen Milstead Region 2 TAACCCT Roundtable July 29-31, 2014

2. 2 Topics Overview of the National Evaluation Managing Your Third-Party Evaluation Group Activity

3. 3 Overview of the National TAACCCT Evaluation

4. National Evaluation Round 1 $500 million 49 grants Round 2 $500 million 79 grants Round 3 $474.5 million 57 grants Third-party evaluator required 4

5. National Evaluation: Objective To develop and implement a mixed-method evaluation design including an outcome analysis, a formal implementation analysis, and a systematic review of evaluation findings to better understand the TAACCCT grants program 5

6. National Evaluation: Research Questions Innovations in service delivery and/or system reform? Resulting in capacity building? How were models implemented? Can they be replicated? Areas for further research? 6

7. National Evaluation: Activities Implementation Analysis – Survey of all colleges – Site visits to selected grantees Outcomes Analysis Synthesis of third-party evaluation findings Support to third-party evaluators 7

8. National Evaluation: Working with Third-Party Evaluators House evaluation resources on SharePoint site Hold virtual roundtables provide peer-learning opportunities for smaller groups of third-party evaluators Provide semi-annual webinars on evaluation topics Hold evaluation workshops as a part of national DOL conferences for grantees and third-party evaluators (if feasible) 8

9. National Evaluation: Reports Interim Reports on TAACCCT Evaluation Findings – Round 1 grants (2015) – Round 2 grants (2016) – Round 3 grants (incorporated into Final report) Final Report (August 2017) – Analyze implementation and outcome data from all rounds implementation data and systematic review of third-party evaluation findings 9

10. For access to SharePoint site and notification of events, please email TAACCCTeval@urban.org TAACCCTeval@urban.org 10

11. 11 Managing Your Third-Party Evaluation

12. Managing Third-Party Evaluations: Most Important Thing! Ensure the most rigorous, independent evaluation possible, with both an implementation and impact/outcomes analysis 12

13. Managing Third-Party Evaluations: Expectations All SGA requirements are met (research questions, outcomes, reports, data, etc.) An implementation analysis that answers key research questions A participant impact/outcomes analysis that is of sufficient rigor (establishing causal inferences, if at all possible) 13

14. Managing Third-Party Evaluations: Deliverables Evaluation Plan – Round 2: Evaluation Plan All grants have been approved to implement evaluation plan – Round 3: Summary and Detailed Evaluation Plan All grants can work on evaluation plan, and must rely on the expertise of the third-party evaluator Some grants have been approved to implement evaluation plan Feedback on detailed evaluation plans Some grants will be required to submit a revised plan Interim Report Final Report 14

15. Managing Third-Party Evaluations: Tips for Implementation Analysis Research Questions: Consider additional implementation research questions that are tailored to your TAACCCT- funded activities in addition to the required research questions from the SGA Logic Model: Be sure to develop a logic model/theory of change with your third-party evaluator to ensure there is a common understanding of the TAACCCT-funded activities and the intended outcomes Capacity Measures: Work with your third-party evaluator to develop measures of how capacity was built at your institution(s) with your TAACCCT funds (e.g., additional class seats/programs, permanent hires, new technology) 15

16. TAACCCT Conceptual Framework 16 Current Workforce Needs TAACCCT Activities Capacity-Building Outputs Long-Term Outcomes TAA-eligible and other workers - Need training for new or upgraded skills Education and Training Program Development TAACCCT Core Elements Evidence-based design Stacked and latticed credentials Online and technology-enabled learning Transferability and articulation Strategic alignment Student/Participant Education Outcomes Enrollment Persistence Completion Credentials Establishing Innovative Education and Training Strategies Expand student services Train and hire staff Upgrade supplies/equipment Build/improve technology infrastructure Add new facilities/renovate existing facilities Develop articulation agreements Build/improve evaluation capacity Develop partnerships with employers, other institutions, and the community Improved student/participant employment and earnings outcomes Skilled workforce to meet employer demand Well-connected, efficient systems in community/region Increased employer involvement Effective training programs at community colleges with capacity to serve students Contextual Factors: Economic Conditions and Policy/Budget Climate Systems - Need to be better connected and more integrated Community Colleges - Need to improve capacity to train students for high- demand occupations Employers – Need for skilled workers Transformation of training for adult workers

17. Managing Third-Party Evaluations: Tips for Implementation Analysis Key Informants: Help your third-party evaluator identify key informants/ respondents (e.g., program staff, students, partners) and inform them of the evaluation purpose and activities Check-Ins: Build in timely check-ins with your third-party evaluator to understand whether the TAACCCT-funded activities are being implemented as designed (i.e., fidelity to the model) Program Changes/Improvements: If changes to the original model have occurred, work with your third-party evaluator to make sure that the planned impact/outcomes analysis is not affected by these changes. 17

18. Managing Third-Party Evaluations: Tips for Impact/Outcomes Analysis Methods: Be educated consumers of the methods selected by your third-party evaluator to assess the impact of your TAACCCT-funded activities on participants and the level of evidence of effectiveness the evaluation is likely to produce for DOL and the field – Weak evidence: outcomes only study (with no comparison group or a non-matched comparison group) – Moderate evidence: quasi-/non-experimental study (with matched comparison group) – Strong evidence: experimental design (with random assignment) 18

19. Managing Third-Party Evaluations: Tips for Impact/Outcomes Analysis Comparison Group(s): Assist your third-party evaluator in identifying appropriate student comparison groups for your TAACCCT-funded activities – Could be historical cohorts or concurrent cohorts within the same institution or other institutions and in a similar program (by occupation, for example) but not receiving the intervention – Appropriate = having characteristics, including motivation and ability, that are similar but for the intervention and having sufficient numbers to conduct the impact analysis 19

20. Managing Third-Party Evaluations: Tips for Impact/Outcomes Analysis Data: Serve as an intermediary with departments within your institution and with other institutions or government agencies to help your third-party evaluator access needed administrative data (often better than survey data, esp. for employment outcomes) – Help them develop relationships with your IR office to access student records – Develop data sharing agreements with the state labor department to access unemployment insurance wage records – Connect them with other institutions that may be able to provide student comparison group data 20

21. 21 Group Exercise – Addressing Evaluation Issues

22. Group Exercise Create a group of 4-6 people, with people who are not from your institution Read and discuss how your group might address the evaluation issue provided for 10 minutes Assign a group member to report out to the audience about your potential solutions and/or strategies to develop solutions 22

23. Discussing the Issue Recognize that there are probably no perfect solutions and you may want to come up with more than one solution, depending on what might work for your institutional circumstances. Think creatively about who you might reach out to – within your institution, employer partners, other institutions, other contacts in government – to help you problem-solve. 23

24. Evaluation Issue #1 Your third-party evaluator provides formative feedback from their early evaluation work indicating that the students being recruited into the TAACCCT programs are different from what was originally planned. Now, the comparison groups the evaluator wanted to use do not seem appropriate, jeopardizing the impact analysis. How do you help them identify alternatives and/or reconsider your recruitment efforts? 24

25. Evaluation Issue #2 Your third-party evaluator plans to conduct a student survey but they are concerned about ensuring a high response rate, especially for students that may no longer be at the institution. How can you help them find and encourage the students to complete the survey? What things could you and your staff do to help early in the project ensure your evaluator can reach these students? Are there non- monetary incentives that could be offered? 25

26. Evaluation Issue #3 Staff at the Institutional Research Office are concerned about your evaluator using individual-level student data. How do you address the privacy concerns? Are there others at your institution or at the state level that could address this issue? Are there things you could do early in the project to obtain informed consent for the research? 26

27. Contact Information Questions about the evaluation: TAACCCT@dol.gov (copy your FPO) TAACCCT@dol.gov To obtain access to the Sharepoint website: TAACCCTeval@urban.org 27

28. 28