April 7, 2019 Peer Learning Community Forum: Using the Evaluation Readiness Assessment Tool Evaluations and Research Webinar Series [#2-Oct. 16, 2018] 3:00 PM – 4:30 PM

Gloria Salas-Kos Senior Research and Evaluation Analyst Employment and Training Administration [ETA] U. S. Department of Labor [DOL]

New Community of Practice Access to evaluation and research tools and resources Tailored to meet your needs! Flash Webinars Peer Learning Opportunities

Welcome new and returning members Explain evaluation peer learning context Present recap of evaluation readiness Learn from peers and share experiences with evaluation readiness: Strengths Opportunities for improvement Next steps

Did you attend the Evaluation Flash Webinar on October 2? Yes No

Did you have a chance to use the Evaluation Readiness Assessment Tool? Yes, I completed the assessment. Yes, I completed part of the assessment. I reviewed the tool but have not yet begun the assessment. No, I have not reviewed or used the tool.

Welcome New and Returning Members! States represented: Alabama Arizona Idaho Illinois Kansas Louisiana Texas Utah Washington

Wayne Gordon Karen Armstrong Director, Division of Research and Evaluation/ Office of Policy Development and Research DOL-ETA Karen Armstrong Project Director, Dissemination and Technical Assistance IMPAQ International

Lori Frazier Bearden Jennifer Bishop Assistant Director of Workforce Programs Alabama Department of Commerce Jennifer Bishop Dissemination Lead IMPAQ International

Why Use Evaluation Readiness Assessment? Wayne Gordon Director, Division of Research and Evaluation/ Office of Policy Development and Research DOL-ETA

Mission: Support state workforce agencies in their efforts to use and choose evaluation and research Identify Needs Tools and Resources Identify Disseminate Apply Create Evidence Peer Learning Community Shared Wisdom TA Products for Evaluation Practices Disseminate Evaluation Results Use Choose

Why Use the Evaluation Readiness Assessment Tool? Gauge readiness Spark enthusiasm Celebrate strengths Identify areas for improvement Foster evaluation culture Integrate evaluation into state activities

Why participate in the evaluation and research PLC? Exchange ideas and strategies with peers for building evaluation capacity Learn about latest evaluation reports and tools from DOL Provide DOL continued feedback on how best to support states and localities

Flash Webinar Recap: Navigating the Readiness Assessment Tool Karen Armstrong Project Director, Dissemination and Technical Assistance IMPAQ International

The Five Elements: Digging Deeper

Key Evaluation Elements 1 Evaluation culture and awareness Do agency staff and partners understand the benefits of evaluation, use evidence-based results to inform decisions, and plan to conduct evaluations to add to the existing evidence base? 2 Funding strategies Does the state or region actively pursue funding for and invest in evaluations? 3 Data management Does the agency have adequate operational capacity, IT infrastructure, and policies and procedures for collecting and using data for evaluations? 4 Staff skills, capacity, and knowledge Does the agency have sufficient staff with the skills, knowledge and experience needed to conduct or oversee third-party evaluations? 5 Strategic planning Does the state have a comprehensive strategic plan that includes evaluation as an integral part of the plan?

Rate Progress and Access the Information You Need

Rate Progress and Access the Information You Need

Challenge for PLC members Share with your staff and partners. Independently complete the tool. There are no wrong answers! Compare and explore responses: What are your strengths? What are your opportunities for improvement? Discuss next small and achievable steps. Source: https://sites.google.com/site/rtisurveyofexceptionalities/home/c-assessment

Another State’s Perspective: Modeling the Assessment in Alabama Lori Frazier Bearden Assistant Director of Workforce Programs Alabama Department of Commerce

Putting the Evaluation Readiness Assessment Tool to Work “This is a great tool. It took less time than I thought it would.”– WIOA Performance Manager “Sorry. I cannot provide evaluation feedback.” - Labor Market Information Division The tool was provided via email to the WIOA Performance Unit, WIOA State Board Unit, Labor Market Information Division, and Adult Education Division Discussion Follow-up occurred

And the Survey Says…..

Digging a little deeper….. Organizational Weakness Organizational Strengths Lowest Rating was Section 4: Staff Skills, Capacity, and Knowledge with an Average Score of 2 Need for Improvement: Staff Capacity “The Workforce Development Division is not large enough to have a Research & Evaluation Unit’ Highest Rating was Section 3: Data Management with an average score of 3 “The State has data sharing agreements to facilitate interstate exchange of UI wage record data…” was given the highest rating of all Sections.

So what do we do now? Using this tool for Improvement This survey tool served as a “starting point” to engage in dialogue of our shortfalls and successes in research and evaluation Next Steps include: Examining funds for evaluation Developing Key Evaluation Priorities & Next Steps No Statement in any Section of the Assessment achieved an average rating of 4 or 5

Please choose the statement that best represents your state/locality (select all that apply). We have a research and evaluation unit (or its equivalent) that has 7 or more staff devoted to it. We have a research and evaluation unit with 4-6 staff members. We have a research and evaluation unit with 2-3 staff members. We have a research and evaluation unit with only 1 staff member. We do not have a unit that is specifically focused on research and evaluation.

Which statement below best represents your state/locality’s research and evaluation (R&E) activities? We use contractors for all our R&E activities. We mostly use contractors for our R&E activities. We mostly conduct R&E activities in-house, but have a few R&E contracts. We do all our R&E work in-house. We are currently not conducting any R&E activities.

Peer Questions

Your Experience with Evaluation Readiness Jennifer Bishop Dissemination Lead IMPAQ International

Discussing Your Experiences and Observations PLC Members will now have the opportunity to share their experiences. We have assigned you to small breakout groups to: Share what you learned from the assessment tool Ask each other questions about different components to help understand the barriers towards improvement and explore ways to address barriers Challenge each other to take next steps Refer to the Evaluation Readiness Assessment tool (download from the files window).

The Small Group Process You will be routed to a new conference ‘room’ with no more than eight participants. Each room will also have a facilitator, a technical lead, and a note taker. Facilitator will guide the discussion. Note taker will record your observations. Last five minutes, everyone will review notes to make sure the highlights were captured. Facilitator (or a volunteer from the group) will report back to the large group.

Small Group Discussions Focus on Evaluation Activities in your States or Local Areas

Large Group Discussion Share highlights and next steps

Peer Learning Community Takeaways What were some conversation highlights? What were some next steps that groups came up with? What would be a helpful follow-on to this meeting?

Open Discussion

Look for WorkforceGPS invitations to the next evaluation and research series Flash Webinar: Evaluation Design, Research Questions and Methods: An Assessment Toward Implementation (30 minutes) Wednesday, November 29, 3:00-3:30 ET Learning Assignment: How logic models define research questions and evaluation designs (2 weeks) Nov 29-Dec 13 Peer Learning Community Forum: Interactive discussion about logic models, evaluation design, and research questions (90 minutes) Thursday, December 13, 3:00-4:30 ET

Evaluation and Research Hub A community point of access to support policymakers and program managers with their efforts to use evaluations to improve services and choose evaluations and research to inform program policies and evidence-based practices. (TBD) Workforce System Strategies Contains over 1200 profiles of workforce development research and evaluation studies, toolkits, and other resources to support program management and operations, education and training, and employment, retention or advancement activities. https:strategies.workforcegps.org USDOL-ETA Division of Research and Evaluation Manages ETA’s rigorous research and evaluation approach under Section 169 of the Workforce Innovation and Opportunity Act https://www.doleta.gov/research/ CLEAR CLEAR identifies and summarizes many types of research on labor topics, including descriptive statistical studies and outcome analyses, implementation, and causal impact studies https://clear.dol.gov USDOL Evaluation Policy Statement Presents the key principles that govern the Department of Labor’s approach to planning, conducting, and using program evaluations https://www.dol.gov/asp/evaluation/EvaluationP olicy.htm

Gloria Salas-Kos Karen Armstrong Sr. Research and Evaluation Analyst Office of Policy Development and Research USDOL-ETA/Workforce System Strategies workforcepractices@dol.gov Karen Armstrong Project Director, ETA Evaluation Dissemination and Technical Assistance IMPAQ International ETA_ResearchDisseminationPLC@impaqint.com