1. April 7, 2019
Peer Learning Community Forum: Using the Evaluation Readiness Assessment Tool
Evaluations and Research Webinar Series [#2-Oct. 16, 2018]
3:00 PM – 4:30 PM

2. Gloria Salas-Kos Senior Research and Evaluation Analyst
Employment and Training Administration [ETA]
U. S. Department of Labor [DOL]

3. New Community of Practice
Access to evaluation and research tools and resources
Tailored to meet your needs!
Flash Webinars
Peer Learning Opportunities

4. Welcome new and returning members
Explain evaluation peer learning context
Present recap of evaluation readiness
Learn from peers and share experiences with evaluation readiness:
Strengths
Opportunities for improvement
Next steps

5. Did you attend the Evaluation Flash Webinar on October 2?
Yes No

6. Did you have a chance to use the Evaluation Readiness Assessment Tool?
Yes, I completed the assessment.
Yes, I completed part of the assessment.
I reviewed the tool but have not yet begun the assessment.
No, I have not reviewed or used the tool.

7. Welcome New and Returning Members!
States represented:
Alabama
Arizona
Idaho
Illinois
Kansas
Louisiana
Texas
Utah
Washington

8. Wayne Gordon Karen Armstrong
Director, Division of Research and Evaluation/ Office of Policy Development and Research
DOL-ETA
Karen Armstrong
Project Director, Dissemination and Technical Assistance
IMPAQ International

9. Lori Frazier Bearden Jennifer Bishop
Assistant Director of Workforce Programs
Alabama Department of Commerce
Jennifer Bishop
Dissemination Lead
IMPAQ International

10. Why Use Evaluation Readiness Assessment?
Wayne Gordon
Director, Division of Research and Evaluation/ Office of Policy Development and Research
DOL-ETA

11. Mission: Support state workforce agencies in their efforts to use and choose evaluation and research
Identify Needs
Tools and Resources
Identify
Disseminate
Apply
Create Evidence
Peer Learning Community
Shared Wisdom
TA Products for Evaluation Practices
Disseminate Evaluation Results
Use
Choose

12. Why Use the Evaluation Readiness Assessment Tool?
Gauge readiness
Spark enthusiasm
Celebrate strengths
Identify areas for improvement
Foster evaluation culture
Integrate evaluation into state activities

13. Why participate in the evaluation and research PLC?
Exchange ideas and strategies with peers for building evaluation capacity
Learn about latest evaluation reports and tools from DOL
Provide DOL continued feedback on how best to support states and localities

14. Flash Webinar Recap: Navigating the Readiness Assessment Tool
Karen Armstrong
Project Director, Dissemination and Technical Assistance
IMPAQ International

15. The Five Elements: Digging Deeper

16. Key Evaluation Elements1
Evaluation culture and awareness
Do agency staff and partners understand the benefits of evaluation, use evidence-based results to inform decisions, and plan to conduct evaluations to add to the existing evidence base? 2
Funding strategies
Does the state or region actively pursue funding for and invest in evaluations? 3
Data management
Does the agency have adequate operational capacity, IT infrastructure, and policies and procedures for collecting and using data for evaluations? 4
Staff skills, capacity, and knowledge
Does the agency have sufficient staff with the skills, knowledge and experience needed to conduct or oversee third-party evaluations? 5
Strategic planning
Does the state have a comprehensive strategic plan that includes evaluation as an integral part of the plan?

17. Rate Progress and Access the Information You Need

18. Rate Progress and Access the Information You Need

19. Challenge for PLC members
Share with your staff and partners.
Independently complete the tool. There are no wrong answers!
Compare and explore responses:
What are your strengths?
What are your opportunities for improvement?
Discuss next small and achievable steps.
Source:

20. Another State’s Perspective: Modeling the Assessment in Alabama
Lori Frazier Bearden
Assistant Director of Workforce Programs
Alabama Department of Commerce

21. Putting the Evaluation Readiness Assessment Tool to Work
“This is a great tool. It took less time than I thought it would.”– WIOA Performance Manager
“Sorry. I cannot provide evaluation feedback.” - Labor Market Information Division
The tool was provided via to the WIOA Performance Unit, WIOA State Board Unit, Labor Market Information Division, and Adult Education Division
Discussion Follow-up occurred

22. And the Survey Says…..

23. Digging a little deeper…..
Organizational Weakness
Organizational Strengths
Lowest Rating was Section 4: Staff Skills, Capacity, and Knowledge with an Average Score of 2
Need for Improvement: Staff Capacity
“The Workforce Development Division is not large enough to have a Research & Evaluation Unit’
Highest Rating was Section 3: Data Management with an average score of 3
“The State has data sharing agreements to facilitate interstate exchange of UI wage record data…” was given the highest rating of all Sections.

24. So what do we do now? Using this tool for Improvement
This survey tool served as a “starting point” to engage in dialogue of our shortfalls and successes in research and evaluation
Next Steps include:
Examining funds for evaluation
Developing Key Evaluation Priorities & Next Steps
No Statement in any Section of the Assessment achieved an average rating of 4 or 5

25. Please choose the statement that best represents your state/locality (select all that apply).
We have a research and evaluation unit (or its equivalent) that has 7 or more staff devoted to it.
We have a research and evaluation unit with 4-6 staff members.
We have a research and evaluation unit with 2-3 staff members.
We have a research and evaluation unit with only 1 staff member.
We do not have a unit that is specifically focused on research and evaluation.

26. Which statement below best represents your state/locality’s research and evaluation (R&E) activities?
We use contractors for all our R&E activities.
We mostly use contractors for our R&E activities.
We mostly conduct R&E activities in-house, but have a few R&E contracts.
We do all our R&E work in-house.
We are currently not conducting any R&E activities.

27. Peer Questions

28. Your Experience with Evaluation Readiness
Jennifer Bishop
Dissemination Lead
IMPAQ International

29. Discussing Your Experiences and Observations
PLC Members will now have the opportunity to share their experiences.
We have assigned you to small breakout groups to:
Share what you learned from the assessment tool
Ask each other questions about different components to help understand the barriers towards improvement and explore ways to address barriers
Challenge each other to take next steps
Refer to the Evaluation Readiness Assessment tool (download from the files window).

30. The Small Group Process
You will be routed to a new conference ‘room’ with no more than eight participants.
Each room will also have a facilitator, a technical lead, and a note taker.
Facilitator will guide the discussion.
Note taker will record your observations.
Last five minutes, everyone will review notes to make sure the highlights were captured.
Facilitator (or a volunteer from the group) will report back to the large group.

31. Small Group Discussions
Focus on Evaluation Activities in your States or Local Areas

32. Large Group Discussion
Share highlights and next steps

33. Peer Learning Community Takeaways
What were some conversation highlights?
What were some next steps that groups came up with?
What would be a helpful follow-on to this meeting?

34. Open Discussion

35. Look for WorkforceGPS invitations to the next evaluation and research series
Flash Webinar: Evaluation Design, Research Questions and Methods: An Assessment Toward Implementation (30 minutes)
Wednesday, November 29, 3:00-3:30 ET
Learning Assignment: How logic models define research questions and evaluation designs (2 weeks)
Nov 29-Dec 13
Peer Learning Community Forum: Interactive discussion about logic models, evaluation design, and research questions (90 minutes)
Thursday, December 13, 3:00-4:30 ET

36. Evaluation and Research Hub
A community point of access to support policymakers and program managers with their efforts to use evaluations to improve services and choose evaluations and research to inform program policies and evidence-based practices.
(TBD)
Workforce System Strategies
Contains over 1200 profiles of workforce development research and evaluation studies, toolkits, and other resources to support program management and operations, education and training, and employment, retention or advancement activities.
USDOL-ETA Division of Research and Evaluation
Manages ETA’s rigorous research and evaluation approach under Section 169 of the Workforce Innovation and Opportunity Act
CLEAR
CLEAR identifies and summarizes many types of research on labor topics, including descriptive statistical studies and outcome analyses, implementation, and causal impact studies
USDOL Evaluation Policy Statement
Presents the key principles that govern the Department of Labor’s approach to planning, conducting, and using program evaluations
olicy.htm

37. Gloria Salas-Kos Karen Armstrong
Sr. Research and Evaluation Analyst Office of Policy Development and Research
USDOL-ETA/Workforce System Strategies
Karen Armstrong
Project Director, ETA Evaluation Dissemination and Technical Assistance
IMPAQ International