1. Jobs Accelerator Grantee Evaluations
Webinar Date: Thursday, November 13th 2014
Presented by: U.S. Department of Labor, Employment and Training Administration

2. Enter your location in the Chat window – lower left of screen
Where are you?
Enter your location in the Chat window – lower left of screen #

3. Today’s Moderators Sarah Sunderlin, Workforce Analyst
Division of Strategic Investments
U.S. Department of Labor, Employment and Training Administration
Scott Henry, Senior Advisor
Office of Entrepreneurial Development
U.S. Small Business Administration #

4. Today‘s Presenters
U.S. Department of Labor, Employment and Training Administration
Jeanne Bellotti
Jobs Accelerator Evaluation Project Director, Mathematica Policy Research
Gloria Salas-Kos
Senior Program Analyst, Office of Policy Development and Research, Employment and Training Administration
U.S. Small Business Administration
Jennifer Auer
Research Associate, Optimal Solutions Group, LLC
U.S. Department of Commerce, Economic Development Administration
Samantha Schasberger
Program Analyst, Performance and National Programs Division, U.S. Department of Commerce, Economic Development Administration #

5. WEBINAR OBJECTIVES
This webinar will provide the Jobs Innovation and Accelerator (JIAC) grantees an opportunity to learn more about the collaborative evaluation efforts led by three Federal funding agencies:
U.S. Department of Labor, Employment and Training Administration
U.S. Department of Commerce, Economic Development Administration
U.S. Small Business Administration
Presenters will outline the evaluation framework of the JIAC clusters, describe how these three agencies are collaborating across the evaluations, and discuss the status of the evaluations including results where applicable.  Presenters will also share information about next steps and how you as a JIAC can play an active role in engaging in the evaluation work. #

6. Today’s Agenda
Interagency JIAC Evaluation Model Department of Labor (DOL) Evaluation Plans Small Business Administration (SBA) Evaluation Next Steps Economic Development Agency (EDA) Evaluation Outcomes #

7. Interagency Collaboration#

8. Interagency JIAC/AMJIAC Evaluation Model
The Framework for Measuring Interagency Place-Based Initiatives
DOL
Evaluation
EDA
Evaluation
SBA
Evaluation
Quantitative/Qualitative Assessment
JIAC/AMJIAC Combined Evaluation
Data collection: On going
Quantitative/Qualitative Framework
JIAC Only
Data collection: Completed
Quantitative/Qualitative Assessment
JIAC and AMJIAC evaluated separately
JIAC Data collection: Mostly complete
AMJIAC Data collection: through Sept 2015
Coordinated effort to avoid duplication and lessen burden on grantees
Each agency issues own report

9. Employment and Training Administration (ETA)
U.S. Department of Labor
Employment and Training Administration (ETA)
Evaluation Research Design #

10. SPEAKER Gloria Salas-Kos
Senior Program Analyst, Office of Policy Development and Research
Employment and Training Administration

11. DOL Evaluation SBA Evaluation EDA Evaluation EDA Evaluation
Quantitative/Qualitative Assessment
Research Questions
The role of the multi- agency collaboration
The development of regional clusters, programs and partnerships
Goals towards meeting projected outcomes
Program management and sustainability
Replicability and lessons learned
SBA
Evaluation
Evaluation Products
Evaluation Design Report (early 2013 – Peer reviewed design)
Interim Report (2015 – Analysis of grantee applications, Quarterly Progress Reports, and federal interview data)
Final Report (late – Analysis of site visit data, survey data, cross-site analysis)
Data Sources
Grantee quarterly reports
Site visits to a purposive sample of sites
Survey data from all grant administrators and a subset of partners
EDA
Evaluation
EDA
Evaluation
CONTENT FORTHCOMING

12. DOL Evaluation: Overview
Process study that assesses
Cross-agency collaboration and partnerships
Workforce-related outcomes
Initiative management and sustainability practices; and
Identifies lessons learned and replicability
Primarily focuses on employment and training activities
Started in 2013 as a three-year study for, both, JIAC and AM-JIAC grants that includes data collection from:
Administered by:
Mathematica Policy Research
W.E. Upjohn Institute for Employment Research

13. SPEAKER Jeanne Bellotti
Jobs Accelerator Evaluation Project Director, Mathematica Policy Research

14. DOL Evaluation: Research Questions
What is the role of multiagency collaboration in the planning and implementation of cluster activities?
How and in what ways do regional clusters, programs, and partnerships develop under the grant?
What workforce-related outcomes did the clusters report achieving through this initiative?
How is the initiative managed within each cluster?
What practices are being implemented to promote sustainability of grant resources, partnerships and activities?
What are the key lessons learned through implementation?
How and under what circumstances might these lessons be replicated?

15. DOL Evaluation: Data Collection
Data Source
Description
Time Frame
Grant Applications and Integrated Work Plans
Review of grant structure and planned activities at application
Fall 2013
Grantee Reports
Review of IWP Reports and ETA Quarterly Progress Reports and Narratives
Fall 2014;
Summer 2015
Interviews with Federal Funding Agencies
Two rounds of interviews with representatives from ETA, EDA, NIST, SBA, and DOE
Fall 2013; Summer 2014
Interviews with ETA Federal Project Officers
Interviews with all ETA FPOs overseeing ETA’s JIAC/AM-JIAC grants
Summer 2014
Site Visits
In-depth visits to a subset of 9 clusters
Early 2015
Cluster Survey
Online survey of cluster administrators, ETA grant administrators and a subset of partner agencies/organizations

16. DOL Evaluation: Site Visits
Visits to a subset of nine regional/sector clusters
Early 2015 (pending OMB approval)
Two- to three-day visits
Interviews with:
Cluster management staff
Activity leaders
Frontline staff
Training participants
Key partners, including the LWIB, employer groups and local economic development agencies #

17. DOL Evaluation: Online Survey
Will take 30 minutes to complete
Survey topics include, but are not limited to:
Role and strength of partnerships
Cluster activities
Interactions with federal agencies
Outcomes
Survey respondents will include:
Cluster administrators
ETA grant administrators
Subset of partners #

18. DOL Evaluation: Survey Process
Mathematica sends initial partner lists to clusters
Clusters update partner lists
OMB approves study design
Cluster administrators and selected partners complete online survey
Mathematica selects sample and sends survey #

19. DOL Evaluation: Partner Participation
Organization Name
Contact Person Information: Name, , Phone Number
Involved in ETA Activities
Received Funds from:
Participation Level
ETA
SBA
EDA
NIST- MEP
DOE
Example 1
Tom Smith
yes no
high
Example 2
Sue Jones
medium
Example 3
Jim Scott
Example 4
Bruce Bobb
low #

20. DOL Evaluation: Partner List
DOL Evaluation: Partner Lists
DOL Evaluation: Partner List
What is a cluster or grantee partner for the purposes of the study?
Any organization participating in activities related to the JIAC/AM- JIAC project
Organizations that receive or do not receive grant funds
How will the partner lists be collected?
Develop and send initial partner lists based on information from grant applications
Clusters leads will be asked to respond quickly with updated partner lists
Why are partner lists important to this study?
The completeness of partner lists will help us gather survey data
Survey data will help us tell your story #

21. DOL Evaluation: Reports
Content
Time Frame
Study Design
Detailed design for data collection and analysis, including instruments
January 2014
Interim Report
Overview of grant structure and clusters’ proposed activities; federal perspective on multi-agency collaboration; review of cluster progress based on grantees’ reports and federal interviews; recommendations for clusters to participate in evaluation site visits
Winter 2014
Final Report
Findings related to all research questions using all evaluation data sources
Spring 2016 #

22. Questions? #

23. Small Business Administration (SBA)
Data Collection Overview and Joint Evaluation Framework #

24. Jennifer Auer Research Associate Optimal Solutions Group, LLC
SPEAKER
Jennifer Auer
Research Associate
Optimal Solutions Group, LLC

25. Cluster Evaluations
Motivations
Assess results of new economic development approaches
Ensure small businesses find value in cluster activity
Not an audit

26. Manufacturing Assistance
AMJIAC Evaluation
Assess Common Outputs and Intermediate Outcomes
Manufacturing Assistance
Workforce Training
DOL
Economic Development
EDA
Business Assistance
SBA
Study Questions
Energy Efficiencies
SBA
Evaluation
NIST
DOE
Have clusters fully implemented their plans for serving small businesses?
What barriers and lessons were learned when engaging small businesses?
How did implementation approaches and challenges vary by cluster characteristics and administrative organization?
In what ways and to what extent is cluster activity related to small business outputs and outcomes?
Coverage: October 2012 – September 2015

27. AMJIAC Evaluation November 2014: cluster lead interviews
Data Collection Efforts
November 2014: cluster lead interviews
80% complete
December 2014: disadvantaged business trackers
Provide a cumulative list
January 2015: cluster lead outputs survey
Bonus, a possible foundation for your final reports
February 2015: Cluster lead follow-up interviews and small business interviews
On Going: small business outcomes survey (via MEP)
Please submit your client information in a timely manner

28. Cluster Evaluations
Uses
Convey value of SBA-funded initiatives to stakeholders
Marketing and promotion tool for the clusters
Inform future SBA initiatives and approaches

29. Data Collected/Reviewed
JIAC Evaluation
Implementation Assessment
Workforce Training
DOL
Economic Development
EDA
Business Assistance
SBA
Study Questions
SBA
Evaluation
EDA
Evaluation
Data Collected/Reviewed
Project closeout reports (9 JIACs)
Quarterly progress reports (19 JIACs)
Rosters of disadvantaged businesses (19 JIACs)
Implementer interviews (6 JIACs)
Small business participant interviews (5 JIACs)
In what ways are small businesses supported through the JIAC projects?
In what ways are disadvantaged businesses included in JIAC projects and industries?
What challenges exist and lessons were learned in working with small and disadvantaged businesses?
What lessons have been learned from implementing the Initiative?
Coverage: October 2011 – July 2014

30. SBA JIAC Results 4 Models of JIAC Business Support Startup Bootcamps
Product Development “Labs”
Industry Capacity Building
Technology Linking

31. SBA JIAC Results Small and disadvantaged business outreach
Lessons Learned
Small and disadvantaged business outreach
Big and small wins
Grant requirements and duration

32. Questions? #

33. U.S. Department of Commerce Economic Development Administration (EDA)
Improving EDA Performance Measurement and How the JIAC Assessment will Inform the Joint Evaluation Framework #

34. SPEAKER Samantha Schasberger Program Analyst
Performance and National Programs Division
U.S. Department of Commerce, Economic Development Administration

35. A Unique Hybrid: Informing EDA Performance Measurement and a JIAC Assessment
Goal to inform the development of performance measurement, including future evaluation efforts for EDA.
The testing of metrics and evaluation tools shed some light on performance of EDA JIAC program.
Timing: Field work was completed in April 2014, with a final report submitted by November 2014.
The research team reviewed printed materials, conducted site visits and a survey and tested possible third party data sources.

36. Findings: Types of client/participant served by JIACs Hit the Target
Source: SRI survey of JIAC clients/participants

37. JIAC Findings: Activities
Figure V-4 Source: SRI survey of JIAC clients/participants
JIAC Grantee Activities, by Logic Model Category
Activities are diverse
Networking is extensive
Participant Satisfaction
was high

38. JIAC Findings: Outputs–Detailed Categories

39. JIAC Findings: Outcomes - Enhanced Capabilities
Figure VI-4 Source: SRI survey of JIAC clients/participants
Enhanced Regional Capabilities: JIAC Impacts Experienced by Participating Organizations
Source: SRI survey of JIAC clients/participants

40. JIAC Findings: Capacity Outcomes
Figure VI-1 Source: SRI survey of JIAC clients/participants
JIAC Findings: Capacity Outcomes

41. Questions? #

42. Webinar Highlights
Thank you for you past and future participation in data collection!
Coordinated questions and timing to lessen burden on grantees
Continue to “track” data about your “clients” to ease later data collection
These evaluations will help us amplify the great work you are all doing
For the AMJIACs -- the survey referral process of all your business clients (to MEP) is very important #

43. REMINDER: AMJIAC Survey Process
Survey of companies assisted by the AMJIAC funding grants
Importance
Determines impacts the 5 funding streams of the AMJIAC and the total impact of AMJIAC
Process
ALL of the companies that YOU are assisting - not just those that MEP assists - should be surveyed
You provide basic information to the MEP Center about the companies
Companies are surveyed 6 months after the assistance projects are completed
Status - We need your help!
Approximately 25 companies either completed the survey or are in the queue
More companies have been assisted thus far…. #

44. CONTACT US Department of Labor: jobsaccelerator@dol.gov
Small Business Administration:
Economic Development Administration:
Where you can find out the point of contact for your state and your local economic development representative.

45. Please enter your questions in the Chat Room!#

46. Find resources for workforce system success at:
Thank You!
Find resources for workforce system success at: #