Jobs Accelerator Grantee Evaluations Webinar Date: Thursday, November 13th 2014 Presented by: U.S. Department of Labor, Employment and Training Administration

Enter your location in the Chat window – lower left of screen Where are you? Enter your location in the Chat window – lower left of screen #

Today’s Moderators Sarah Sunderlin, Workforce Analyst Division of Strategic Investments U.S. Department of Labor, Employment and Training Administration Scott Henry, Senior Advisor Office of Entrepreneurial Development U.S. Small Business Administration #

Today‘s Presenters U.S. Department of Labor, Employment and Training Administration Jeanne Bellotti Jobs Accelerator Evaluation Project Director, Mathematica Policy Research Gloria Salas-Kos Senior Program Analyst, Office of Policy Development and Research, Employment and Training Administration U.S. Small Business Administration Jennifer Auer Research Associate, Optimal Solutions Group, LLC U.S. Department of Commerce, Economic Development Administration Samantha Schasberger Program Analyst, Performance and National Programs Division, U.S. Department of Commerce, Economic Development Administration #

WEBINAR OBJECTIVES This webinar will provide the Jobs Innovation and Accelerator (JIAC) grantees an opportunity to learn more about the collaborative evaluation efforts led by three Federal funding agencies: U.S. Department of Labor, Employment and Training Administration U.S. Department of Commerce, Economic Development Administration U.S. Small Business Administration Presenters will outline the evaluation framework of the JIAC clusters, describe how these three agencies are collaborating across the evaluations, and discuss the status of the evaluations including results where applicable.  Presenters will also share information about next steps and how you as a JIAC can play an active role in engaging in the evaluation work. #

Today’s Agenda Interagency JIAC Evaluation Model Department of Labor (DOL) Evaluation Plans Small Business Administration (SBA) Evaluation Next Steps Economic Development Agency (EDA) Evaluation Outcomes #

Interagency Collaboration #

Interagency JIAC/AMJIAC Evaluation Model The Framework for Measuring Interagency Place-Based Initiatives DOL Evaluation EDA Evaluation SBA Evaluation Quantitative/Qualitative Assessment JIAC/AMJIAC Combined Evaluation Data collection: On going Quantitative/Qualitative Framework JIAC Only Data collection: Completed Quantitative/Qualitative Assessment JIAC and AMJIAC evaluated separately JIAC Data collection: Mostly complete AMJIAC Data collection: through Sept 2015 Coordinated effort to avoid duplication and lessen burden on grantees Each agency issues own report

Employment and Training Administration (ETA) U.S. Department of Labor Employment and Training Administration (ETA) Evaluation Research Design #

SPEAKER Gloria Salas-Kos Senior Program Analyst, Office of Policy Development and Research Employment and Training Administration

DOL Evaluation SBA Evaluation EDA Evaluation EDA Evaluation Quantitative/Qualitative Assessment Research Questions The role of the multi- agency collaboration The development of regional clusters, programs and partnerships Goals towards meeting projected outcomes Program management and sustainability Replicability and lessons learned SBA Evaluation Evaluation Products Evaluation Design Report (early 2013 – Peer reviewed design) Interim Report (2015 – Analysis of grantee applications, Quarterly Progress Reports, and federal interview data) Final Report (late 2016 – Analysis of site visit data, survey data, cross-site analysis) Data Sources Grantee quarterly reports Site visits to a purposive sample of sites Survey data from all grant administrators and a subset of partners EDA Evaluation EDA Evaluation CONTENT FORTHCOMING

DOL Evaluation: Overview Process study that assesses Cross-agency collaboration and partnerships Workforce-related outcomes Initiative management and sustainability practices; and Identifies lessons learned and replicability Primarily focuses on employment and training activities Started in 2013 as a three-year study for, both, JIAC and AM-JIAC grants that includes data collection from: Administered by: Mathematica Policy Research W.E. Upjohn Institute for Employment Research

SPEAKER Jeanne Bellotti Jobs Accelerator Evaluation Project Director, Mathematica Policy Research

DOL Evaluation: Research Questions What is the role of multiagency collaboration in the planning and implementation of cluster activities? How and in what ways do regional clusters, programs, and partnerships develop under the grant? What workforce-related outcomes did the clusters report achieving through this initiative? How is the initiative managed within each cluster? What practices are being implemented to promote sustainability of grant resources, partnerships and activities? What are the key lessons learned through implementation? How and under what circumstances might these lessons be replicated?

DOL Evaluation: Data Collection Data Source Description Time Frame Grant Applications and Integrated Work Plans Review of grant structure and planned activities at application Fall 2013 Grantee Reports Review of IWP Reports and ETA Quarterly Progress Reports and Narratives Fall 2014; Summer 2015 Interviews with Federal Funding Agencies Two rounds of interviews with representatives from ETA, EDA, NIST, SBA, and DOE Fall 2013; Summer 2014 Interviews with ETA Federal Project Officers Interviews with all ETA FPOs overseeing ETA’s JIAC/AM-JIAC grants Summer 2014 Site Visits In-depth visits to a subset of 9 clusters Early 2015 Cluster Survey Online survey of cluster administrators, ETA grant administrators and a subset of partner agencies/organizations

DOL Evaluation: Site Visits Visits to a subset of nine regional/sector clusters Early 2015 (pending OMB approval) Two- to three-day visits Interviews with: Cluster management staff Activity leaders Frontline staff Training participants Key partners, including the LWIB, employer groups and local economic development agencies #

DOL Evaluation: Online Survey Will take 30 minutes to complete Survey topics include, but are not limited to: Role and strength of partnerships Cluster activities Interactions with federal agencies Outcomes Survey respondents will include: Cluster administrators ETA grant administrators Subset of partners #

DOL Evaluation: Survey Process Mathematica sends initial partner lists to clusters Clusters update partner lists OMB approves study design Cluster administrators and selected partners complete online survey Mathematica selects sample and sends survey #

DOL Evaluation: Partner Participation Organization Name Contact Person Information: Name, Email, Phone Number Involved in ETA Activities Received Funds from: Participation Level ETA SBA EDA NIST- MEP DOE Example 1 Tom Smith TSmith@mail.com 222.333.444 yes no high Example 2 Sue Jones SJones@mail.com 777.888.9999 medium Example 3 Jim Scott JScott@mail.com 567.890.1234 Example 4 Bruce Bobb BBobb@mail.com 321.456.7890 low #

DOL Evaluation: Partner List DOL Evaluation: Partner Lists DOL Evaluation: Partner List What is a cluster or grantee partner for the purposes of the study? Any organization participating in activities related to the JIAC/AM- JIAC project Organizations that receive or do not receive grant funds How will the partner lists be collected? Develop and send initial partner lists based on information from grant applications Clusters leads will be asked to respond quickly with updated partner lists Why are partner lists important to this study? The completeness of partner lists will help us gather survey data Survey data will help us tell your story #

DOL Evaluation: Reports Content Time Frame Study Design Detailed design for data collection and analysis, including instruments January 2014 Interim Report Overview of grant structure and clusters’ proposed activities; federal perspective on multi-agency collaboration; review of cluster progress based on grantees’ reports and federal interviews; recommendations for clusters to participate in evaluation site visits Winter 2014 Final Report Findings related to all research questions using all evaluation data sources Spring 2016 #

Questions? #

Small Business Administration (SBA) Data Collection Overview and Joint Evaluation Framework #

Jennifer Auer Research Associate Optimal Solutions Group, LLC SPEAKER Jennifer Auer Research Associate Optimal Solutions Group, LLC

Cluster Evaluations Motivations Assess results of new economic development approaches Ensure small businesses find value in cluster activity Not an audit

Manufacturing Assistance AMJIAC Evaluation Assess Common Outputs and Intermediate Outcomes Manufacturing Assistance Workforce Training DOL Economic Development EDA Business Assistance SBA Study Questions Energy Efficiencies SBA Evaluation NIST DOE Have clusters fully implemented their plans for serving small businesses? What barriers and lessons were learned when engaging small businesses? How did implementation approaches and challenges vary by cluster characteristics and administrative organization? In what ways and to what extent is cluster activity related to small business outputs and outcomes? Coverage: October 2012 – September 2015

AMJIAC Evaluation November 2014: cluster lead interviews Data Collection Efforts November 2014: cluster lead interviews 80% complete December 2014: disadvantaged business trackers Provide a cumulative list January 2015: cluster lead outputs survey Bonus, a possible foundation for your final reports February 2015: Cluster lead follow-up interviews and small business interviews On Going: small business outcomes survey (via MEP) Please submit your client information in a timely manner

Cluster Evaluations Uses Convey value of SBA-funded initiatives to stakeholders Marketing and promotion tool for the clusters Inform future SBA initiatives and approaches

Data Collected/Reviewed JIAC Evaluation Implementation Assessment Workforce Training DOL Economic Development EDA Business Assistance SBA Study Questions SBA Evaluation EDA Evaluation Data Collected/Reviewed Project closeout reports (9 JIACs) Quarterly progress reports (19 JIACs) Rosters of disadvantaged businesses (19 JIACs) Implementer interviews (6 JIACs) Small business participant interviews (5 JIACs) In what ways are small businesses supported through the JIAC projects? In what ways are disadvantaged businesses included in JIAC projects and industries? What challenges exist and lessons were learned in working with small and disadvantaged businesses? What lessons have been learned from implementing the Initiative? Coverage: October 2011 – July 2014

SBA JIAC Results 4 Models of JIAC Business Support Startup Bootcamps Product Development “Labs” Industry Capacity Building Technology Linking

SBA JIAC Results Small and disadvantaged business outreach Lessons Learned Small and disadvantaged business outreach Big and small wins Grant requirements and duration

Questions? #

U.S. Department of Commerce Economic Development Administration (EDA) Improving EDA Performance Measurement and How the JIAC Assessment will Inform the Joint Evaluation Framework #

SPEAKER Samantha Schasberger Program Analyst Performance and National Programs Division U.S. Department of Commerce, Economic Development Administration

A Unique Hybrid: Informing EDA Performance Measurement and a JIAC Assessment Goal to inform the development of performance measurement, including future evaluation efforts for EDA. The testing of metrics and evaluation tools shed some light on performance of EDA JIAC program. Timing: Field work was completed in April 2014, with a final report submitted by November 2014. The research team reviewed printed materials, conducted site visits and a survey and tested possible third party data sources.

Findings: Types of client/participant served by JIACs Hit the Target Source: SRI survey of JIAC clients/participants

JIAC Findings: Activities Figure V-4 Source: SRI survey of JIAC clients/participants JIAC Grantee Activities, by Logic Model Category Activities are diverse Networking is extensive Participant Satisfaction was high

JIAC Findings: Outputs–Detailed Categories

JIAC Findings: Outcomes - Enhanced Capabilities Figure VI-4 Source: SRI survey of JIAC clients/participants Enhanced Regional Capabilities: JIAC Impacts Experienced by Participating Organizations   Source: SRI survey of JIAC clients/participants

JIAC Findings: Capacity Outcomes Figure VI-1 Source: SRI survey of JIAC clients/participants JIAC Findings: Capacity Outcomes

Questions? #

Webinar Highlights Thank you for you past and future participation in data collection! Coordinated questions and timing to lessen burden on grantees Continue to “track” data about your “clients” to ease later data collection These evaluations will help us amplify the great work you are all doing For the AMJIACs -- the survey referral process of all your business clients (to MEP) is very important #

REMINDER: AMJIAC Survey Process Survey of companies assisted by the AMJIAC funding grants Importance Determines impacts the 5 funding streams of the AMJIAC and the total impact of AMJIAC Process ALL of the companies that YOU are assisting - not just those that MEP assists - should be surveyed You provide basic information to the MEP Center about the companies Companies are surveyed 6 months after the assistance projects are completed Status - We need your help! Approximately 25 companies either completed the survey or are in the queue More companies have been assisted thus far…. #

CONTACT US Department of Labor: jobsaccelerator@dol.gov Small Business Administration: SBAJobsAccelerator@sba.gov Economic Development Administration: http://www.eda.gov/contact/ Where you can find out the point of contact for your state and your local economic development representative.

Please enter your questions in the Chat Room! #

Find resources for workforce system success at: Thank You! Find resources for workforce system success at: www.workforce3one.org #