1. Don McCorquodale, Chair Commission on Workforce Development
Strengthening North Carolina’s Workforce Development System: One Step at a Time
Don McCorquodale, Chair
Commission on Workforce Development
Good Afternoon, I’m Don McCorquodale, Chair of the Commission on Workforce Development.
The Program Evaluation Division of the General Assembly released a report in March 2012 which identified a number of deficiencies within the workforce development system. Recommendations from this report were the foundation for the workforce reform legislation (Session Law ) which has guided much of the work of the commission over the last year and a half.

2. Session Law Requirements
Revise Memorandum of Understanding for JobLink Career Centers
Completed
Establish criteria for JobLink Career Centers
Review and Evaluate North Carolina’s workforce development plans
Develop and report on system-wide performance measures to assess the effectiveness of the entire system
One of my key roles as chair of the commission is to make sure that we are improving the workforce development system and meeting the requirements laid out in the workforce reform legislation (Session Law ). And, I’m happy to report we have met each of the requirements on time. This table identifies each of the activities that you directed us to do
We revised the MOU for JobLink Career Centers and implemented this MOU by July 1, 2013.
We established new criteria for these JobLink Centers and office across the state are working to meet these criteria and be certified
We reviewed and evaluated the workforce development plans of all the state workforce agencies and we’ll continue to do this each year
We developed the first ever report on system-wide performance measures to assess the effectiveness of the workforce system, which Catherine Moga Bryant will provide more information on shortly
We are continuing to work to improve the workforce system and we have several projects in process. The one you may be most interested in is the comprehensive workforce development plan that the commission is developing that will provide guidance and direction for the whole workforce system. This plan will help bring direction to the whole workforce system, find efficiencies, and ensure that we are providing the best customer service to businesses and individuals across the state.

3. Evaluating the Workforce System
Program Evaluation Division identified need for performance measures
Role of the Commission on Workforce Development
Phased implementation
Today, we are here to report on the performance measures that the commission has developed. As identified by the Program Evaluation Division, up to this point there has been a lack of consistent access to meaningful system-wide data about the system. And, the commission was directed to develop these measures and report back to you.
To begin developing these measures I created a working group of members and workforce agency representatives. The working group agreed to develop these measures in two phases.
Phase one, which is what we’ll be presenting to you today, provides information on the performance of the whole system.
Phase two will provide more detailed information on performance by program.
This initial set of data that we’re presenting today will be reviewed by the commission at our meeting next week. We’ll use this information to set goals and success measures for our system as part of our strategic plan.
I’d be happy to take any questions, if it’s ok with the chair…
Now, I’d like to turn it over to Catherine who’ll walk you through the process and the results of this initial set of performance data.

4. Measuring the Performance of North Carolina’s Workforce Development System
Catherine Moga Bryant
Director, Governance and Strategic Planning
Division of Workforce Solutions
Good afternoon Mister Chair and members.
I’m Catherine Moga Bryant, Director of Governance and Strategic Planning in the Division of Workforce Solutions
Today I’ll be presenting you information on the performance of the workforce development system. This information has never been available before and what I’ll present to you is just the first step in the development of performance measures.

5. Overview Background Development of performance measures
Data limitations
Initial system-wide performance measures
Improvement plan
First, I’ll provide some background on the project
Then, I’ll provide a brief overview on how we developed the measures
There are some limitations to the data we have available to develop performance measures and I’ll discuss with you some of those limitations and our plans for improving the data.
Next, I’ll review our initial performance measures
And, I’ll conclude with our plans to continue to improve these measures.

6. Overview
Session Law directed the Commission on Workforce Development to develop and continuously improve performance measures to assess the effectiveness of workforce training and employment. The workforce development system is comprised of a variety of programs with a broad range of activities serving a diverse population.
Session Law 2012 – 131 directed the Commission on Workforce Development to develop and continuously improve performance measures to assess the effectiveness of workforce training and employment in North Carolina. The commission established a working group of its members and representatives from each of the state workforce agencies to develop these measures.
Developing these measures was no easy task because the workforce development system is comprised of a variety of programs with a broad range of activities that serve a diverse population. For example, activities range from
Self-service to staff assisted
Short- and long-term training programs
And Financial support programs for assistance with tuition, child care, and transportation
And these programs serve diverse populations with varying levels of
Education
Employment experience
And Barriers
Because of range of activities and diverse populations served, developing meaningful system wide measures was no easy task. This report is just an initial step, but we are committed to continuing to develop more detailed information that will help assess the success of the system

7. Developing the Measures
Established working group
Determined which programs to include
Identified performance measures
Defined time frame
Determined population groups
The working group established to develop these measures met to determine how the measures would be developed and presented.
The first step was to determine which programs should be included in the measures
The group also had to determine what the performance measures should be,
what time frame to use,
and which population groups to report on
I’ll now briefly discuss each of these.

8. Workforce Development Programs
Agency
Workforce Program
Dept. of Commerce
Wagner-Peyser
Workforce Investment Act
Dept. of Labor
Apprenticeship
Dept. of Health & Human Services
Food and Nutrition Services, Employment and Training
Services for the Blind, Employment and Training
Work First, Employment and Training
Vocational Rehabilitation, Employment and Training
Dept. of Public Instruction
Career and Technical Education
N.C. Community College System
Basic Skills
BioNetwork
Customized Training
Human Resource Development
Small Business Center Network
Workforce Continuing Education
Postsecondary Career, Technical, and Vocational Education
There were two criteria for determining which programs would be included in the performance measures,
1st – they had to meet the workforce development program definition established by the Program Evaluation Division in its March 2012 report on the workforce system
And 2nd – programs had to participate in the Common Follow up System (or CFS) administered by the Labor and Economic Analysis Division in the Department of Commerce. CFS is a database containing information on
Individuals that have participated in public supported education, employment, and training programs in NC
Service received by each individuals; and
Wages from employers covered by NC unemployment insurance laws.
This table shows the programs that were included in this initial set of performance measures.

9. What to Measure
Programs help individuals obtain and maintain adequate employment through training, education, and support.
Performance measures should report on:
Participation
Skills obtained
Employment
Average Wages
Next, the working group had to determine what the performance measures should be.
Although NC’s workforce development programs offer a wide variety of services to a broad range of individuals, the overall goals were generally consistent: to help individuals obtain and maintain adequate employment through training, education, and support.
As a result, the group determined the performance measures should report on
Participation
Skills obtained
Employment, and
Average wages.

10. System-Wide Performance Measures
How many individuals participated in the workforce development system?
How many individuals continued to participate in the system?
How many individuals were employed?
What were the average wages of those employed?
How many individuals continued their education in the public higher education system in N.C.?
The working group identified six questions to answer:
How many individuals participated in the workforce development system?
How many individuals continued to participate in the system?
How many individuals were employed?
What were the average wages of those employed?
How many individuals continued their education in the public higher education system in N.C.?

11. Cohort of Workforce Development System Participants
Time Frame
Cohort of Workforce Development System Participants
(1.6 million)
Status 1year later
Status 2 years later
Next, the working group needed to decide what time frame to use.
Workforce development program outcomes are traditionally based on the performance of individual participants after they leave a program. However, since many of the state’s workforce development participants
receive services from a variety of programs,
transition from one type of service to another,
work while enrolled in training,
or return to the workforce development system after gaining employment
the working group decided to use a cohort based approach which follows participants across three years.
The cohort was defined by their participation in a workforce development system program any time during a fiscal year – in this case FY
The length of participation varies from person to person based on their situation and needs – so participation in the base year includes individuals you may have received one job referral to individuals who are participating in a multi-year training program.
In addition, because many people continue to participate in the workforce system across many years, for example continuously improve their skills through courses at a community college, the working group wanted to see how all participants in the system were doing, not just those that had left the system.
The cohort was tracked for two years after the base year and information is presented separately for individuals who continue to participate and for individuals after their participation has ended.
Base Year:
Fiscal Year
Fiscal Year
Fiscal Year

12. Additional Cohort Subsets
Services
Services
Training
Education
Training
To best address the wide range of activities and the diversity of participants, the working group agreed to provide information about subsets of those served by age, because employment and wage outcomes for youth participants would likely be different than for adult participants
These two subsets were then divided into three additional subsets based on the type of activities – services, training, and education.
Education
Adults
Youth

13. Data Limitations Common Follow-up System (CFS) Wage records
Missing programs
Missing unique identifiers
No private education and training programs
Wage records
No out-of-state employers
No federal employers
Anytime you work with data, there are limitation that need to be identified.
First, these measures are limited by the data submitted to CFS by the state workforce agencies, so we can only report on the information available. We found that there are several programs that are not currently reporting data
Second, CFS uses social security numbers as the unique identifier to match participation records to wages records. Some programs are not reporting social security numbers, making it impossible to report outcomes for those individuals. The biggest group of participants without social security numbers are students in the Department of Public Instruction’s Career and Technical Education programs. Approximately, 85% of school districts report SSNs to CFS, but the percentage of students in CFS is less because the two largest districts (Meck & Wake) no longer report SSNs.
Third, CFS only contains information on publically supported education and training programs. – information from private or out of state colleges and universities are not included meaning we don’t know if a person is continuing their education or training at one of these organizations
Finally, the wage record data includes information on individuals who are employed with organizations covered under North Carolina’s unemployment insurance laws. If an individual is employed by an organization that isn’t covered, then their status can not be determined. This incudes people working for out of state and federal employers.

14. Data Improvements Additional program data
Access to employment and wage data from employers in other states and the federal government
Improvements to CFS
System redesign
Data quality
Capacity to match data
As part of the plan for improving these measures, there are several initiatives in progress to improve the data.
First, we are working to ensure that all programs report data to CFS
Second, LEAD is working to gain access to employment and wage data from employers in other states and the federal government
And, third, LEAD is making improvements to CFS by redesigning the system, improving the data quality, and expanding the capacity to match data.

15. Performance Measures
Now, I will provide a brief overview of the initial data on the performance of the workforce development system. I’ll just be reporting on the highlights, but there is much more information in the report.

16. How many individuals participated in the workforce development system?
Our first question we asked was… How many individuals participated in the workforce development system?

17. Base Year Participation
1.6 million participants in the cohort
North Carolina’s workforce development system served 1.6 million people in the base year, Fiscal Year
85% of participants were adults, over age 18
15% were youth, age 18 or younger

18. Participants by Agency
Number Served
Percentage of Cohort
Dept. of Commerce
804,082
50%
Community College System
610,034
38%
Dept. of Health and Human Services
289,287
18%
Dept. of Public Instruction
192,711
12%
Dept. of Labor
8,353
0.5%
Individuals participated in programs in 5 state agencies. Some participants received only one service from one agency, while other received multiple services from multiple agencies. Enrollment in multiple programs does not necessarily mean that there is duplication of service. For example, an individual could go to one of the Department of Commerce workforce offices to asses their skills, look for a job, and plan their next career move and enroll in a community college course in the same year.
This table shows the number and percentage of the cohort that received services from the different agencies.

19. Workforce Development System Participants
Make up 20% of the Workforce
We wanted to see what percentage of North Carolina’s workforce had participated in the state’s workforce system. We found that in FY , 20% of all individuals working in N.C., had participated in the workforce development system the previous fiscal year and they were employed in every sector of North Carolina’s economy. The green on this chart shows the workforce system participants by sector.

20. How many individuals continued to participate in the workforce development system after the base year?
Our second question was… How many individuals continued to participate in the workforce development system after the base year?
The speed of change in the economy is increasing and employers want people with more education and training than was required in the past. More and more workers are increasing their skills by accessing continuing education services through the workforce development system and we wanted to know how many.

21. Participants Continued to Participate and Improve their Skills
One year after the base year over 955,000 individuals continue to participate and two years later over 700,000 continued their participation. Each year, over half of the participants are participating in training or education programs.

22. How many individuals from the cohort were employed?
Employment is a key outcome of all the workforce programs, so we wanted to know how many individuals from the cohort were employed.

23. Employment Rates of Adult Participants
Because so many participants continue to improve their skills year after year and we know that many participants benefit from their participation even if they have not completed their program, we wanted to know the employment rates for all participants.
This chart show the employment rate for all adult participants in the base year and the next two years by the type of activity.
The employment rate for participants in training programs is the highest. Many of these programs are employment based, meaning the individual must be employed to participate – these are programs such as customized training, on-the-job training, and apprenticeship.
Adults participating in education programs were found to have the greatest increase in percentage employed of the three areas. Education programs are those that lead to the receipt of an associate’s degree, post-secondary certificate or diploma, or a high school diploma or equivalency.
(All members of the cohort.)

24. Employment Rates of Youth Participants
The data shows that there is a difference between the employment rates of youth and adult participants. Youth participants may be enrolled in a high school or community college or they may have left the education system.
While employment rates start low for this subset of the cohort, the percentage employed increase one and two years later. In addition to employment, positive outcomes for youth participants are continuing in education and training programs through the workforce system or in higher education.

25. What were the average wages of those employed?
Next, we wanted to see what were the average wages of those employed?

26. Wages Increase 20% increase $19.7 billion $16.4 billion
Average wages for all wage earners increased by 6%
$19.7 billion
(2 years later)
20% increase
During FY over 945,000 people in the cohort were employed an earned nearly $16.4 billion in wages. Two years later the total wages of the cohort increased by 20% to nearly $19.7 billion. This wage increase is higher than the average wage increase experienced in N.C. during this time period, which was a 6% increase for all wage earners.
$16.4 billion
(base year)

27. Wages Increased for Adults
Wages typically decrease during participation in the workforce development system – this may be due to the individual's participation in a training or education program which limits the person’s ability to work full-time or because the person has been laid off prior to participation, but in the same program year.
Therefore, the working group was especially interested in the wages of the individuals after participation. All subsets of the cohort experienced increases in average wage greater than the increase in the average wage for all wage earners in N.C.
(After participation)

28. Wages Increased for Youth Participants
As expected, youth wages are lower than adult wages. Many youth that participate in the workforce development system are not working full-time and therefore the average wage is lower. However, youth participants also experienced increases in average wages after participating.
(after participation)

29. How many individuals from the cohort continued their education in the public higher education system?
Finally, we wanted to know how many individuals from the cohort continued their education in the public higher education system.
As I stated earlier, continuous learning is essential for workers to stay well informed of new technology and to gain the skills needed to find good jobs. Therefore a positive outcome for participants, especially youth, is to continue in the public higher education system.

30. Youth Participants Enroll in Higher Education
After participating in the workforce development system, 20% of youth were enrolled in public higher education one year later, and 27% were enrolled two years later.

31. Next Steps Develop program specific measures
Identify expected performance levels
Measure the level of employer participation
Determine customer satisfaction
This report is just the first step in the development of meaningful system-wide performance measures for the workforce development system. The working group has already outlined the next steps and will continue to meet to improve these measures and provide valuable information on the success of the workforce development system.
The next steps for the working group are to:
Develop program specific measures
Identify the expected performance levels for the system
Measure the levels of employer participation
And Determine customer satisfaction

32. Catherine Moga Bryant catherine.mogabryant@nccommerce.com
Thank you very much for the opportunity to present this information to you today. As I mentioned, this report is just the first step in the development of performance measures for the workforce development system and we will be continuing to work to improve these measures.
There is additional data located in the report that we submitted to the committee and I’d be happy to answer any questions about what I’ve presented today or what is in the report at the direction of the chair.