Presentation is loading. Please wait.

Presentation is loading. Please wait.

INTRODUCTION TO WIOA COMMON MEASURES: Measurement and Data Collection

Similar presentations


Presentation on theme: "INTRODUCTION TO WIOA COMMON MEASURES: Measurement and Data Collection"— Presentation transcript:

1 INTRODUCTION TO WIOA COMMON MEASURES: Measurement and Data Collection
Presented at: NINAETC Los Angeles, CA May 2017 Congress amended the Workforce Investment Act and the result? The Workforce Innovation and Opportunity Act Ron D’Amico Social Policy Research Associates Oakland, CA

2 WORKSHOP GOALS Explain the WIOA measures and how they differ from WIA
Timeline Key terms and how different from WIA How standards are to be set Observations and implications

3 WHERE TO START?

4 Who says so? Authority in the WIOA law
Section 116(b) defines the common measures The INA program is subject to the common measures [WIOA Sections 166(e) and 166(h)] WIOA Section 166(i) allows INA grantees to request waivers from most statutory requirements Guidance in the Final Rule and TEGLs Additional guidance on the measures is provided in the regulations ETA issued several TEGLs providing clarification

5 WIOA IMPLEMENTATION TIMELINE
PY 2014 PY 2015 PY 2016 PY 2017 INA performance & reporting requirements may go into effect (TEGL & TEN 8-16) TEGL clarifies the common measures (12/19/16) Most provisions take effect (except performance) 7/1/15 Final Rule issued (8/19/16) New reporting requirements are approved (6/29/16) WIOA signed into law 7/22/14 Performance requirements go into effect (except INA) (7/1/16)

6 IMPORTANT NOTE Further guidance specific to the INA program will be forthcoming Some of the WIOA common measures may take effect for PY 2017, but only to gather baseline data New reporting system (to replace Bear Tracks) will be developed

7 Don’t worry! Performance has been very good

8 KEY DEFINITIONS Participant Date of Participation Qualifying Service
Exit Date Exit Quarter

9 Eligible for INA Program Receives a Qualifying Service
PARTICIPANT Eligible for INA Program Receives a Qualifying Service Both criteria must be met to be counted as a participant. If an individual has been determined eligible for WIOA Section 166 but never receives a WIOA service they are not counted in the performance measures because they are not a participant. This makes them “active.” No change from WIA

10 DATE OF PARTICIPATION Eligible Received a qualifying service
Date of participation is the date the participant got the first qualifying service No change from WIA It follows that the date of participation is the first day that an individual begins receiving a service funded by the program. This is the start of program participation and will be key to understanding the timing of common measure calculations. Bear Tracks automatically makes someone “active” (a participant) as soon as they are determined eligible and assigned a Qualifying Service.

11 QUALIFYING SERVICE Qualifying services include (among other things)
Initial and comprehensive assessments Career counseling Development of employability plan Individualized assistance with job search Prevocational services & work experience Training Does not include Eligibility determination Self-services Providing general information Follow-up services after exit For the INA program, no change from WIA

12 NEW NAMES FOR SOME SERVICES
WIA CORE WIA Intensive WIOA Career Services WIA core & intensive services combined into WIOA career services

13 DATE OF EXIT The exit date is the last date the individual received a Qualifying Service. Exit must occur when: The participant has not received a Qualifying Service for 90 days, and No further Qualifying Services are planned (i.e., there is no planned gap) BearTracks will automatically exit someone with no service after 90 days, unless there is a planned gap Exit date is the date of last service, not the end of the 90-day period No change from WIA

14 Quarters CALENDAR QUARTERS
Exits and employment outcomes are measured in calendar quarters Jan-Mar Jul-Sept The exit quarter is the calendar quarter in which the exit date falls No change from WIA Apr-June Oct-Dec

15 EXAMPLE The Scenario What Is Eligibility determination is April 21st
First service (assessment) is provided May 4th Last service (job referral) is provided Sept 4th No further services are provided for the next 90 days Customer gets a job December 15th What Is The date of participation? The date of exit? the exit quarter?

16 EXAMPLE The exit quarter is the quarter in which the date of exit occurs Apr May June July Aug Sept Oct Nov Dec Period of participation 90 days with no service Eligibility 4/21/16 Got a job 12/15/16 Date of first service 5/4/16 Date of last service (Exit date) 9/4/16 End of 90-days 12/2/16 Exit Quarter 7/1 to 9/30

17 MEASUREMENT QUARTERS Employment outcomes are measured in the four quarters after the exit quarter Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 1ST Qtr After 2nd Qtr After 3rd Qtr After 4th Qtr After

18 GLOBAL EXCLUSIONS Participants are excluded from the common measures for the following reasons: Medical condition expected to last more than 90 days causes the person to exit the program and precludes employment Incarcerated Hospitalized or in other treatment facility Deceased Reservists called to active duty

19 THE MEASURES

20 WIOA MEASURES Adult Measures Youth Measures
Employed in the 2nd quarter after exit Employed in the 4th quarter after exit Median earnings in the 2nd quarter after exit Credential attainment Skill gains Service to employers Other measures as determined Youth Measures Employed or in education or training in the 2nd quarter after exit Employed or in education or training in the 4th quarter after exit Median earnings in the 2nd quarter after exit Credential attainment Skill gains Service to employers

21 Measure 1: 2ND QUARTER EMPLOYMENT RATE
DEFINITION: The percent of exiters who were employed anytime in the second quarter after the exit quarter Number employed in second quarter after exit quarter All exiters in that exit quarter (except global exclusions) Notes Unsubsidized employment anytime in the second quarter counts Evidence can come from UI and federal employment matches, or supplemental data (data provided by grantees in the SPIR) Not the same as the WIA Entered Employment Rate

22 2ND QUARTER EMPLOYMENT RATE – EXAMPLE
Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 2nd Qtr After

23 2ND QUARTER EMPLOYMENT RATE QUIZ
Scenario First service is May 4th Last service (job referral) is August 18th Customer gets a job Sept 15th Customer loses job Dec 30th Customer finds another job April 10th What is The measurement quarter for the employment rate? Is the customer a positive on this outcome?

24 2ND QUARTER EMPLOYMENT RATE – QUIZ
CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exits 8/18 Gets Job Loses Job Gets Job Exit Quarter 2nd Qtr After Exit Employed 9/15 to 12/30 Not Employed Employed 4/10 to …

25 Measure 2: 4th QUARTER EMPLOYMENT RATE
DEFINITION: The percent of exiters who were employed anytime in the fourth quarter after the exit quarter Number employed in fourth quarter after exit quarter All exiters in that exit quarter (except global exclusions) Notes Unsubsidized employment anytime in the fourth quarter counts Evidence can come from UI wages and federal employment matches, or supplemental data

26 4th QUARTER EMPLOYMENT RATE – EXAMPLE
Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 4th Qtr After

27 Measure 3: MEDIAN EARNINGS
DEFINITION: The median quarterly earnings of those employed in the second quarter after the exit quarter Notes Measured only for those employed anytime in the 2nd quarter Evidence can come from UI and federal employment matches or supplemental data If you use supplemental data to count someone as employed in the 2nd quarter, you must also measure that person’s earnings Differs from WIA Average Earnings Measurement quarter is different WIOA measures median earnings, not average earnings WIOA includes earnings measured from supplemental data

28 MEDIAN EARNINGS – EXAMPLE
Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 2nd Qtr After

29 Half of people below midpoint
CALCULATING A MEDIAN Earnings in 2nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 Half of people below midpoint Midpoint Half of people above midpoint Median= $6,000 (the earnings of the person at the midpoint) Average = 42,400 = $6,057 7

30 MEDIAN MINIMIZES THE EFFECT OF EXTREME VALUES
Scenario 1 Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 Scenario 2 Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $15,500 Sum $49,500 Midpoint Median= $6,000 (the midpoint) Median= $6,000 (the midpoint) Average = 42,400 = $6,057 7 Average = 49,500 = $7,071 7

31 Middle two people CALCULATING A MEDIAN WITH AN EVEN
NUMBER OF PARTICIPANTS Earnings in 2nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Person 8 $8,500 Middle two people Median= ( ) =$6,500 2

32 Measure 4: CREDENTIAL RATE
DEFINITION: Percent of those enrolled in education or training (excluding OJT & customized training) who obtain a postsecondary credential or high school diploma or equivalent within one year after exit date. Those obtaining a high school diploma or equivalent count as a success only if they are employed or in education or training leading to a recognized post-secondary credential within the year.

33 CREDENTIAL RATE Who is in the base? Who counts as a success?
Exiters (except global exclusions) who, while a participant: Were in postsecondary education or training (other than OJT or customized training), or Were in secondary school (without already having a diploma) Who counts as a success? Those who, during participation or within one year after exit Obtain a postsecondary credential, or Obtain a secondary school diploma or equivalent, but only if they are also employed or enrolled in postsecondary education leading to a credential anytime within the year after exit

34 CREDENTIAL RATE – EXAMPLE
Measured anytime from date of participation to 1 year after exit date (not tied to calendar quarters) Participation date 5/15/16 Exit date 9/4/16 CY 2016 APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR CY 2017 Participation Year After Exit Date Measurement Period is 5/15/16 to 9/3/17 (Date of Participation to 1 Year After Exit Date)

35 TYPES OF CREDENTIALS Designed to measure attainment of measurable technical or occupational skills necessary to obtain employment or advance Includes High school diploma or GED Associates, bachelors, or graduate degree An occupational license (e.g., awarded by a State) An occupational certificate Certificate awarded for completion of a registered apprenticeship program An occupational certification recognized by a professional, industry, or employer organization

36 TYPES OF CREDENTIALS Examples of what counts
Certified Nursing Assistant (CNA) license Automotive Service Excellence (ASE) certification series (National Institute for ASE) Microsoft Certified IT Professional (Microsoft) Machining Level I (National Institute of Metalworking) Examples of what doesn’t count Certificates awarded by workforce boards Work readiness certificates Attainment of general skills related to safety, hygiene, etc., even if required to qualify for employment (example: OSHA 10) Certificates of completion

37 TESTING OUR KNOWLEDGE: EXAMPLE 1
Scenario High school student enrolls April Chris was a high school student when he enrolled in the WIOA CSP (Adult) program in April He was in danger of dropping out, so the program provided him with tutoring and other dropout prevention activities. He also participated in work experience during the summer. Exits Nov By the fall, he was doing much better in school, and he exited the WIOA program in November 2016. Gets diploma in May Chris graduated from high school in May and got a job in June Unfortunately, he got laid off in Sept He found a great new job in Jan 2018. What are Chris’s outcomes? Is he a positive on the credential attainment rate? What other outcomes might he achieve?

38 TESTING OUR KNOWLEDGE : EXAMPLE 1
Got a new job /10/18 Exit date 11/18/16 Enrollment date 4/15/16 Graduated 5/25/17 Got a job 6/20/17 Lost job 9/18/17 APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR Oct Nov Dec Jan CY 2016 CY 2017 4th Qtr After Exit Qtr 2nd Qtr After Period of Participation Year After Exit Date Measurement period for obtaining a credential is 4/15/16 to 11/17/17 (Date of Participation to 1 Year After Exit Date)

39 TESTING OUR KNOWLEDGE: EXAMPLE 1
Answer Chris gets credit as a success on the Credential Attainment Rate High school diploma is an allowable credential type He obtained his diploma sometime in the measurement period To get credit for a high school diploma, you must be employed sometime in the year after the exit date, and he was Chris gets credit as a success on the 2nd Quarter Employment Rate He was employed sometime in the second quarter after the exit quarter He is included in the calculations of Median Earnings (but has very low quarterly earnings, since he was only employed 10 days in the quarter) He is a negative on the 4th Quarter Employment Rate

40 TESTING OUR KNOWLEDGE: EXAMPLE 2
Scenario Enrolled and exited. Carmen enrolled in WIOA in July Based on her assessment results, she enrolled in a community college Automotive Service Technician program in Sept It was a nine-month program paid for through WIOA. She dropped out of training in March 2017, before she completed, and moved out of town. Despite trying to reach her, the WIOA program didn’t hear from her for four or five months. Re-enrolled and exited. Carmen returned to the area in August, and went to the grantee for help. The grantee re-enrolled her in WIOA and paid for her to finish her training. She resumed training in Sept and finished in November, when she exited the program the second time. Outcomes. She passed the test for required for the Automotive Service Excellence Certification in Dec, and got a job shortly after. She remained employed at this firm for several years. What are Carmen’s Outcomes?

41 TESTING OUR KNOWLEDGE: EXAMPLE 2
A Clue!! Carmen enters and exits twice That means she has two periods of participation in WIOA She counts as two separate participants She has two measurement periods

42 TESTING OUR KNOWLEDGE : EXAMPLE 2
Complete training & exit WIOA 11/25/17 WIOA enrollment 7/15/16 Exit from training & WIOA 3/15/17 Return to college 9/8/17 Gets ASE certification & gets job 12/12/17 Enroll in training 9/8/16 Re-enroll in WIOA 8/25/17 CY 2016 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2017 CY 2018 First spell (7/15 to 3/15) Second spell (8/25 to 11/25)

43 TESTING OUR KNOWLEDGE : EXAMPLE 2
First spell of participation WIOA enrollment 7/15/16 Exit from training & WIOA 3/15/17 Gets ASE certification & gets job 12/12/17 Enroll in training 9/8/16 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2016 CY 2017 CY 2018 Exit Qtr 2nd Qtr After 4th Qtr After First measurement period for obtaining a credential is 7/15/16 to 3/14/18 (Date of Participation to 1 Year After Exit Date)

44 TESTING OUR KNOWLEDGE : EXAMPLE 2
Complete training & exit WIOA 11/25/17 Second spell of participation Return to college 9/8/17 Gets ASE certification & gets job 12/12/17 Re-enroll in WIOA 8/25/17 CY 2016 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2017 CY 2018 Exit Qtr 2nd Qtr After Second measurement period for obtaining a credential is 8/25/17 to 11/24/18

45 TESTING OUR KNOWLEDGE: EXAMPLE 2
Carmen appears as two WIOA participants Outcomes for First Spell of Participation Carmen’s first spell counts as a success on Credential Attainment Rate She earned a certification within one year of exit Carmen counts as a negative on the 2nd Quarter Employment Rate She counts as a success on the 4th Quarter Employment Rate Her earnings do not count for Median Earnings (she was not employed in the second quarter after the exit quarter) Outcomes for Second Spell of Participation Carmen’s second spell also counts as a success on Credential Attainment Carmen counts as a positive on both the 2nd Quarter Employment Rate and the 4th Quarter Employment Rate Her earnings count for Median Earnings

46 Measure 5: SKILL GAIN DEFINITION: Percent of participants who, during the course of the program year, are in education or training programs that lead to a recognized postsecondary credential or employment and who are achieving measurable skill gains, defined as documented academic, technical, occupational, or other forms of progress, towards such a credential or employment

47 SKILL GAIN continued Key Elements
Designed to measure interim progress of participants, as well as those who exited during the reporting period Base includes those who, during the reporting period, were in education or training programs that lead to a recognized postsecondary credential or employment Includes those in secondary school Includes those in OJT Does not include work experience (this is not training, according to WIOA) Each participant can count as a success only once during a reporting period, even if the individual has more than one skill gain Participants can count in multiple reporting periods if they have at least one skill gain in each period Key Elements

48 Indicators of Skill Gains
SKILL GAIN continued Documentation of skill gain includes: Increasing at least one educational functional level for those in programs below the postsecondary level Attainment of high school diploma or GED Making satisfactory progress in school (e.g., earning sufficient credit hours) Secondary school: transcript or report care for 1 semester Postsecondary: at least 12 credit hours in a semester or, for part-time students, 12 credit hours over 2 consecutive semesters Making satisfactory progress towards measurable standards as attested by an employer or training provider Passing an exam designed to measure occupational or technical skills Indicators of Skill Gains

49 Measure 6: SERVING EMPLOYERS
Effectiveness in serving employers Three approaches being considered Retention: Percent of exiters employed in the 2nd quarter after exit who are employed with the same employer in the 4th quarter after exit Repeat Employer Customer: Percent of employers served during the year who were also served in any of the prior 2 years Employer Penetration Rate: Percent of all employers in the service area who use WIOA services

50 SERVING EMPLOYERS continued
Calculation of the Retention Rate Number employed in the 4th quarter with the same employer as in the 2nd quarter Number employed in the 2nd quarter after exit quarter Logistics Would be calculated strictly through UI wage matching Does not require additional data collection by the grantee Could not be calculated until the 4th quarter after exit

51 Measure 7. INA SPECIFIC MEASURE
Ability to effectively serve native peoples and communities has always been understood as a critical part of grant responsibility Rationale In addition to the required measures, ETA, in consultation with the Advisory Council, “must develop a set of performance indicators and standards that are applicable to the INA program.” Logistics ETA hasn’t made progress on these measures yet This is a statutory requirement, so work on this should start soon Discussion

52 (apply to those enrolled in the Supplemental Youth Services Program)
The Youth Measures

53 Measure 1: 2ND QUARTER EDUCATION or EMPLOYMENT RATE
DEFINITION: The percent of exiters who were employed or in education or training anytime in the second quarter after the exit quarter Number employed or in education/training anytime in second quarter after exit quarter All exiters (except global exclusions) Notes Unsubsidized employment or participation in secondary or postsecondary education or occupational training anytime in the second quarter counts as a positive outcome Evidence of employment can come from UI wages and federal employment matches, or supplemental data

54 Measure 2: 4th QUARTER EDUCATION or EMPLOYMENT RATE
DEFINITION: The percent of exiters who were employed or in education or training anytime in the fourth quarter after the exit quarter Number employed or in education/training anytime in fourth quarter after exit quarter All exiters (except global exclusions) Notes Same as prior measure, but measured for the 4th quarter after the exit quarter

55 OTHER WIOA YOUTH MEASURES
The remaining youth measures are identical to adult program measures just discussed: Median Earnings Credential Attainment Rate Skill Gain Service to Employers

56 (apply to those enrolled in the Supplemental Youth Services Program)
Measurement Issues

57 WHY USE CALENDAR QUARTERS?
Performance is measured using Unemployment Insurance (UI) wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages Use of UI data requires a lag of 2 quarters before data become available We also use calendar quarters because performance is measured using Unemployment Insurance (UI) wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages. More on this later in the presentation…

58 HOW EMPLOYMENT OUTCOMES ARE MEASURED
Both wage matching and grantee data can be used Wage matching Grantees submit SSNs of participants in the SPIR DOL matches against Unemployment Insurance (UI) wage files and federal employment database, which contain employment records for (nearly) everyone employed in the U.S. Supplemental data (provided by grantees) Also as part of the SPIR, grantees indicate which participants are employed according to the grantee's own records

59 TYPES OF SUPPLEMENTAL DATA
Documentation for supplemental data (data provided by the grantee) include: Pay stubs Letter from an employer Follow-up survey of the participant Detailed case notes signed by the counselor If supplemental data is being used, the same source must be used for measuring employment and wages

60 What difference does each data source make?
PY 15 Performance on the WIA Measures Grantee Data Only UI Data Only Both EER 41.5% 54.9% 66.9%* Retention 75.2% 73.9% 78.8%* Avg Earnings $10,562 $9,722* na * This is the official performance in WIA

61 SETTING STANDARDS Using an Adjustment Model
For the State programs, standards should be set using a statistical model that takes into account: Economic conditions Characteristics of participants Performance standards for the Section 166 program should use “to the extent practicable, the statistical adjustment model” used for states [WIA Section 166(h)]

62 OBSERVATIONS & NEXT STEPS

63 OBSERVATIONS Statutory Language Limits DINAP’s Options
Legislation requires use of common measures for the INA program Some Important Flexibility Grantees can seek waivers Advisory Council can help develop INAP specific measures Observations and speculation: Statutory Language Limits DINAP’s Options: Legislation requires use of common measures for program Some Important Flexibility: 1) Grantees can seek waivers. 2) Advisory Council can help develop INAP specific measures

64 IMPLICATIONS FOR REPORTING
Changed reporting forms: 9084 & 9085 will be replaced PIRL replaces the SPIR New forms have already been approved by DOL and OMB

65 NEW DATA SYSTEM Bear Tracks is being replaced
Eliminate concepts of core and intensive services New outcomes need to be reported Some changes to info on participants’ background characteristics for the PIRL (but not many) May need to provide data on participants, not just exiters Will likely be a cloud-based system Individual-level data system being developed for both CSP and Supplemental Youth Program

66 SIGNIFICANT CHALLENGES
There are significant challenges and unknowns It is more challenging to time exits to maximize success Measuring as far out as 4 quarters after exit will be difficult Measuring skills gains will be challenging Measures for the Supplemental Youth Program are especially challenging given the limited funding for program services Substantial uncertainty remains Phase-in and timing Employer measures and INA-specific measure(s) New data system Flexibility to request waivers and waiver process

67 WHAT TO EXPECT PY 2017 is likely to be a transition year
No standards set, but baseline data collection likely to begin Bear Tracks SPIR will be examined for what it reveals Data on employment in the 2nd quarter Wages Credential attainment SSNs can be sent for UI matching to measure new outcomes Advisory Council likely to begin work on new measures DINAP likely to issue guidance on transition and phase-in during the summer Work on new data system to proceed

68 WHAT TO DO? What to do in the meantime
Aggressively enter data in Bear Tracks Credentials attained Supplemental data on employment SSNs, if appropriate (UI wage matching is very helpful) Make better use of follow-up services Think about allocating budget for follow-up services Supportive services and counseling are allowed for up to 12 months after exit (see Final Rule (d)) Keep the focus on serving your customers and community Support your Advisory Council Make your challenges and needs known Seek as much flexibility as allowable under the law

69 RESOURCES Final Rule for ETA (issued 8/19/16)
Provides regulations for ETA-only WIOA provisions Regulations for the INA program at Section 684 Available from the Federal Register at Final Joint Rule (issued 8/19/15) Describes performance standards requirements (see Section 677) Available from the Federal Register at Approved Information Collection Request Describes proposed new individual-level record layout (to replace the SPIR) and new aggregate report Available at

70 QUESTIONS What challenges do you see? Questions?

71 Social Policy Research Associates
CONTACT INFORMATION 71 Ron D’Amico Principal Social Policy Research Associates


Download ppt "INTRODUCTION TO WIOA COMMON MEASURES: Measurement and Data Collection"

Similar presentations


Ads by Google