INTRODUCTION TO WIOA COMMON MEASURES: Measurement and Data Collection Presented at: NINAETC Los Angeles, CA May 2017 Congress amended the Workforce Investment Act and the result? The Workforce Innovation and Opportunity Act Ron D’Amico Social Policy Research Associates Oakland, CA

WORKSHOP GOALS Explain the WIOA measures and how they differ from WIA Timeline Key terms and how different from WIA How standards are to be set Observations and implications

WHERE TO START?

Who says so? Authority in the WIOA law Section 116(b) defines the common measures The INA program is subject to the common measures [WIOA Sections 166(e) and 166(h)] WIOA Section 166(i) allows INA grantees to request waivers from most statutory requirements Guidance in the Final Rule and TEGLs Additional guidance on the measures is provided in the regulations ETA issued several TEGLs providing clarification

WIOA IMPLEMENTATION TIMELINE PY 2014 PY 2015 PY 2016 PY 2017 INA performance & reporting requirements may go into effect (TEGL 24-15 & TEN 8-16) TEGL 10-16 clarifies the common measures (12/19/16) Most provisions take effect (except performance) 7/1/15 Final Rule issued (8/19/16) New reporting requirements are approved (6/29/16) WIOA signed into law 7/22/14 Performance requirements go into effect (except INA) (7/1/16)

IMPORTANT NOTE Further guidance specific to the INA program will be forthcoming Some of the WIOA common measures may take effect for PY 2017, but only to gather baseline data New reporting system (to replace Bear Tracks) will be developed

Don’t worry! Performance has been very good

KEY DEFINITIONS Participant Date of Participation Qualifying Service Exit Date Exit Quarter

Eligible for INA Program Receives a Qualifying Service PARTICIPANT Eligible for INA Program Receives a Qualifying Service Both criteria must be met to be counted as a participant. If an individual has been determined eligible for WIOA Section 166 but never receives a WIOA service they are not counted in the performance measures because they are not a participant. This makes them “active.” No change from WIA

DATE OF PARTICIPATION Eligible Received a qualifying service Date of participation is the date the participant got the first qualifying service No change from WIA It follows that the date of participation is the first day that an individual begins receiving a service funded by the program. This is the start of program participation and will be key to understanding the timing of common measure calculations. Bear Tracks automatically makes someone “active” (a participant) as soon as they are determined eligible and assigned a Qualifying Service.

QUALIFYING SERVICE Qualifying services include (among other things) Initial and comprehensive assessments Career counseling Development of employability plan Individualized assistance with job search Prevocational services & work experience Training Does not include Eligibility determination Self-services Providing general information Follow-up services after exit For the INA program, no change from WIA

NEW NAMES FOR SOME SERVICES WIA CORE WIA Intensive WIOA Career Services WIA core & intensive services combined into WIOA career services

DATE OF EXIT The exit date is the last date the individual received a Qualifying Service. Exit must occur when: The participant has not received a Qualifying Service for 90 days, and No further Qualifying Services are planned (i.e., there is no planned gap) BearTracks will automatically exit someone with no service after 90 days, unless there is a planned gap Exit date is the date of last service, not the end of the 90-day period No change from WIA

Quarters CALENDAR QUARTERS Exits and employment outcomes are measured in calendar quarters Jan-Mar Jul-Sept The exit quarter is the calendar quarter in which the exit date falls No change from WIA Apr-June Oct-Dec

EXAMPLE The Scenario What Is Eligibility determination is April 21st First service (assessment) is provided May 4th Last service (job referral) is provided Sept 4th No further services are provided for the next 90 days Customer gets a job December 15th What Is The date of participation? The date of exit? the exit quarter?

EXAMPLE The exit quarter is the quarter in which the date of exit occurs Apr May June July Aug Sept Oct Nov Dec Period of participation 90 days with no service Eligibility 4/21/16 Got a job 12/15/16 Date of first service 5/4/16 Date of last service (Exit date) 9/4/16 End of 90-days 12/2/16 Exit Quarter 7/1 to 9/30

MEASUREMENT QUARTERS Employment outcomes are measured in the four quarters after the exit quarter Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 1ST Qtr After 2nd Qtr After 3rd Qtr After 4th Qtr After

GLOBAL EXCLUSIONS Participants are excluded from the common measures for the following reasons: Medical condition expected to last more than 90 days causes the person to exit the program and precludes employment Incarcerated Hospitalized or in other treatment facility Deceased Reservists called to active duty

THE MEASURES

WIOA MEASURES Adult Measures Youth Measures Employed in the 2nd quarter after exit Employed in the 4th quarter after exit Median earnings in the 2nd quarter after exit Credential attainment Skill gains Service to employers Other measures as determined Youth Measures Employed or in education or training in the 2nd quarter after exit Employed or in education or training in the 4th quarter after exit Median earnings in the 2nd quarter after exit Credential attainment Skill gains Service to employers

Measure 1: 2ND QUARTER EMPLOYMENT RATE DEFINITION: The percent of exiters who were employed anytime in the second quarter after the exit quarter Number employed in second quarter after exit quarter All exiters in that exit quarter (except global exclusions) Notes Unsubsidized employment anytime in the second quarter counts Evidence can come from UI and federal employment matches, or supplemental data (data provided by grantees in the SPIR) Not the same as the WIA Entered Employment Rate

2ND QUARTER EMPLOYMENT RATE – EXAMPLE Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 2nd Qtr After

2ND QUARTER EMPLOYMENT RATE QUIZ Scenario First service is May 4th Last service (job referral) is August 18th Customer gets a job Sept 15th Customer loses job Dec 30th Customer finds another job April 10th What is The measurement quarter for the employment rate? Is the customer a positive on this outcome?

2ND QUARTER EMPLOYMENT RATE – QUIZ CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exits 8/18 Gets Job Loses Job Gets Job Exit Quarter 2nd Qtr After Exit Employed 9/15 to 12/30 Not Employed Employed 4/10 to …

Measure 2: 4th QUARTER EMPLOYMENT RATE DEFINITION: The percent of exiters who were employed anytime in the fourth quarter after the exit quarter Number employed in fourth quarter after exit quarter All exiters in that exit quarter (except global exclusions) Notes Unsubsidized employment anytime in the fourth quarter counts Evidence can come from UI wages and federal employment matches, or supplemental data

4th QUARTER EMPLOYMENT RATE – EXAMPLE Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 4th Qtr After

Measure 3: MEDIAN EARNINGS DEFINITION: The median quarterly earnings of those employed in the second quarter after the exit quarter Notes Measured only for those employed anytime in the 2nd quarter Evidence can come from UI and federal employment matches or supplemental data If you use supplemental data to count someone as employed in the 2nd quarter, you must also measure that person’s earnings Differs from WIA Average Earnings Measurement quarter is different WIOA measures median earnings, not average earnings WIOA includes earnings measured from supplemental data

MEDIAN EARNINGS – EXAMPLE Based on our example Exit date 9/4/16 CY 2016 CY 2017 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Exit Quarter 2nd Qtr After

Half of people below midpoint CALCULATING A MEDIAN Earnings in 2nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 Half of people below midpoint Midpoint Half of people above midpoint Median= $6,000 (the earnings of the person at the midpoint) Average = 42,400 = $6,057 7

MEDIAN MINIMIZES THE EFFECT OF EXTREME VALUES Scenario 1 Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 Scenario 2 Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $15,500 Sum $49,500 Midpoint Median= $6,000 (the midpoint) Median= $6,000 (the midpoint) Average = 42,400 = $6,057 7 Average = 49,500 = $7,071 7

Middle two people CALCULATING A MEDIAN WITH AN EVEN NUMBER OF PARTICIPANTS Earnings in 2nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Person 8 $8,500 Middle two people Median= (6000 + 7000) =$6,500 2

Measure 4: CREDENTIAL RATE DEFINITION: Percent of those enrolled in education or training (excluding OJT & customized training) who obtain a postsecondary credential or high school diploma or equivalent within one year after exit date. Those obtaining a high school diploma or equivalent count as a success only if they are employed or in education or training leading to a recognized post-secondary credential within the year.

CREDENTIAL RATE Who is in the base? Who counts as a success? Exiters (except global exclusions) who, while a participant: Were in postsecondary education or training (other than OJT or customized training), or Were in secondary school (without already having a diploma) Who counts as a success? Those who, during participation or within one year after exit Obtain a postsecondary credential, or Obtain a secondary school diploma or equivalent, but only if they are also employed or enrolled in postsecondary education leading to a credential anytime within the year after exit

CREDENTIAL RATE – EXAMPLE Measured anytime from date of participation to 1 year after exit date (not tied to calendar quarters) Participation date 5/15/16 Exit date 9/4/16 CY 2016 APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR CY 2017 Participation Year After Exit Date Measurement Period is 5/15/16 to 9/3/17 (Date of Participation to 1 Year After Exit Date)

TYPES OF CREDENTIALS Designed to measure attainment of measurable technical or occupational skills necessary to obtain employment or advance Includes High school diploma or GED Associates, bachelors, or graduate degree An occupational license (e.g., awarded by a State) An occupational certificate Certificate awarded for completion of a registered apprenticeship program An occupational certification recognized by a professional, industry, or employer organization

TYPES OF CREDENTIALS Examples of what counts Certified Nursing Assistant (CNA) license Automotive Service Excellence (ASE) certification series (National Institute for ASE) Microsoft Certified IT Professional (Microsoft) Machining Level I (National Institute of Metalworking) Examples of what doesn’t count Certificates awarded by workforce boards Work readiness certificates Attainment of general skills related to safety, hygiene, etc., even if required to qualify for employment (example: OSHA 10) Certificates of completion

TESTING OUR KNOWLEDGE: EXAMPLE 1 Scenario High school student enrolls April 2016. Chris was a high school student when he enrolled in the WIOA CSP (Adult) program in April 2016. He was in danger of dropping out, so the program provided him with tutoring and other dropout prevention activities. He also participated in work experience during the summer. Exits Nov 2016 . By the fall, he was doing much better in school, and he exited the WIOA program in November 2016. Gets diploma in May 2017. Chris graduated from high school in May 2017 and got a job in June 2017. Unfortunately, he got laid off in Sept 2017. He found a great new job in Jan 2018. What are Chris’s outcomes? Is he a positive on the credential attainment rate? What other outcomes might he achieve?

TESTING OUR KNOWLEDGE : EXAMPLE 1 Got a new job 1/10/18 Exit date 11/18/16 Enrollment date 4/15/16 Graduated 5/25/17 Got a job 6/20/17 Lost job 9/18/17 APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR Oct Nov Dec Jan CY 2016 CY 2017 4th Qtr After Exit Qtr 2nd Qtr After Period of Participation Year After Exit Date Measurement period for obtaining a credential is 4/15/16 to 11/17/17 (Date of Participation to 1 Year After Exit Date)

TESTING OUR KNOWLEDGE: EXAMPLE 1 Answer Chris gets credit as a success on the Credential Attainment Rate High school diploma is an allowable credential type He obtained his diploma sometime in the measurement period To get credit for a high school diploma, you must be employed sometime in the year after the exit date, and he was Chris gets credit as a success on the 2nd Quarter Employment Rate He was employed sometime in the second quarter after the exit quarter He is included in the calculations of Median Earnings (but has very low quarterly earnings, since he was only employed 10 days in the quarter) He is a negative on the 4th Quarter Employment Rate

TESTING OUR KNOWLEDGE: EXAMPLE 2 Scenario Enrolled and exited. Carmen enrolled in WIOA in July 2016. Based on her assessment results, she enrolled in a community college Automotive Service Technician program in Sept 2016. It was a nine-month program paid for through WIOA. She dropped out of training in March 2017, before she completed, and moved out of town. Despite trying to reach her, the WIOA program didn’t hear from her for four or five months. Re-enrolled and exited. Carmen returned to the area in August, and went to the grantee for help. The grantee re-enrolled her in WIOA and paid for her to finish her training. She resumed training in Sept and finished in November, when she exited the program the second time. Outcomes. She passed the test for required for the Automotive Service Excellence Certification in Dec, and got a job shortly after. She remained employed at this firm for several years. What are Carmen’s Outcomes?

TESTING OUR KNOWLEDGE: EXAMPLE 2 A Clue!! Carmen enters and exits twice That means she has two periods of participation in WIOA She counts as two separate participants She has two measurement periods

TESTING OUR KNOWLEDGE : EXAMPLE 2 Complete training & exit WIOA 11/25/17 WIOA enrollment 7/15/16 Exit from training & WIOA 3/15/17 Return to college 9/8/17 Gets ASE certification & gets job 12/12/17 Enroll in training 9/8/16 Re-enroll in WIOA 8/25/17 CY 2016 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2017 CY 2018 First spell (7/15 to 3/15) Second spell (8/25 to 11/25)

TESTING OUR KNOWLEDGE : EXAMPLE 2 First spell of participation WIOA enrollment 7/15/16 Exit from training & WIOA 3/15/17 Gets ASE certification & gets job 12/12/17 Enroll in training 9/8/16 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2016 CY 2017 CY 2018 Exit Qtr 2nd Qtr After 4th Qtr After First measurement period for obtaining a credential is 7/15/16 to 3/14/18 (Date of Participation to 1 Year After Exit Date)

TESTING OUR KNOWLEDGE : EXAMPLE 2 Complete training & exit WIOA 11/25/17 Second spell of participation Return to college 9/8/17 Gets ASE certification & gets job 12/12/17 Re-enroll in WIOA 8/25/17 CY 2016 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN CY 2017 CY 2018 Exit Qtr 2nd Qtr After Second measurement period for obtaining a credential is 8/25/17 to 11/24/18

TESTING OUR KNOWLEDGE: EXAMPLE 2 Carmen appears as two WIOA participants Outcomes for First Spell of Participation Carmen’s first spell counts as a success on Credential Attainment Rate She earned a certification within one year of exit Carmen counts as a negative on the 2nd Quarter Employment Rate She counts as a success on the 4th Quarter Employment Rate Her earnings do not count for Median Earnings (she was not employed in the second quarter after the exit quarter) Outcomes for Second Spell of Participation Carmen’s second spell also counts as a success on Credential Attainment Carmen counts as a positive on both the 2nd Quarter Employment Rate and the 4th Quarter Employment Rate Her earnings count for Median Earnings

Measure 5: SKILL GAIN DEFINITION: Percent of participants who, during the course of the program year, are in education or training programs that lead to a recognized postsecondary credential or employment and who are achieving measurable skill gains, defined as documented academic, technical, occupational, or other forms of progress, towards such a credential or employment

SKILL GAIN continued Key Elements Designed to measure interim progress of participants, as well as those who exited during the reporting period Base includes those who, during the reporting period, were in education or training programs that lead to a recognized postsecondary credential or employment Includes those in secondary school Includes those in OJT Does not include work experience (this is not training, according to WIOA) Each participant can count as a success only once during a reporting period, even if the individual has more than one skill gain Participants can count in multiple reporting periods if they have at least one skill gain in each period Key Elements

Indicators of Skill Gains SKILL GAIN continued Documentation of skill gain includes: Increasing at least one educational functional level for those in programs below the postsecondary level Attainment of high school diploma or GED Making satisfactory progress in school (e.g., earning sufficient credit hours) Secondary school: transcript or report care for 1 semester Postsecondary: at least 12 credit hours in a semester or, for part-time students, 12 credit hours over 2 consecutive semesters Making satisfactory progress towards measurable standards as attested by an employer or training provider Passing an exam designed to measure occupational or technical skills Indicators of Skill Gains

Measure 6: SERVING EMPLOYERS Effectiveness in serving employers Three approaches being considered Retention: Percent of exiters employed in the 2nd quarter after exit who are employed with the same employer in the 4th quarter after exit Repeat Employer Customer: Percent of employers served during the year who were also served in any of the prior 2 years Employer Penetration Rate: Percent of all employers in the service area who use WIOA services

SERVING EMPLOYERS continued Calculation of the Retention Rate Number employed in the 4th quarter with the same employer as in the 2nd quarter Number employed in the 2nd quarter after exit quarter Logistics Would be calculated strictly through UI wage matching Does not require additional data collection by the grantee Could not be calculated until the 4th quarter after exit

Measure 7. INA SPECIFIC MEASURE Ability to effectively serve native peoples and communities has always been understood as a critical part of grant responsibility Rationale In addition to the required measures, ETA, in consultation with the Advisory Council, “must develop a set of performance indicators and standards that are applicable to the INA program.” Logistics ETA hasn’t made progress on these measures yet This is a statutory requirement, so work on this should start soon Discussion

(apply to those enrolled in the Supplemental Youth Services Program) The Youth Measures

Measure 1: 2ND QUARTER EDUCATION or EMPLOYMENT RATE DEFINITION: The percent of exiters who were employed or in education or training anytime in the second quarter after the exit quarter Number employed or in education/training anytime in second quarter after exit quarter All exiters (except global exclusions) Notes Unsubsidized employment or participation in secondary or postsecondary education or occupational training anytime in the second quarter counts as a positive outcome Evidence of employment can come from UI wages and federal employment matches, or supplemental data

Measure 2: 4th QUARTER EDUCATION or EMPLOYMENT RATE DEFINITION: The percent of exiters who were employed or in education or training anytime in the fourth quarter after the exit quarter Number employed or in education/training anytime in fourth quarter after exit quarter All exiters (except global exclusions) Notes Same as prior measure, but measured for the 4th quarter after the exit quarter

OTHER WIOA YOUTH MEASURES The remaining youth measures are identical to adult program measures just discussed: Median Earnings Credential Attainment Rate Skill Gain Service to Employers

(apply to those enrolled in the Supplemental Youth Services Program) Measurement Issues

WHY USE CALENDAR QUARTERS? Performance is measured using Unemployment Insurance (UI) wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages Use of UI data requires a lag of 2 quarters before data become available We also use calendar quarters because performance is measured using Unemployment Insurance (UI) wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages. More on this later in the presentation…

HOW EMPLOYMENT OUTCOMES ARE MEASURED Both wage matching and grantee data can be used Wage matching Grantees submit SSNs of participants in the SPIR DOL matches against Unemployment Insurance (UI) wage files and federal employment database, which contain employment records for (nearly) everyone employed in the U.S. Supplemental data (provided by grantees) Also as part of the SPIR, grantees indicate which participants are employed according to the grantee's own records

TYPES OF SUPPLEMENTAL DATA Documentation for supplemental data (data provided by the grantee) include: Pay stubs Letter from an employer Follow-up survey of the participant Detailed case notes signed by the counselor If supplemental data is being used, the same source must be used for measuring employment and wages

What difference does each data source make? PY 15 Performance on the WIA Measures Grantee Data Only UI Data Only Both EER 41.5% 54.9% 66.9%* Retention 75.2% 73.9% 78.8%* Avg Earnings $10,562 $9,722* na * This is the official performance in WIA

SETTING STANDARDS Using an Adjustment Model For the State programs, standards should be set using a statistical model that takes into account: Economic conditions Characteristics of participants Performance standards for the Section 166 program should use “to the extent practicable, the statistical adjustment model” used for states [WIA Section 166(h)]

OBSERVATIONS & NEXT STEPS

OBSERVATIONS Statutory Language Limits DINAP’s Options Legislation requires use of common measures for the INA program Some Important Flexibility Grantees can seek waivers Advisory Council can help develop INAP specific measures Observations and speculation: Statutory Language Limits DINAP’s Options: Legislation requires use of common measures for program Some Important Flexibility: 1) Grantees can seek waivers. 2) Advisory Council can help develop INAP specific measures

IMPLICATIONS FOR REPORTING Changed reporting forms: 9084 & 9085 will be replaced PIRL replaces the SPIR New forms have already been approved by DOL and OMB

NEW DATA SYSTEM Bear Tracks is being replaced Eliminate concepts of core and intensive services New outcomes need to be reported Some changes to info on participants’ background characteristics for the PIRL (but not many) May need to provide data on participants, not just exiters Will likely be a cloud-based system Individual-level data system being developed for both CSP and Supplemental Youth Program

SIGNIFICANT CHALLENGES There are significant challenges and unknowns It is more challenging to time exits to maximize success Measuring as far out as 4 quarters after exit will be difficult Measuring skills gains will be challenging Measures for the Supplemental Youth Program are especially challenging given the limited funding for program services Substantial uncertainty remains Phase-in and timing Employer measures and INA-specific measure(s) New data system Flexibility to request waivers and waiver process

WHAT TO EXPECT PY 2017 is likely to be a transition year No standards set, but baseline data collection likely to begin Bear Tracks SPIR will be examined for what it reveals Data on employment in the 2nd quarter Wages Credential attainment SSNs can be sent for UI matching to measure new outcomes Advisory Council likely to begin work on new measures DINAP likely to issue guidance on transition and phase-in during the summer Work on new data system to proceed

WHAT TO DO? What to do in the meantime Aggressively enter data in Bear Tracks Credentials attained Supplemental data on employment SSNs, if appropriate (UI wage matching is very helpful) Make better use of follow-up services Think about allocating budget for follow-up services Supportive services and counseling are allowed for up to 12 months after exit (see Final Rule 684.310(d)) Keep the focus on serving your customers and community Support your Advisory Council Make your challenges and needs known Seek as much flexibility as allowable under the law

RESOURCES Final Rule for ETA (issued 8/19/16) Provides regulations for ETA-only WIOA provisions Regulations for the INA program at Section 684 Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/FR-2016-08-19/pdf/2016-15975.pdf Final Joint Rule (issued 8/19/15) Describes performance standards requirements (see Section 677) Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/FR-2016-08-19/pdf/2016-15977.pdf Approved Information Collection Request Describes proposed new individual-level record layout (to replace the SPIR) and new aggregate report Available at https://doleta.gov/performance/reporting/eta_default.cfm

QUESTIONS What challenges do you see? Questions?

Social Policy Research Associates CONTACT INFORMATION 71 Ron D’Amico Principal Social Policy Research Associates 510-788-2484 Ron_damico@spra.com