1. Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training Richard West Social Policy Research Associates

2. Topics to Be Covered WIA Performance Measures –Definitions and calculations Challenges –Working with wage records –Supplemental data –When outcomes occur and are measured –Understanding performance reports –Implementing new data collection processes –What about those JTPA exiters included in WIA performance measures?

3. Separate measures for funding streams and youth age groups –Adults –Dislocated workers –Youth under 19 at registration –Youth 19 to 21 (under 22) at registration. Individual served with multiple funding streams included in all relevant measures. (Nearly) Common measures for adults, dislocated workers, and older youth. 17 Performance Measures

4. For adults, dislocated workers, older youth: –Entered employment rate –Employment retention rate –Earnings change (Adults, Older Youth ) or earnings replacement rate (DWs) –Credential attainment For younger youth: –Goal attainment –Diploma attainment –Retention Customer satisfaction –Participants –Employers 3x4=12 3 2 17

5. Performance Measures Keyed to the Exit Quarter Exit quarter is the quarter containing the exit date. Individuals exit if: –Formally withdraw from or complete services OR –There is no service for 90 days (WIA or tracked partner-funded services) Exit date is the date of withdrawal or completion or the date of the last service.

6. WIA Exit WIA ParticipationNo services Last Service End of 90-day period WIA Exit Date

7. Time Periods Used for the Measures  Measures that use UI wage records.  Exiters during the last 3 quarters of the previous PY and the first quarter of the current PY.  October 1 to September 30  Applies to all other measures  Measures that use "real time" data.  Exiters who leave during the program year.  Applies to measures that do not use UI wage records:  Younger youth goal attainment  Younger youth diploma  Customer satisfaction

8. Exit Periods for Performance 14321432 Previous PYCurrent PY Real Time Measures Wage Record Measures EE2Q Lag

9. Challenge: Working with Wage Records

10. Sources of Employment Data Wage records the primary source for entered employment and retention; Wage records do not include: –Individuals in uncovered employment, such as self-employment –Federal government employees and others outside the state UI system May use out-of-state wage records –Bilateral agreements with nearby states –Wage Record Interchange System (WRIS)

11. When Are Wage Records Good Enough? Measures sensitive to incomplete wage records Example: –Wage records 90% complete –“True” employment is 70% –Then measured employment is: 0.9 x 70 = 63% A seven percentage point bias Really want records close to 99% complete Typically a one or two quarter lag before wage records are good enough

12. WRIS Enables states to obtain out-of-state wage records efficiently and cost-effectively. WRIS is currently fully operational with 20 pilot states By July 1, 2001 all states will be able to join. Expected to be self supporting –Cost of participation has not been determined. –An initial pricing study is available on WorkforceATM.org, as is other more information on WRIS

13. How Does WRIS Work? State submits a list of SSNs for matching WRIS matches those SSNs against an index to identify the states with records for those SSNs. Wage request is sent by WRIS to only those states with records for an SSN. Receiving states send wage records back to WRIS. Requesting states can download the information from WRIS.

14. Challenge: Developing a Realistic Process for Collecting Supplemental Data

15. Sources of Employment Data 2 Supplemental data sources may be used if not person not found in wage records. –Case management, follow-up services, participant surveys. –Records sharing or automated record matching with other administrative databases. Note: records that provide information on quarterly earnings will be considered as wage records, –Must be collected within 30 days after individual determined to not be in wage records

16. Supplemental Data “By the Book” State gets wage record data Local area is told who is not in wage records Local area collects supplemental data within 30 days Supplemental data is reported to state

17. 19876432 Quarters After Exit Exit EE2Q Lag SD 30 days to get supplemental 5 Can get supplemental early during follow-up services RTN Follow-up Services Get Wage Records SD Data Collection Timing for Employment

18. “Real World” Supplemental Data Local areas collect supplemental data during follow-up services –For everyone –For those likely to not be in wage records: Self-employed, other noncovered jobs Out of state employment State gets wage record data and tells local areas who is not in wage records Use supplemental for those not in wage records Supplemental data reported to state

19. Calculating Performance Measures

20. Global Exclusions Exclusions from all performance measures: –Adults & dislocated workers who received only self-service and informational core services –Institutionalized at exit (Item 622 = 1) –Health/medical exits (Item 622 = 2) –Deceased (Item 622 = 3) –National Emergency Grants only –Rapid Response only (except additional assistance) –Some statewide programs only

21. What Statewide Programs Can Be Excluded? Very specialized statewide programs with their own specific goals, for example: –Incumbent worker training authorized under Section 134(a)(3)(A)(iv)(I), or –A project for chemically dependent TANF recipients. These statewide programs are also excluded from WIASRD reporting (if they are the only program)

22. Youth Statewide? Item 307 = 1 Youth YES Displaced Homemaker Item 308 = 1 Adult or Youth YES Incumbent Worker Item 309 = 1 only Excluded YES Other Statewide Item 310 = 1 Adult or Dislocated Worker YES Rapid Response only Item 311 = 1 Excluded YES Rapid Response Additional Assistance Item 312 = 1 Dislocated Worker YES In What Performance Groups do Statewide Participants Count?

23. Displaced Homemaker Item 308 = 1 Youth Other Statewide Item 310 = 1 Adult Ambiguous Cases--Suggested Approach If already in a performance group, ignore these statewide programs Has Dislocation Date? Item 125 or also in NEG grant Item 313a Dislocated Worker NO YES NO YES NO Age < 18? Age >= 22? YES NO Youth Services? YES NO

24. Entered Employment Rate –Wage records the primary data source –Supplemental data may be used for persons not found in wage records –Exclude adults, older youth employed at registration –Exclude older youth in advanced training/ postsecondary education (but not employed) in quarter after exit. Number Employed in Quarter after Exit divided by Number of Exiters

25. YES NO Entered Employment Rate Adults and Dislocated Workers Employed at Registration? Item 115 = 1 Excluded Employed in Quarter After Exit ? Item 601 = 1 Numerator Denominator NO Dislocated Worker Measure? YES NO

26. YES NO YES Entered Employment Rate Older Youth Excluded Numerator Denominator Excluded YES In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 NO Employed in Quarter After Exit Item 601 = 1 NO Employed at Registration? Item 115 = 1

27. Employment Retention –Based on those employed in quarter after exit –Wage records the primary data source –Can use supplemental data if not in wage records. –Excludes older youth in advanced training/ postsecondary education (but not employed) in 3rd quarter after exit. Of those employed in the quarter after exit: Number Employed in 3rd Quarter after Exit divided by Number Employed in Quarter after Exit

28. NO YES Employment Retention Rate Adults and Dislocated Workers Excluded Numerator Denominator NO Employed in 3rd Quarter After Exit Item 608 = 1 YES Employed in Quarter After Exit Item 601 = 1

29. YES NO YES NO YES Employment Retention Rate Older Youth Employed Quarter After Exit? Item 601 = 1 Excluded Employed 3rd Quarter After Exit Item 608 =1 Numerator Denominator Excluded NO In Advanced Training or Postsecondary Education in 3rd Quarter After Exit Item 624 = 1

30. Earnings Change Adults and Older Youth –Wage records only Exclude if supplemental data used to determine employment: –in either the 1st or 3rd quarter after exit. Exclude if preprogram quarters too far back to get data –Exclude older youth in advanced training/ postsec. education (but not employed) in 3rd quarter after exit. divided by Number employed in quarter after exit Of those employed in the quarter after exit: minus Earnings in 2nd and 3rd Quarters after Exit Earnings in 2nd and 3rd Quarters before Registration

31. Earnings Replacement Dislocated Workers –Wage records only Exclude if supplemental data used to determine employment: –in either the 1st or 3rd quarter after exit. Exclude if preprogram quarters too far back to get data Use 2nd and 3rd quarters before registration if no dislocation date or dislocation date after registration date –Add earnings over all included exiters before dividing Of those employed in the quarter after exit: Earnings in 2nd and 3rd Quarters after Exit divided by Earnings in 2nd and 3rd Quarters before Dislocation

32. NO Earnings Change--Adults Earnings Replacement--DWs Employed in Quarter After Exit Item 601 = 1 Excluded Numerator and Denominator YES Supplemental Data Used Q1 Item 602 = 1 or 2 Supplemental Data Used Q3 Item 609 = 1 or 2 YES NO YES

33. In Advanced Training or Postsecondary Education in 3rd Quarter After Exit Item 624 = 1 YES Yes NO Earnings Change Older Youth Employed Quarter After Exit? Item 601 = 1 Excluded Supplemental Data Used Q1 Item 602 =1 or 2 Included NO Employed 3rd Quarter After Exit Item 608 =1 Supplemental Data Used Q3 Item 609 =1 or 2 YES NO YES NO YES

34. Credential Rate Flexible definition of credential –Standard credentials –Certificates that recognize the successful completion of training services …. Credential attainment by end of 3rd qtr after exit Positive outcome is attain credential and: –Enter employment (adults, dislocated workers) –Enter employment or in advanced training/postsec. educ. in quarter after exit (older youth) Includes: –Adults/DWs who received training. –All older youth

35. Credential--Adults and DWs Of those who receive training : Credential--Older Youth Number who enter employment and or are in advanced training or postsecondary education in the quarter after exit divided by Number of exiters Number who enter employment and attain credential divided by Number of exiters

36. YES NO YES Employment and Credential Rate Adults and Dislocated Workers Excluded Numerator Denominator NO YES Attained a credential Item 621 = 1, 2,3, 4, 5, or 6 Employed in Quarter After Exit Item 601 =1 YES Received Training? Item 333 is a valid date

37. YES Credential Rate--Older Youth Employed in Quarter After Exit Item 601 =1 Denominator NO YES Numerator NO In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 Attained a credential Item 621 = 1, 2,3, 4, 5, or 6

38. Younger Youth Goal Attainment Outcome attained during participation 3 goal areas: –Basic skills A basic skills goal must be set if the youth is deficient in basic skills –Work readiness skills –Occupational skills Flexibility in defining goals Youth has a year to attain the goal Maximum of 3 goals set per year (for performance) –First year begins on registration date and ends a year later. –Goals used for performance must be identified up front

39. Younger Youth Goal Attainment-2 Add numerator and denominator over all youth before dividing. Include all in-school youth and out-of- school youth assessed as needing basic skills, work readiness skills, or occupational skills Number of goals attained divided by Number of goals attained or reached anniversary/exit date without attaining

40. HS Diploma/GED Attainment Diploma can be attained during participation or by end of quarter after exit Exclude youth still in secondary school at exit Of youth who were not HS grads at registration: Number of exiters who attained diploma divided by Number of exiters not still in school at exit

41. YES NO YES NO Younger Youth Diploma Rate HS grad at registration? Item 123 > 11; Item 130 = 2 or 4 Excluded Attained Diploma? Item 673=1 or 2 Numerator Denominator Excluded YES In Secondary School at Exit? Item 673 = 3

42. Younger Youth Retention In one or more qualifying outcomes in 3rd quarter after exit –Postsecondary education (Item 676 = 1) –Advanced training (Item 676 = 2) –Military service (Item 676 = 3) –Qualified apprenticeship (Item 675 = 4) –Unsubsidized employment (Item 676 = 5) Exclude youth in secondary school at exit Not related to outcome in quarter after exit

43. YES NO Younger Youth Retention Rate In Secondary School at Exit? Item 673 = 3 Numerator Denominator Excluded Retained in 3rd Quarter After Exit Item 676=1,2,3,4, or 5

44. Challenge Understanding When Outcomes Occur and When they Are Measured

45. Q1Q2Q4Q3Q2Q1Q4Q3 PY 2000PY 2001 Entered Employ. Exit EE Retention RTN Credential 12 Mo Youth Diploma       12 Mo. Retention When outcome occurs  =When Measured When Outcomes Occur

46. Challenge Understanding Performance Reports What you get when you ask for “the most recent data available”

47. 14321432 Previous PYCurrent PY Real Time Measures Wage Record Measures EE Exiters RTN Earnings Yth Diploma Look at Data Available May 15 Yth Goals Credential Cust Sat

48. Challenge: Implementing New Data Collection Processes

49. Outcome Data Collection Activities Wage records –Collect quarterly (both in and out of state) Supplemental employment data –Collect during follow-up services –Collect quarterly after not found in wage records if necessary Credential –Collect during participation –Update after the 3rd quarter after exit Diploma –Collect during participation –Update after the quarter after exit

50. Challenge: What about those JTPA Exiters Included in WIA Performance Measures?

51. JTPA exiters from October 1, 1999 to June 30, 2000 are included in : –All adult, dislocated worker, and older youth measures –Younger youth retention measure. Are you getting wage records for them? Are you collecting supplemental employment data? (Do you need it to meet your goal?) Are you collecting credential data? Are you collecting younger youth retention outcomes? It’s not too late now, but …...