1. WIA PERFORMANCE MEASURES AND STANDARDS: The WIASRD, Common Measures and Standards Negotiation Challenges Christopher T. King Ray Marshall Center for the Study of Human Resources University of Texas, Austin ctking@uts.cc.utexas.edu 512/471-2186 David W. Stevens The Jacob France Institute University of Baltimore dstevens@ubmail.ubalt.edu 410/837-4729 April 22, 2003

2. BRIEFING TOPICS 1.Highlights from PY 2000 program outcome information in the WIASRD files from the seven ADARE Project states, focusing on the quality of the data elements. 2.Negotiated, actual and actual minus negotiated difference in PY 2000 performance data for the seven ADARE Project states. 3.Observations about the proposed common measures. 4.WIA performance standards negotiation challenges and opportunities (including pros and cons of regression modeling). 5.Other challenges that will follow reauthorization.

3. EMPLOYED IN QUARTER AFTER EXIT QUARTER The data element code choices are: yes, no and not yet available Georgia, Illinois and Missouri did not use the not yet available code. The four ADARE Project states that used the not yet available code used it the following percent of the time:  Florida44 percent  Maryland73 percent  Texas23 percent  Washington50 percent

4. USE AND SOURCE OF SUPPLEMENTAL DATA The data element code choices are: used case management files and record sharing/matching  Florida, Missouri and Washington did not report any use of supplemental data sources.  Georgia reported only three instances of supplemental data use.  Texas reported using supplemental data one percent of the time.  Illinois and Maryland reported using supplemental data three percent of the time.

5. OCCUPATIONAL CODE of any job held since exit This information is to be reported if the individual is reported as employed in the quarter after exit. The information can be based on information derived from case management files, follow-up services or other sources. It is not necessary to wait until information on employed in quarter after exit is available.  Florida, Georgia and Maryland used only the nine-digit DOT code.  Illinois and Texas used only the five-digit OES code.  Washington used both the DOT and OES coding taxonomies.  Missouri used the five-digit or six-digit O*Net98 code.

6. ENTERED TRAINING RELATED EMPLOYMENT Two-thirds of the yes or no entries for this data element were recorded as a yes. The range of affirmative entries was from a low of 29 percent for Maryland to a high of 94 percent for Florida.  The reported method used by Florida, Maryland, Texas and Washington to determine training related employment was ‘other appropriate method’.  The reported method used most often by Georgia, Illinois and Missouri was ‘a comparison of the occupational codes of the training activity and the job’, but each of these three states also used ‘a comparison of the industry of employment with the occupation of training using an appropriate crosswalk’.

7. ENTERED NONTRADITIONAL EMPLOYMENT The nontraditional employment designation can be based on either local or national data.  Six percent of the yes or no entries for this data element were reported as a yes.  The range of affirmative entries among the seven ADARE Project states was from a low of one percent to a high of fifteen percent.  Texas did not report yes or no entries for this data element.

8. TYPE OF RECOGNIZED EDUCATIONAL/OCCUPATIONAL CERTIFICATE, CREDENTIAL, DIPLOMA OR DEGREE ATTAINED  Seven codes are provided. States and localities have flexibility in choosing the methods used to collect data documenting this data element.  Each of the seven ADARE Project states reported award of some credentials in each of the six type of credential categories.

9. PY 2000 CORE MEASURES OF PERFORMANCE SEVEN ADARE PROJECT STATES The four Adult and Dislocated Worker performance measures are covered.  Entered employment rate.  Employment and credential rate.  Retention rate.  Earnings change Each of the four charts that follow ‘flies in’ PY 2000 negotiated, actual and actual minus negotiated performance measure values for the seven ADARE Project states.

10. QUESTIONS TO ASK WHEN LOOKING AT THE CHARTS THAT FOLLOW  Do I know enough about the criteria for specifying each negotiated performance measure value to interpret the observed differences in these negotiated values among the seven ADARE Project states?  Do I know enough about the data sources that were used to calculate the actual performance measure values to interpret the actual minus negotiated differences in these values among the seven ADARE Project states?  What management and/or policy conclusions can I reach based on my answers to the previous two questions?  Can I be confident in making incentive awards and imposing sanctions based on actual minus negotiated value differences?

11. Program Year 2000 (July 2000-June 2001): Entered Employment Rate

12. Program Year 2000 (July 2000-June 2001): Employment And Credential Rate

13. Program Year 2000 (July 2000-June 2001): Retention Rate

14. Program Year 2000 (July 2000-June 2001): Earnings Change

15. REVISITING THE QUESTIONS ASKED HAVING LOOKED AT THE CHARTS  Do I know enough about the criteria for specifying each negotiated performance measure value to interpret the observed differences in these negotiated values among the seven ADARE Project states?  Do I know enough about the data sources that were used to calculate the actual performance measure values to interpret the actual minus negotiated differences in these values among the seven ADARE Project states?  What management and/or policy conclusions can I reach based on my answers to the previous two questions?  Can I be confident in making incentive awards and imposing sanctions based on actual minus negotiated value differences?

16. COMMON MEASURE ISSUES Performance Measure Quality ENTERED EMPLOYMENT RATE  Registration date  Employed or not employed at registration  Exit date  Entered employment by the end of the first quarter after exit ISSUES  Staff decision whether and when to register a customer  Quality of ‘employed or not employed at registration’ data element  Unintended consequences of this measure  Staff decision when to enter or allow automatic entry of exit date  Use of supplemental data sources

17. COMMON MEASURE ISSUES Performance Measure Quality EMPLOYMENT RETENTION RATE  Employed first quarter after exit (regardless of employment status at time of registration)  Employed second quarter after exit  Employed third quarter after exit ISSUES  Stakeholder interest in this measure  Drill-down questions that will be asked  Use of supplemental data sources  Timeliness of availability for intended uses

18. COMMON MEASURE ISSUES Performance Measure Quality EARNINGS INCREASE  Earnings in second quarter prior to registration  Employed in first quarter after exit  Earnings in first quarter after exit  Earnings in third quarter after exit ISSUES  Stakeholder interest in this measure  Drill-down questions that will be asked  Number of ‘pays’ in each reference quarter  Use of supplemental data sources  Timeliness of availability for intended uses

19. COMMON MEASURE ISSUES Performance Measure Quality EFFICIENCY  The dollar amount specification to serve as the numerator  The number of participants figure to serve as the denominator ISSUES  Stakeholder interest in this measure  Drill-down questions that will be asked  Quality of data elements  Unintended consequences

20. COMMON MEASURE ISSUES Performance Measure Quality PLACEMENT IN EMPLOYMENT OR EDUCATION  Registration date  Enrolled in secondary education at registration  Exit date  Not enrolled in post-secondary education at registration  Not employed at registration  Enrolled in secondary education at exit  Employed in first quarter after exit  In military service in first quarter after exit  Enrolled in post-secondary education in first quarter after exit  Enrolled in advanced training/occupational skills training in first quarter after exit CONTINUED……

21. COMMON MEASURE ISSUES Performance Measure Quality PLACEMENT IN EMPLOYMENT OR EDUCATION CONTINUED…. ISSUES  Stakeholder interest in this measure  Drill-down questions that will be asked  Quality/uniformity of data definitions and sources  Cost of data collection  Access to education records  Timeliness of data availability for intended uses  Unintended consequence—proliferation of credentials

22. COMMON MEASURE ISSUES Performance Measure Quality ATTAINMENT OF A DEGREE OR CERTIFICATE  Registration date  Enrolled in education  Exit date  Attained a diploma, GED, or certificate by the end of the third quarter after exit ISSUES  Stakeholder interest in this measure  Drill-down questions that will be asked  Access to education records  Quality/uniformity of data definitions and sources  Timeliness of data availability  Unintended consequences

23. COMMON MEASURE ISSUES Performance Measure Quality LITERACY OR NUMERACY GAINS ?

24. COMMON MEASURE ISSUES Performance Measure Quality FIVE ISSUES ARE OF PARTICULAR IMPORTANCE AND CONCERN:  The accuracy and probable unintended consequences associated with the employed or not employed at registration data element  The integrity and value-added of supplemental data use  Selection of denominator and numerator definitions for the proposed efficiency measure  The complexity and value-added of the placement in employment or education measure  Expected unintended consequences associated with the attainment of a degree of certificate measure

25. PERFORMANCE STANDARD ISSUES THREE TOPICS ARE OF PARTICULAR IMPORTANCE: State and Local Workforce Area Benchmarking  The Census Bureau LEHD Program as a potential source of local demographic and economic activity information for discretionary use in negotiation of state and local performance standards  Benchmarking of own performance over time  Benchmarking against other ‘similar’ states or Local Workforce Areas  Return to regression modeling? Pros and cons CONTINUED….

26. PERFORMANCE STANDARD ISSUES Challenges Associated with Pursuing Continuous Improvement  Integrity of state and local management information systems over time  Continuity of data source availability and content over time  Expected unintended consequences

27. PERFORMANCE STANDARD ISSUES Vulnerability to Unintended State and Local Actions  Discretionary opportunities to define selection in criteria, assignment to service components criteria (including whether and when to use partner services) and timing of exit criteria  Investment in staff development can reduce the frequency of some of the unwanted behaviors that will otherwise follow introduction of the common measures

28. OTHER CHALLENGES  Occupations in demand  Required registration of some customers  Stakeholder interest in number of customers served