1 Performance Measurement & Reporting For Employment And Training Programs

2 Agenda Quick review of Agenda and Ground Rules for the Session This morning’s session: –Foundation for Performance –Revisions to existing reporting requirements –Data sources –Calculating outcomes on the performance measures

3 Ground Rules for Today & Tomorrow 1)Cell phones on mute 2)Questions, Answers and Parking Lot issues– Flip Charts 3)The session is designed to provide information on the definitions of the common measures as outlined in TEGL and on the revised reporting instructions. We cannot debate whether or not common measures are “good or bad.”

4 A Foundation for Performance Management

5 Improving Performance Results Reporting performance is a fundamental element of improving customer services and good public administration Performance information should be easily understood by all customers, stakeholders, and operators of the workforce investment system Establishing common measures and standardizing customer data collection improves the comparability and understanding of performance results Ensuring the accuracy and timeliness of performance results is necessary for demonstrating system integrity and value

6 Establishing Common Measures President’s Management Agenda Affects six agencies (29 programs) –Department of Labor –Department of Education –Department of Health and Human Svcs. –Department of Veterans Affairs –Department of the Interior –Department of Housing and Urban Development TEGL issued on April 15, 2005 –Rescinds TEGL –Policy applies to DOL funded programs only –July 1, 2005, implementation for W-P, VETS and WIA; October 1, 2005, implementation for TAA

7 The Common Measures Adult measures: –Entered employment –Employment retention –Earnings increase Youth and lifelong learning measures: –Placement in employment or education –Attainment of a degree or certificate –Literacy or numeracy gains Designations of adult or youth are spelled out in each program’s eligibility requirements

8 Benefits of Common Measures Focus on the core purposes of the workforce system; employment for adults and skill gain for youth Break down barriers to integration resulting from different definitions, data and reports for each workforce program Resolve questions raised by GAO and other oversight agencies regarding the consistency and reliability of data Reduce confusion among our customers and stakeholders who want to know about results

9 The Intersection of Common Measures and Revised Reporting Requirements for Wagner-Peyser, VETS, TAA & WIA Programs

10 Common measure outcomes will be collected through revised reports Common measure definitions replace existing program-specific performance measure definitions where possible Efficiency measures to be computed at Federal level only Roll-Out Of Common Measures

11 Wagner-Peyser and VETS Program Measures Roll-Out Of Common Measures Wagner-Peyser Wagner-Peyser Performance Measures Job seeker entered employment rate Job seeker employment retention at six months Job seeker customer satisfaction score Employer customer satisfaction score Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase

12 Adult earnings increase Adult employment retention rate Job seeker employment retention at six months Wagner-Peyser Wagner-Peyser Performance Measures Job seeker customer satisfaction score Employer customer satisfaction score Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase Adult entered employment rate Job seeker entered employment rate Wagner-Peyser and VETS Program Measures Roll-Out Of Common Measures PY05 will be the baseline year for these measures. Goals will be negotiated beginning with PY06.

13 TAA TAA Performance Measures Entered employment rate Employment retention rate at six months Earnings replacement rate Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase TAA Performance Measures Roll-Out Of Common Measures

14 Adult earnings increase Earnings replacement rate Adult employment retention rate Employment retention rate at six months TAA TAA Performance Measures Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase Adult entered employment rate Entered employment rate TAA Performance Measures Roll-Out Of Common Measures States are responsible for achieving National Goals for the Trade program.

15 WIA Adult WIA Adult Performance Measures Entered employment rate Employment retention rate at six months Earnings gain Employment and credential rate Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase WIA Adult Program Measures Roll-Out Of Common Measures

16 Adult earnings increase Earnings gain Adult employment retention rate Employment retention rate at six months WIA Adult WIA Adult Performance Measures Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase Adult entered employment rate Entered employment rate Employment and credential rate WIA Adult Program Measures Roll-Out Of Common Measures Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential for PY05 and PY06.

17 WIA Dislocated Worker WIA Dislocated Worker Performance Measures Entered employment rate Employment retention rate at six months Earnings replacement Employment and credential rate Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase WIA Dislocated Worker Program Measures Roll-Out Of Common Measures

18 Adult earnings increase Earnings replacement Adult employment retention rate Employment retention rate at six months WIA Dislocated Worker WIA Dislocated Worker Performance Measures Adult Common Performance Measures Adult entered employment rate Adult employment retention rate Adult earnings increase Adult entered employment rate Entered employment rate Employment and credential rate WIA Dislocated Worker Program Measures Roll-Out Of Common Measures Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential measures for PY05 and PY06.

19 WIA Youth Program Measures Roll-Out Of Common Measures Youth Common Performance Measures Placement in employment or education/ advanced training (PY05) Attainment of a degree or certificate (PY05) Literacy and numeracy gains (PY05/06) States have not negotiated performance goals for the youth common measures, however states will be expected to begin reporting on these measures in PY05 and PY06. States have negotiated performance goals for WIA programs according to statutory measures for PY 2005 and PY 2006 Younger Youth Skill attainment rate Diploma attainment rate Retention rate Older Youth Entered employment Employment retention Earnings change Employment and credential rate

20 Revised Reporting Requirements ETA received emergency approval in April 2005 to incorporate common measures into the WIA, W-P, VETS, and TAA program reporting systems Remember, where possible the definitions for common performance measures are replacing “old” definitions These changes are effective on July 1, 2005 for WIA W-P and VETS programs, and October 1, 2005, for the TAA program  The Department published three 60-day public comment Information Collection Requests in the Federal Register as part of the process to obtain full approval of these revisions. The public comment period ends September 12, Comments should be sent to: Dr. Esther R. Johnson, Administrator Office of Performance and Technology Department of Labor/Employment and Training Administration 200 Constitution Avenue NW Room S5206 Washington, DC Electronic copies should be sent to

21 Revised Reporting Requirements What the revised reporting accomplishes Facilitates the collection and reporting of information, including the common performance measures Where possible, standardizes reporting elements across programs Continues to communicate program performance results and holds states accountable for federal funds

22 Revisions to Reporting Requirements WIA Performance Reporting System  Quarterly Report Collects aggregate counts of self-service participants, the number of participants in training, and youth (by age and school status) Collects current WIA youth measures plus common measures Separate reporting of common measures for Adults, Dislocated Workers, and NEG participants Rolling four-quarter reporting methodology to align with W-P employment services Reporting System Eliminated customer satisfaction reporting on a quarterly basis  Annual Report Collects similar aggregate counts of participants and exiters as the Quarterly Report New report table to collect youth common measures  WIASRD Collects additional data on participant services and outcomes to calculate common measures States do not submit WIASRD files on participants who receive only self- services and informational activities States must report co-enrollment of individuals in Wagner-Peyser or TAA programs

23 Revisions to Reporting Requirements Wagner-Peyser Reporting System (W-P & VETS)  Aligns performance reporting cohorts with WIA reporting cohorts  Eliminates all customer satisfaction reporting  Collects aggregate counts on participants, exiters, and those receiving workforce information services  Collects aggregate outcomes for the adult common measures  Collects aggregate data on services to Homeless Veterans (VETS 200 C report only)  Eliminates reporting on each category of veteran (e.g., campaign, disabled) by age grouping Trade Act Participant Report (TAPR)  Collects data elements to calculate common measures  Collects information on participant services as authorized under the TAA Reform Act of 2002

24 Important: Reporting Reminder!!  Reporting for WIA, W-P, and VETS for PY 2004 (which ended June 30, 2005) has not changed and does not reflect common performance measures –No change to 4 th quarter ES 9002 or VETS 200 reports, due in August –No change to 4 th quarter WIA 9090, due in August, or the WIA Annual or WIASRD, due in October  Reporting for TAA for FY 2005 (which will end September 30, 2005) has not changed and does not reflect common performance measures –No change to 4th quarter TAPR, due in November

25 Important: Data Validation Update ETA is upgrading data validation and reporting software to reflect changes in the WIA reporting system –Revised import file format will be available in early August 2005 Based on revised WIASRD specifications Modified edit checks/coding procedures Conversion rules from old-to-new WIASRD format –Projected release of revised WIA data validation and reporting software is Mid-October 2005 All data validation and reporting documentation will be available on the web at

26 Key Concepts

27 Program Participation A participant is a person who is determined eligible to participate in a program and receives a service funded by the program in a physical location The date of participation is the date of the first service after eligibility determination Individuals who receive services that are accessed by virtual means outside of the program’s facility need not, but may, be treated as participants ___________________

28 Program Participation Generally, all participants will be taken into account when computing performance outcomes under the common measures Two types of participants may be excluded from the performance calculations: Participants who do not voluntarily disclose valid Social Security numbers Participants who are excluded from the calculations by law ____________________

29 “Participant” – Wagner-Peyser Act Wagner-Peyser employment services –Universal Access –No formal determination of eligibility is required DVOP/LVER services –veterans, eligible persons, transitioning service members who meet eligibility criteria

30 Clarification of “Self-directed Job Search” Exclusion Application to W-P employment services: –Exclusion is limited to the act of conducting a job search, job bank browsing or modifying job search requirements. –Some self-directed job search activities should be considered reportable service (e.g. building resumes, accessing workforce information, and enrolling in online job search mini courses.)

31  At a minimum, data needed from self- service participants:  social security number (SSN)  employment status at the point of participation.  Upon request of a customer’s SSN, the customer must also be offered the opportunity to provide Equal Employment/demographic information. Minimum Data Needed from Self-Service Participants

32 Program Exit  Exit occurs when... Participant does not receive a service or partner service for 90 consecutive calendar days; Participant has no planned gap in service; and Participant is not scheduled for future services  Date of exit is applied retroactively to the last day of service funded by the program or partner program  Must have the capability to track partner services in order to extend the date of exit  No More Hard Exit

33 Program Exit Days included in any planned gap in services do not count when determining the exit date –Delay before the beginning of training –Health condition or providing care for a family member with a health condition –Temporary move from the area –All gaps in service must be documented

34 “Exiter” – Wagner-Peyser Act Significant change from former Wagner- Peyser performance measures, which calculated performance outcomes based on registration quarter.

35 Global Exclusions Exclusions from all common measures: –Persons institutionalized (exceptions are detained Responsible Reintegration of Youthful Offenders and disabled individuals residing in institutions) –Persons unable to continue due to health/medical or family care reasons –Deceased –Reservists called to active duty –Relocated to a mandated residential program (youth only) To be excluded from performance calculations, the condition(s) must be expected to last 90 days or more

36 Data Sources

37 Unemployment Insurance wage records –Includes state wage records and additional sources such as WRIS and FEDES –The first source for employment and retention measures –The only source for earnings increase Supplemental data may be used to determine employment and retention Outcome Data Collection

38 Administrative records are the source for outcomes that are not employment- based Administrative data may be obtained through automated data exchanges Administrative record data (except automated data exchanges) and data collection methods must be documented and subject to audit Outcome Data Collection

39 Calculating Outcomes Adult Measures

40 Summary of Major Policy Changes TEGL Old Policy TEGL New Policy ADULT MEASURES Entered Employment RateNo change in methodology from WIA or TEGL Permits grantees to use wage records and supplemental data Employment Retention Rate Measurement at 1 st, 2 nd, and 3 rd quarters No change in methodology from TEGL Permits grantees to use wage records and supplemental data Earnings Increase Measure Separated into two distinct indicators of performance 1.Percent change pre- to post- program Qtr 1 2.Percent change post-program Qtr 1 to Qtr 3 Replaces the two-part earnings increase measure identified in TEGL with a single Six Month Earnings Increase measure (pre- to post-program) Wage records are the only data source Grantees that do not have access to wage records can use supplemental data as an interim means of reporting Program Efficiency Identified as a core common measure Eliminates program efficiency as a reportable measure at the grantee level

41 Adult Entered Employment Exclude adults employed at the time of participation (including those individuals who have taken a job after being laid-off or dislocated) Include adults employed at the time of participation who are on layoff notice Wage records or supplemental data may be used to determine employment in the quarter after exit Supplemental data must be documented Employment at participation is based on information from the individual – not wage records Computing Outcomes Number employed in quarter after exit Number of exiters Of those not employed at participation:

42 YES Numerator Denominator Excluded YES Employed in quarter after exit? NO Employed at participation? Adult Entered Employment NO

43 Adult Employment Retention Based on those employed in quarter after exit Wage records and supplemental data are acceptable data sources –Supplemental data must be documented Employment in the 1 st, 2 nd and 3 rd quarters after exit does not have to be with the same employer Must be employed in both the 2 nd and 3 rd quarters after exit to be retained Computing Outcomes Of those employed in the quarter after exit: Number employed in both the 2 nd and 3 rd quarters after exit Number of exiters

44 Employed in quarter after exit? YES Employed in 2 nd and 3 rd qtr. after exit? YES Numerator Denominator Excluded NO Adult Employment Retention NO

45 Adult Earnings Increase Only wage records may be used for determining pre-program and post- program earnings Includes same population as the adult retention measure Likely to be negative for dislocated workers False, erroneous and misleading earnings data may be excluded by the grantee - Number exiters Of those employed in the quarter after exit: Earnings in 2 nd and 3rd quarter after exit Earnings in 2 nd and 3rd quarter prior to participation Computing Outcomes

46 YES NO Employed in qtr. after exit? Excluded Supplemental data used in Q1? Included NO Supplemental data used in Q3? YES Adult Earnings Increase NO YES

47 Calculating Outcomes Youth Measures

48 Youth Measures Requirements Reporting for PY 05: –7 current WIA youth measures (4 OY and 3 YY) – negotiate on these for accountability –First 2 youth common measures (placement and certificate) –Literacy/Numeracy common measure encouraged, but not required (required in PY 06) –Literacy/Numeracy only include new youth enrolling after July 1 of the year in which the State implements it –No negotiation on common measures in PY 05

49 Youth and Lifelong Learning Common Measures Will include all youth, no distinction between younger/older 1) Placement in Employment, Education, or Training 2) Attainment of a Degree/Certificate 3) Literacy/Numeracy Gains

50 Summary of Major Policy Changes TEGL Old Policy TEGL New Policy YOUTH MEASURES Placement in Employment or Education Excluded individuals in secondary school at exit from performance calculations Individuals in secondary school at exit are now included in the measure Permits grantees to use wage records and supplemental data Attainment of Degree or Certificate Excluded individuals in secondary school at exit from performance calculations Individuals in secondary school at exit are now included in the measure Clarifies the definition of Diploma Includes degrees/certificates awarded by tribally-controlled colleges and universities Literacy and Numeracy Gains Included all in-school and out-of-school youth who were basic skills deficient In-school youth are now excluded from the measure Provides additional guidance on providing reasonable accommodations for testing youth with disabilities Program Efficiency identified as a core common measure Eliminates Program Efficiency as a reportable measure at the grantee level

51 Youth And Lifelong Learning Placement In Employment Or Education Excludes youth in employment, the military, or post- secondary education at participation Includes youth in secondary education at exit Employment, military and education status at participation are based on information from the individual Number of participants in employment or the military or enrolled in post-secondary education and/or advanced training/occupational skills training in the quarter after exit Number of exiters Of those who are not in any of the following at participation - post-secondary education, employment, or the military: Computing Outcomes

52 Youth And Lifelong Learning Placement In Employment Or Education Employment and military service in the 1 st quarter after exit may be based on wage record data or supplemental data Supplemental data must be documented Similar to current YY Retention measure in that it recognizes the same outcomes – improvement from OY Entered Employment However, it includes first quarter after exit only – no concept of retention or earnings change in common measures for youth programs

53 NO YES Numerator Denominator NO Did the qualifying outcome occur in the 1 st quarter after exit? Youth And Lifelong Learning Placement In Employment Or Education In post-secondary education, employment, or the military at participation? Excluded YES

54 Youth And Lifelong Learning Attainment Of Degree Or Certificate Education refers to secondary school, post-secondary school, adult education, or any organized program of study Includes youth in secondary school at exit Outcome can be attained during participation Diploma is any credential the State Education Agency accepts as equivalent to a high school diploma Of those enrolled in education at participation or any time during the program: Number attaining a diploma, GED or certificate by the end of the 3 rd quarter after exit Number of exiters Computing Outcomes

55 Youth And Lifelong Learning Attainment Of Degree Or Certificate A certificate is awarded in recognition of an individual’s attainment of measurable technical or occupational skills (emphasis added) necessary to gain employment or advance within an occupation. These technical or occupational skills are based on standards developed or endorsed by employers. Recognition of generic pre-employment and/or work readiness skills are not considered certificates. Defining “certificate”… Computing Outcomes Definition applies only to youth attainment of degree or certificate common measure

56 Definition of Certificate (cont.) Awarding Institutions include: – A State educational agency – Institution of higher education – Professional, industry or employer organization or a product manufacturer – Registered apprenticeship program – Public regulatory agency – Department of Veterans Affairs –Office of Job Corps –Indian Tribe Higher Education Institution

57 Definition of Certificate (cont.) Clarification of definition : –If a state educational system or employer organization granted certificate status to very specific standards for a work readiness program, this would be acceptable as a certificate under the measure –For example, the U.S. Chamber of Commerce’s recently announced work readiness credential would be allowable under the certificate measure

58 NO Enrolled in education at participation or any time during the program? Excluded NO YES Attained diploma, GED or certificate by the end of the 3 rd quarter after exit? Youth And Lifelong Learning Attainment Of Degree Or Certificate YES Numerator Denominator YES

59 Comparison to Current Measures This measure is similar to a combination of the YY Diploma Rate and OY Credential Rate, but with a tighter definition of certificate Narrower denominator than current credential measure: includes only those in education Does not have employment component as the current credential measure does

60 Youth And Lifelong Learning Literacy Or Numeracy Gains Excludes youth who are not basic skills deficient Excludes youth who are in-school Includes individuals with learning disabilities To be in the numerator, an individual must advance one or more Adult Basic Education (ABE) or English as a Second Language (ESL) functioning levels Number who completed a year of participation Of those out-of-school youth who are basic skills deficient: Number who increase one or more educational functioning levels Number who exit before completing a year of participation + Computing Outcomes

61 Educational skill gain measurement is consistent with the National Reporting System for Adult Education Tests cross-walked with ABE and ESL levels include: –Comprehensive Adult Student Assessment Instrument (CASAS) –Test of Adult Basic Education (TABE) –Adult Basic Learning Examination (ABLE) –Student Performance Levels for ESL –Basic English Skills Test (BEST) for ESL –WorkKeys (for the top 3 ABE levels) Youth And Lifelong Learning Literacy Or Numeracy Gains

62 Requires the use of the same standardized test for both pre and post assessments Individuals should be pre-tested within 60 days of the participation date Individuals with disabilities should be accommodated according to guidelines associated with the assessment tool and state law or policies Individuals should be post-tested by the end of one year of participation and compared to pre-test results obtained during the initial assessment Youth And Lifelong Learning Literacy Or Numeracy Gains Computing Outcomes

63 If an individual continues to be basic skills deficient after the first 12 months of participation, then s/he should continue to receive training in literacy and/or numeracy skills Participants should be post-tested and included in the measure at the completion of the 2 nd year Only included in measure a second time if they complete a full 2 nd year in program Youth And Lifelong Learning Literacy Or Numeracy Gains Computing Outcomes

64 Literacy/Numeracy-- Comparison to Current Measures Creates specific focus on basic skills rather than broad focus on work readiness, occupational skills, and basic skills –Does not mean work readiness and occupational skills no longer important (impact of those skills will be seen in placement rate) Standard for success much higher than with less-defined skill attainment rate

65 For Further Information… ETA’s Performance And Results Website ETA’s Performance And Results ADDRESS National Reporting System for Adult Education Website