1. PROGRAM REPORTING AND PERFORMANCE MEASUREMENT 1 General Session #1

2. Agenda Introduction and Background Program Reporting Requirements/Revisions Participation, Exit and Related Terminology Basic Measurement Rules/Concepts Calculating Outcomes on Adult Common Measures 2

3. Introduction and Background 3

4. Conference Organization and Expectations Lots of information to share! General sessions precede specific workshops Team approach Materials to be shared after all three events 4

5. Programs Covered Two agencies, same Department –VETS and ETA are both agencies within USDOL with similar missions and visions Funding –WIA, Wagner-Peyser and Trade Act monies fund positions, programs and systems whereas VETS monies primarily fund positions “VETS-funded programs” deliver services through Local Veterans’ Employment Representatives (LVERs) and the Disabled Veterans’ Outreach Program (DVOPs) WIA, W-P, VETS, Trade 5

6. Establishing Common Measures Effort led by Office of Management and Budget (OMB) as part of President’s Management Agenda –Linking performance to budget –Supporting effective programs –Comparing similar programs across agencies –www.whitehouse.gov/omb/budintegration Includes six federal Departments –Labor; Education; Health and Human Services; Veterans’ Affairs; Interior; HUD 6

7. USDOL Policy Common measures became effective for WIA, W-P and VETS on 7/1/05, and for Trade on 10/1/05 Final policy located in TEGL 17-05, dated 2/17/06 –Rescinds previous policy guidance TEGL 28-04, TEGL 7-99, TEGL 6-00 and 6-00 Change 1 –Applies to DOL-funded programs only –The basis for this conference! 7

8. The Common Measures Adult Measures –Entered Employment –Employment Retention –Average Earnings* Youth Measures –Placement in Employment or Education –Attainment of a Degree or Certificate –Literacy or Numeracy Gains *Average Earnings becomes effective PY 2006; for PY 2005, the measure is an Earnings Change 8

9. Common Measures: Benefits Focus on the core purposes of the workforce system; employment for adults and skill attainment for youth Break down barriers to integration resulting from different definitions, data and reports for each workforce program Resolve questions raised by GAO and other oversight agencies regarding consistency and reliability of data Reduce confusion among customers and stakeholders who want to know about results 9

10. Information Needs Common measures will be the core information for all programs but other management information will continue to be necessary –Management and oversight of programs will continue to require information relevant to each program –Additional information about participants, services and outcomes must supplement the story told by common measures 10

11. Reality Check PY 2005PY 2005 performance accountability requirements for WIA, Wagner-Peyser, Trade and VETS-funded programs include required performance measures, required reporting and recommended reporting PY 2006PY 2006 will be different –Reporting package finalized/effective for 3 years –Baselines generated in PY 2005 used in PY 2006 –Baseline year begins for one common measure –Earnings Change replaced by Average Earnings Challenge: Although certain accountability requirements are “phased in,” system adaptations should be on a faster track 11

12. (1) PY 2005 Accountability Accountable for specific Common Measures –Entered Employment Rate (WIA, W-P, VETS, Trade) –Employment Retention Rate (WIA, W-P, VETS, Trade) –Earnings Change (WIA, W-P, VETS, Trade) Accountable for specific WIA statutory measures –Adult and DW Employment & Credential –OY Entered Employment, Retention, Earnings, Credential –YY Retention, Skill Attainment, Diploma Attainment –Customer Satisfaction (Participants and Employers) 12

13. (2) PY 2005 Accountability Accountable for reporting on specific Common Measures –Youth Attainment of Degree/Certificate –Youth Placement in Employment & Education Reporting recommended for Youth Literacy/Numeracy –Required beginning PY 2006 WIAWIA performance accountability for PY 2005 still consists of the “17” measures –With some changed definitions 13

14. Program Reporting Requirements/Revisions 14

15. Overview of Basic Reporting Requirements Two Types of Federal Reporting Required Program Reports Reporting Schedules and Due Dates Rolling Four Quarter Reporting Methodology 15

16. Two Types of Federal Reporting Required ALL grantees of federal funds (whether formula or discretionary) are required to submit two types of federal reports within specified time frames 1.Financial (or fiscal)  How much are you spending, on what, and at what rate? 2.Program (or programmatic)  Who are you serving, what are they getting, and with what results? The focus of this conference is program reporting 16

17. Required Program Reports Wagner-Peyser –ETA 9002 (A through E) VETS –VETS 200 series reports (A through C) WIA –WIA Standardized Record Data or WIASRD (pronounced wizard) –Quarterly (9090) and Annual (9091) Reports Trade –Trade Adjustment Participant Report or TAPR (pronounced tapper) –ETA 563 17

18. Reporting Schedules & Due Dates Wagner-Peyser, VETS, WIA and Trade require quarterly fiscal and program reports –W-P, VETS and WIA programs operate on a program year basis (July through June); the first quarter of the program year is July—September –Trade programs operate on a program year basis, with the program year based on the federal fiscal year; the first quarter of the program year is October—December Quarterly reports are due "no later than 45 days after the end of the quarter" –If the 45 th day falls on a Saturday or Sunday, reports are due the Friday before 18

19. Reporting Methodology: Rolling Four-Quarter Wagner-Peyser, VETS and WIA (not Trade) report based on a rolling four-quarter methodology –Allows us to speak in terms of four quarters of data (or annual results) whether we’re talking about calendar years, program years, or federal or state fiscal years –For any report quarter and for any reporting element, we have one year’s worth of data; the most recent data available is reported Discussed further in individual workshops 19

20. Overview of Reporting Revisions Across Programs ETA’s General Policy on Program Reporting Unique Identifiers for Participants Minimum Data for Self-Service Customers Collection of Workforce Information Reporting Co-Enrollment Reporting on Veterans 20

21. Introduction Reporting requirements were changed/modified in order to incorporate the common measures –Reporting changes for One-Stop employment and workforce information services (W-P/VETS)* are most profound (e.g., from registration-based performance system to exit-based) integrationInformation added, revised or clarified across all or most program reports to support integration efforts Program-specific reporting changes discussed in individual workshops 21 *new moniker to replace ‘labor exchange’

22. ETA’s General Policy on Program Reporting In general, individuals who are eligible for a program and receive any service, regardless of where or how the service is accessed, should be reported in participant counts –This is NOT the same as saying that all individuals included in participant counts are also included in performance calculations –Example: WIA participants who receive only self-service and informational activities are included in participant counts but not included in performance calculations 22

23. Emphasis on Unique/Consistent Identifiers The same identifier must be used for the same individual for every period of participation in the state, every local area across the state and when participants receive services funded by Wagner-Peyser, VETS, WIA and/or Trade –Consistent with efforts to facilitate integration across partner programs Federal reports cannot contain SSNs – only alternate IDs or encrypted SSNs are accepted 23

24. Data Requirements for Self-Service Customers Minimum data needed from self-service participants specified 1.Social Security Number* Upon request of SSN, customer must be offered the opportunity to provide Equal Employment information; if no SSN provided, a pseudo-identifier must be used 2.Equal Opportunity Data (4 items) Voluntary and self-identified (Ethnicity, Race, Disability Status, DOB for age) 3.Employment status at participation 24 *Without an SSN, the individual cannot be included in performance calculations

25. Receipt of Workforce Information This is a new data element as of 7/1/05 that affects W-P, VETS and WIA-funded programs* –USDOL spends >$38 million on workforce information grants but there is little data on impact –To better understand the return on the federal investment, programs will collect/report data on provision of workforce information Note: Workforce Information can be accessed in many ways (e.g., through individual counseling or career guidance in a One-Stop Center, remotely via the Internet and/or in satellite agency sites) *Trade programs don’t require this information since their primary focus is training and it is assumed participants will be co-enrolled in other programs such as WIA where this information will be collected 25

26. Receipt of Workforce Information (cont’d) Definition: –Includes information on state and local labor market conditions, local employment dynamics info, industries, occupations and characteristics of the workforce, area business identified skills needs, employer wage and benefit trends, short and long term industry and occupational projections, worker supply and demand, and job vacancies survey results. It also includes local employment dynamics info such as workforce availability, business turnover rates, job creation, and job identification of high growth and high demand industries. This is the definition contained in the ET Handbook No. 406 (for W-P/VETS) and the WIA reporting instructions 26

27. Reporting Co-Enrollment Co-enrollment across WIA, Wagner-Peyser, and Trade programs must be reported If a WIA participant is also receiving Trade-funded services or Wagner-Peyser funded services, that must be captured in the WIA reporting system If a Wagner-Peyser participant is also receiving Trade- or WIA-funded services, that must be captured in the W-P reporting system If a Trade participant is also receiving Wagner-Peyser or WIA-funded services, that must be captured in the Trade reporting system This is consistent with [fill in the blank]_ 27

28. Reporting on “Eligible Veterans” Definitions and reporting specifications (e.g., coding values) were aligned with requirements in the Jobs for Veterans Act (JVA) as of 7/1/05 –Wagner-Peyser reports now reference “Veterans, Eligible Persons, and TSMs” instead of “Veterans and TSMs” (TSM=Transitioning Service Member) –WIA WIASRD and Trade TAPR coding values for veteran changed to include “eligible veteran” and “other eligible persons” Note: “Recently Separated Veteran” refers to 36 months for W-P and VETS, but 48 months for WIA (not collected in Trade) 28

29. Participation, Exit and Related Terminology 29

30. “Participant” An individual determined eligible to participate in the program who receives a service funded by the program in either a physical location (e.g., One- Stop Center) or remotely through electronic technologies –Three components: 1.Determined eligible to participate in the program 2.Receives a funded service 3.In either a physical location or through electronic technologies 30

31. Clarification of Participant Looking at the pieces (which impact programs differently!) 1.Individual determined eligible to participate Depends on program/funding; doesn’t apply in the case of W-P, which is based on universal access 2.Receives a service Not all services trigger participation; it’s important to understand the distinction between those that do and those that don’t (see handout) 3.In a physical location or remotely Adding remote access to the definition is a fundamental change in policy Many substantial services are remotely accessed; this needs to be captured 31

32. Other Terms Related to Participation Participation Date: Date of first program-funded service –It’s possible to record an earlier date if services were being provided by another partner at the time of participation Participation Quarter: Calendar quarter containing the participation date Participant Cohort: Group of individuals who share the same participation quarter 32

33. “Exiter” A participant who hasn’t received a program or partner-funded service for 90 consecutive days and no future services are scheduled –Three components 1.Hasn’t received a service 2.For 90 consecutive days 3.No future services scheduled Often referred to as a “soft exit approach” 33

34. Clarification of Exiter Looking at the pieces (which impact programs differently!) 1.The participant hasn’t received a service Could be program- or partner-funded depending on policies/procedures (e.g., tracking) 2.For 90 consecutive calendar days A gap in service can stop the 90-day clock if based on specific/allowable circumstances 3.No future services scheduled Some services can extend the exit date Does not include any follow-up services or circumstances where the participant voluntarily withdraws or drops out of the program 34

35. Gaps in Service Final common measures policy uses “gap in service” as opposed to the former “planned gap” Three allowable circumstances, where the condition exists for at least 90 days 1.Delay before beginning of training 2.Health/medical condition of participant/family member 3.Temporary move from the area that prevents participation Gaps in service should be <180 days (from date of last service), although a subsequent gap could be initiated as necessary All gaps must be documented and are subject to audit 35

36. Extending the Exit Date Services provided by partner programs can extend the point of exit –More services potentially means higher outcomes –Requires ability to track participants until they exit all services (program and partner) This actually highlights the essence of One-Stop, which is integrated services –No one program or entity can do it all. Only by leveraging limited resources of multiple partners and players can we meet the comprehensive needs of system customers 36

37. Other Terms Related to Exit Exit Date: Last date of service (program or partner-funded) –Once a participant completes all services and 90 days has passed, the exit date is applied retroactively to the last date of service Exit Quarter: Calendar quarter containing the exit date Exit Cohort: Group of individuals who exit during the same calendar quarter 37

38. Further Clarification of DATES Participation and Exit Dates are always dates of service –Participation Date reflects first funded service –Exit Date reflects last funded service Translation of “no more hard exit” –Not intended to take responsibility away from case managers; for WIA, case managers do not have to wait 90 days, for instance, to begin providing follow-up services –However, federal policy states that an exit cannot be officially recorded until that 90 days has elapsed –One suggestion: It’s possible to have a “case closure” code for case managers 38

39. When To Exit 39 Participants should be exited when the service plan or service strategy is complete –The service plan is a “living document,” with additions and changes possible –Co-enrollment in different funding streams, additional partner services and a valid gap in service can extend the exit date Participants should not be exited after a specific program intervention if additional services are needed

40. Illustration: Participation and Exit ParticipationNo Services Exit Date Participation Date Eligible and Receives Service End of 90- Day Period Last Service 40

41. Participation and Exit: A Systems Perspective 1 st Program 3 rd Program 2 nd Program Common Date of Participation Common Exit Date If a participant is served by multiple programs, it is possible to record the earliest date of service as the date of participation In an integrated service delivery system, a participant is exited when all program services are complete, enabling all individual programs to “share” in the final outcomes Program Participation 41

42. Basic Measurement Rules/Concepts 42

43. Overview Only participants are included in performance calculations Calculations are based on exiters* Specific conditions exclude participants from calculations Each measure is based on a specific cohort Specific data sources must demonstrate the outcomes 43 *The one exception is Youth Literacy/Numeracy

44. Only Participants Included in Performance Calculations From a measurement perspective, an individual doesn’t exist until they’re considered a participant –It’s critical for state and local staff to have the same understanding regarding services that trigger participation, which may be remotely accessed Once a participant exits, they are subject to the measures, with exception of Literacy/Numeracy –Translation: 5 of the 6 measures are “exit-based” Therefore, exit policies and procedures are of critical importance 44

45. Excluding Individuals From Performance Calculations Two basic circumstances in which participants are excluded –Statutory Exclusion applying to WIA (adults receiving only self-service and informational activities are not included in performance calculations) –One of six conditions exists either at exit or during the subsequent 3-quarter measurement period Note: The term “global exclusions” no longer part of common measures policy 45

46. Exclusions (cont’d) Allowable Exclusions –Institutionalized –Health/Medical or Family Care –Deceased –Reservists called to active duty (includes National Guard) –Relocated to a residential or non-residential program (applies to youth only) –Invalid or missing SSN 46

47. Each Measure is Based on a Specific Cohort Cohort = a group of similar individuals –Example: Staff members in your unit or department; youth in the U.S. over age of 18 with HS diplomas –In measurement, we reference many different cohorts, including participant cohorts, exit cohorts, reporting cohorts – all specify certain groups of individuals Translation: Not everyone is included in each measure; it’s based on the cohort Understanding specific cohorts is critical, particularly for front-line program staff 47

48. Specific Data Sources Must Demonstrate the Outcome Wage recordsWage records are the primary data source for any measure that includes employment (4 of 6) –Exceptions: Youth Attainment of Degree/Certificate and Youth Literacy/Numeracy Some data sources have the same “stature” as wage records (i.e., considered equivalent) –Include Wage Record Interchange System (WRIS); federal and military wage records accessed through the FEDES (Federal Employment Data Exchange System) 48

49. When There Aren't Wage Records Supplemental DataSupplemental Data can be used to demonstrate employment and retention, but not earnings* –Implications for three common measures Adult Entered Employment, Adult Employment Retention, and Youth Placement in Employment or Education –Allowable supplemental data sources include documented employer contact, employer and participant surveys, case management notes All supplemental data subject to audit *Some grantees will be able to use supplemental data as an interim means of reporting on earnings (e.g., NFJP) 49

50. Data Sources for Measures Not Related to Employment Pertains to all three youth measures  Youth Placement in Employment or Education Administrative records utilized for placement in education (wage records for employment portion)  Attainment of Degree/Certificate Administrative records utilized  Literacy and Numeracy Gains Assessment instrument (Further detail in Workshop #3) 50

51. Summary of Data Sources Common MeasureData Source (s) Adult Entered Employment Rate Wage records and supplemental data sources Adult Employment Retention Rate Wage records and supplemental data sources Adult Earnings Change or Average Earnings Wage records only (except for grantees without current wage record access) Youth Placement in Employment or Education Wage records and supplemental data for placement in employment or military; administrative records for placement in education or training Youth Attainment of Degree or Certificate Administrative records Literacy/Numeracy GainsAssessment instrument 51

52. Calculating Outcomes on ADULT Common Measures 52

53. Definition: For those not employed at participation, the percentage employed in the 1 st quarter after exit Calculation: Of those not employed at participation… Number employed in the 1 st quarter after exit Number who exit during the quarter 53

54. ENTERED EMPLOYMENT RATE Employed at Participation? Employed in 1 st quarter after exit? YES NO Excluded Numerator Denominator NO 54

55. Highlights of Adult EER Excludes adults employed at participation Employment at participation is based on information collected from the individual – not wage records Those who received a notice of termination or the employer has filed a WARN or similar notice are considered not employed and are included in calculations Transitioning Service Members (expected to retire within 24 months or separate within 12 months) are considered not employed and are included in calculations 55

56. Definition: For those employed in the 1 st quarter after exit, the percentage employed in the 2 nd and 3 rd quarter after exit Calculation: Of those employed in the 1 st quarter after exit… Number employed in the 2 nd and 3 rd quarter after exit Number who exit during the quarter 56

57. EMPLOYMENT RETENTION RATE Employed in 1 st quarter after exit? Employed in 2 nd and 3 rd quarter after exit? YES NO YES Excluded Numerator Denominator NO 57

58. Highlights of Adult ERR Includes all those employed in the 1 st quarter after exit regardless of employment status at participation (this is the measurement cohort) Positive outcome requires employment in two post-exit quarters (which does not have to be with same employer) 58

59. EARNINGS Measures There are two earnings measures, one which is effective now – and a revised measure, which will be effective next program year (7/1/06) –Beginning PY 2006, pre-program earnings will no longer be part of calculations PY 2005 – Earnings Change (comparison of pre- and post-program earnings) PY 2006 – Average Earnings (pre-program earnings not utilized) 59

60. (PY 2005) EARNINGS CHANGE Definition: For those employed in the 1 st quarter after exit, the average change between pre- and post-program earnings using the 2 nd and 3 rd quarters prior to and after participation Calculation: Of those employed in the 1 st quarter after exit… Number who exit during the quarter Total earnings in 2 nd and 3 rd quarters after exit Total earnings in 2 nd and 3 rd quarters prior to participation minus 60

61. (PY 2006) AVERAGE EARNINGS Definition: For those employed in the 1 st, 2 nd and 3 rd quarter after exit, average six month earnings in the 2 nd and 3 rd quarter after exit Calculation: Of those employed in the 1 st, 2 nd and 3 rd quarters after exit… Earnings from the 2 nd and 3 rd quarter after exit Number who exit during the quarter 61

62. AVERAGE EARNINGS Employed in 1 st quarter after exit? Employed in 2 nd and 3 rd quarter after exit? NO YES NO YES Excluded Wages from Q2 and Q3 post- exit included YES Were supplemental data sources used in either 1 st, 2 nd or 3 rd quarter after exit? NO 62

63. Highlights of Adult Average Earnings Includes those employed in 1 st, 2 nd and 3 rd quarter after exit, although only wages from the 2 nd and 3 rd quarters are utilized in the calculations Only allowable data source is wage records; no supplemental data* Those whose employment was determined based on supplemental data are excluded from the earnings measure *Some DOL-funded programs permit the use of supplemental data as an interim means of reporting 63

64. In Closing: Implementation is a Process The implementation of common measures and the reporting changes necessary to implement them represent such a challenge for states because they require in many cases nothing short of cultural change –Basic operations must be reviewed, rethought and perhaps redeployed A quote from Yogi Berra comes to mind: –The future ain’t what it used to be! 64

65. Remember this Website! www.doleta.gov/performance … your One-Stop portal for program performance and reporting across programs 65