Changing Perspectives on Workforce System Performance Data Validation Workforce Innovations San Antonio July, 2004

Purpose of Today’s Session Update states and grantees on what is new in data validation Update states and grantees on what is new in data validation Provide information on the findings to date from the first round of validation Provide information on the findings to date from the first round of validation Allow states and grantees to provide feedback to ETA on how to improve the DV process Allow states and grantees to provide feedback to ETA on how to improve the DV process

Role of Data Validation in ETA Performance Assessment Data validation is a key component in overall performance strategy Data validation is a key component in overall performance strategy Program funding is being directly tied to reliable performance outcomes (performance budget integration) Program funding is being directly tied to reliable performance outcomes (performance budget integration) Data validation required by OIG and now being reviewed by GAO Data validation required by OIG and now being reviewed by GAO Data validation is integrated into reporting Data validation is integrated into reporting Validation tools are evolving to meet state needs Validation tools are evolving to meet state needs

Programs Included in the Data Validation Effort Unemployment Insurance Benefits and Tax (UI) Unemployment Insurance Benefits and Tax (UI) Workforce Investment Act (WIA) Workforce Investment Act (WIA) Trade Adjustment Assistance (TAA and NAFTA-TAA) Trade Adjustment Assistance (TAA and NAFTA-TAA) Labor Exchange Labor Exchange National Farmworker Jobs Program (NFJP) National Farmworker Jobs Program (NFJP) Indian and Native American Programs (INA) Indian and Native American Programs (INA) Senior Community Service Employment (SCSEP) Senior Community Service Employment (SCSEP) Office of Apprenticeship, Training, Employment, and Labor Services (OATELS) Office of Apprenticeship, Training, Employment, and Labor Services (OATELS)

How Does Validation Work? Two separate processes are required to ensure that performance data is reliable Two separate processes are required to ensure that performance data is reliable –Report Validation –Data Element Validation ETA provides software to states and grantees that analyzes participant records ETA provides software to states and grantees that analyzes participant records

Report Validation Ensures that performance calculations are accurate Ensures that performance calculations are accurate DV software creates an audit trail for the numerator and denominator for each performance measure DV software creates an audit trail for the numerator and denominator for each performance measure Classifying participant records into performance outcome groups enables non- technical staff to validate and analyze program outcomes Classifying participant records into performance outcome groups enables non- technical staff to validate and analyze program outcomes

Data Element Validation Report will not be accurate if the data being used by the software are wrong Report will not be accurate if the data being used by the software are wrong Requires checking data elements against source documentation to verify compliance with federal definitions Requires checking data elements against source documentation to verify compliance with federal definitions Handbooks contain instructions and examples of acceptable source documents for each data element validated Handbooks contain instructions and examples of acceptable source documents for each data element validated –States identify state-specific source documentation to reflect the variability of state MIS systems and state/local documentation standards

Reporting of Validation Results Data validation software produces Data validation software produces –Report validation summary –Data element validation summary and analytical reports WIA and LX software creates files with the annual report validation values for upload to ETA WIA and LX software creates files with the annual report validation values for upload to ETA

Validation Efforts to Date Many states have shared their validation results Many states have shared their validation results First round of validation was a valuable learning experience for all First round of validation was a valuable learning experience for all ETA has not set standards for acceptable data quality ETA has not set standards for acceptable data quality Standards will be set for PY 2004 data validation Standards will be set for PY 2004 data validation

What is New for PY 2003 Validation New schedule for reporting and validation New schedule for reporting and validation New software New software New policies for collection and retention of source documentation New policies for collection and retention of source documentation

Schedule for Reporting of Validation Results WIA RV will be due October 1, 2004 when the annual report is due WIA RV will be due October 1, 2004 when the annual report is due WIA and TAA data element validation will be due February 1, 2005 WIA and TAA data element validation will be due February 1, 2005 LX report validation will be due November 15, 2004 with the report. LX report validation will be due November 15, 2004 with the report.

Data Validation for National Programs NFJP validation to begin in February NFJP validation to begin in February –Reports due in June –Pilot of process was conducted in spring –Software will be tested further in fall –Training session scheduled for November Data validation to be added to SCSEP in late 2005 Data validation to be added to SCSEP in late 2005 Indian and Native American validation will be incorporated into existing reporting software Indian and Native American validation will be incorporated into existing reporting software

Revised Software WIA Version 3.0 to be released in mid- August WIA Version 3.0 to be released in mid- August New versions TAA (version 1.3) and LX (version 1.8) validation software New versions TAA (version 1.3) and LX (version 1.8) validation software All will include automated upload of DV reports to ETA All will include automated upload of DV reports to ETA

WIA Software Changes Calculate performance for the new reporting periods Calculate performance for the new reporting periods Calculate Table O Calculate Table O More complete edit checks More complete edit checks Ability to filter source table and performance outcome groups to provide greater analytical flexibility Ability to filter source table and performance outcome groups to provide greater analytical flexibility Accept records for participants served only by NEGs Accept records for participants served only by NEGs

Software Upgrades for WIA Data Element Validation Revised Data Element Validation Worksheets to reflect reduction in elements Revised Data Element Validation Worksheets to reflect reduction in elements Improved ability to identify records that have not been validated Improved ability to identify records that have not been validated Ability to identify sampled records that have are missing, invalid, wrong SSN, or whose location is unknown. Ability to identify sampled records that have are missing, invalid, wrong SSN, or whose location is unknown. Ability to trace exported samples Ability to trace exported samples

States Experiences with Data Validation States had to determine staff to be responsible for data validation States had to determine staff to be responsible for data validation Communication of expectations and requirements to local areas Communication of expectations and requirements to local areas Mode of data element validation – onsite, centralized or both Mode of data element validation – onsite, centralized or both

State and Local Roles and Responsibilities States had varying experiences in identifying validation assignments States had varying experiences in identifying validation assignments –Some states had no problem –Some states took time to sort through roles of different units –Some states still have not clarified assignments (particularly for TAA) Organization of case files at local areas was often not standardized or adequate Organization of case files at local areas was often not standardized or adequate

Improving the Clarity of Source Documentation Requirements ETA has not had clear and specific policies for collection and retention of source documentation ETA has not had clear and specific policies for collection and retention of source documentation States need to provide clear guidance to local areas States need to provide clear guidance to local areas ETA will clarify requirements in change 1 to TEGL 3-03 ETA will clarify requirements in change 1 to TEGL 3-03 –Currently in clearance –To be issued in August

Streamlined Data Element Validation Requirements As a result of state feedback, ETA reviewed and reduced the number of elements to be validated As a result of state feedback, ETA reviewed and reduced the number of elements to be validated All elements directly related to performance or eligibility All elements directly related to performance or eligibility

Detail for reduction in elements Program # of Elements for PY02 # of Elements for PY03 Adult4126 Dislocated Worker 4430 Older Youth 4835 Younger Youth Trade3523 NFJP1714 Total285157

Various Methods for Data Element Validation Onsite validation is essential to preserve the integrity of the process Onsite validation is essential to preserve the integrity of the process Ideal for state staff to perform validation onsite Ideal for state staff to perform validation onsite –Promotes communication and mutual understanding In some cases, onsite validation is impractical In some cases, onsite validation is impractical –Distances are too great –Small number of records States can therefore pursue a combination of onsite and remote validation if necessary States can therefore pursue a combination of onsite and remote validation if necessary

Findings for WIA Report Validation States had problems in two areas States had problems in two areas –Problems with extract file imported into software –Problems with calculations

WIA Report Validation -- File Problems Extract file imported into the software is incorrect Extract file imported into the software is incorrect –Data in extract file does not match data in state’s data system –Inconsistent data File is different from the file used to calculate the report submitted to ETA File is different from the file used to calculate the report submitted to ETA –Missing records –Changed/Updated Data

WIA Report Validation – Calculation Problems States excluded older Youth in advanced training/post-secondary school from performance, even if the youth is employed. States excluded older Youth in advanced training/post-secondary school from performance, even if the youth is employed. Failure to distinguish pre-dislocation earnings from pre-registration earnings for dislocated workers Failure to distinguish pre-dislocation earnings from pre-registration earnings for dislocated workers Exclusion of records for earnings calculations due to 99, Exclusion of records for earnings calculations due to 99,

Data Element Validation Findings Significant number of errors – error rates exceeded 20% for some elements Significant number of errors – error rates exceeded 20% for some elements Many errors can be explained by lack of clarity in expectations for local source documentation Many errors can be explained by lack of clarity in expectations for local source documentation Problems with changing wage records and WRIS data Problems with changing wage records and WRIS data

WIA Data Element Validation Results for Adults >20%10-20%5-10%<5% Individual with Disability Date of First Training Service Date of Birth Employed 1 st qtr after Exit Veteran Status Date of First Intensive Service Employment Status at registration Wages second quarter prior to registration Other Reasons for Exit Source of Supplemental Data for 1 st and 3 rd Quarter Low Income Wages first qtr following exit quarter Wages third quarter prior to registration TANF Wage third quarter following exit Type of recognized certificate/credential Date of WIA Exit

WIA Data Element Validation – Results for Dislocated Workers >20%10-20%5-10%<5% Displaced Homemakers Individual with a disability Date of Birth Employed 1 st qtr after Exit Veteran Status Date of actual qualifying dislocation Wages 2 nd and 3 rd quarter prior to registration Date of First Training Service Date of WIA registration Wages 2nd and 3rd quarter prior to dislocation Source of Supplemental Data for 1 st and 3 rd Quarter Date of WIA Exit Wages 1 st, 2 nd, and 3rd quarter following exit quarter Type of recognized certificate/credential Employed in the 3 rd quarter after exit

WIA Data Element Validation – Results for Older Youth >20%10-20%5-10%<5% In post-secondary education or advanced training 3 rd quarter after exit Veteran Status Employment status at registration Date of birth Date of First Training Service Date of WIA Exit Wages 2 nd and 3 rd quarter prior to registration Type of recognized certificate/credential Employed in the 3 rd quarter after exit Wages 1 st, 2 nd, and 3rd quarter following exit quarter In post-secondary education/advanced training 1 st quarter after exit Source of supplemental 3 rd quarter after exit Source of supplemental 1 st quarter after exit Other reasons for exit

WIA Data Element Validation Results for Younger Youth >20%10-20%5-10%<5% Date of WIA Exit Education Status at Registration Date of birth Other reasons for exit Date goals attained Attained secondary sechool diploma Date attained diploma Youth Placement and Retention Information

TAA Data Element Validation – Results by the Numbers >20%10-20%5-10%<5% Most recent qualifying separation Date of registration Employed in the 1 st and 3 rd quarters after exit Date of Exit Earnings 1 st, 2 nd, and 3 rd quarters after exit Earnings 2 nd and 3 rd quarters prior to most recent qualifying separation

Continuing Challenges State wage record files are always changing State wage record files are always changing –One solution is to “freeze” the file to avoid changes –States should track changes in order to validate wages Confidentiality of WRIS data Confidentiality of WRIS data –WRIS has rules restricting access to information –Software allows states to suppress display of wage values

Future of Data Validation Standards for acceptable error rates to be established Standards for acceptable error rates to be established ETA is moving toward a consolidated reporting system ETA is moving toward a consolidated reporting system Data Validation will be integrated into the new reporting system Data Validation will be integrated into the new reporting system

For More Information Contact Information Contact Information Traci Di Martini   MPR Technical Assistance  William Borden –  Jonathan Ladinsky –    ols_datavalidation.cfm ols_datavalidation.cfm

We Need Your Feedback Tell us about your experiences with Data Validation Tell us about your experiences with Data Validation What did you learn that may help others What did you learn that may help others What improvements can be made What improvements can be made