1. ETA Data Validation July 2003

2. Overall ETA Data Validation Project Goals Develop a comprehensive, systematic data validation system to ensure data integrity across programs Increase uniformity in data definitions and data collection across similar programs Increase uniformity in data definitions and data collection across similar programs Strike the proper balance between data integrity and burden to achieve acceptable, sustainable level of error Coordinate closely with DOL Dept. of Inspector General on methods and approach

3. Validity and Verification Dept. of Labor Perspective Develop reputation for reliable and accurate program data Develop reputation for reliable and accurate program data Administration’s focus on management and accountability Administration’s focus on management and accountability Improve basis for incentives and sanctions Improve basis for incentives and sanctions Basis for continuous improvement Basis for continuous improvement

4. Programs Included Unemployment Insurance Benefits and Tax (UI) Unemployment Insurance Benefits and Tax (UI) Workforce Investment Act (WIA) Workforce Investment Act (WIA) Trade Adjustment Assistance (TAA and NAFTA-TAA) Trade Adjustment Assistance (TAA and NAFTA-TAA) Labor Exchange Labor Exchange Migrant and Seasonal Farm Worker Program (MSFW) Migrant and Seasonal Farm Worker Program (MSFW) Division of Indian and Native American Programs (DINAP) Division of Indian and Native American Programs (DINAP) Senior Community Service Employment (SCSEP) Senior Community Service Employment (SCSEP) Office of Apprenticeship, Training, Employment, and Labor Services (OATELS) Office of Apprenticeship, Training, Employment, and Labor Services (OATELS)

5. Stages of the Project 1. Reporting, performance and validation requirements analysis and specifications 2. Develop validation tools 3. Pilot validation methodology 4. Training 5. Technical assistance

6. 1.Requirements Analysis and Specifications Requirements analysis and specifications document the reporting and performance needs of each relevant ETA program Requirements analysis and specifications document the reporting and performance needs of each relevant ETA program Documentation is organized in the ETA Reporting and Performance Database Documentation is organized in the ETA Reporting and Performance Database Database defines each data element and reporting specification for each report and performance item Database defines each data element and reporting specification for each report and performance item

7. 2. Develop Validation Tools – Handbooks Handbooks contain reporting specs and validation instructions, including acceptable source documentation Handbooks contain reporting specs and validation instructions, including acceptable source documentation SCSEP awaiting final specs SCSEP awaiting final specs LX has no handbook, just software and user’s guide, no case-record level data validation at this time LX has no handbook, just software and user’s guide, no case-record level data validation at this time

8. 2. Develop Validation Tools – Software Software completed for LX, WIA, and TAA Software completed for LX, WIA, and TAA Software under development for MSFW and DINAP Software under development for MSFW and DINAP Distribution of handbooks and software via ETA websites Distribution of handbooks and software via ETA websites –LX: www.uses.doleta.gov/rptvalidation.asp –WIA and TAA: www.uses.doleta.gov/dv/

9. 3. Pilot – State Programs Pilot state programs – two formal state pilots Pilot state programs – two formal state pilots –Texas – WIA –Washington State – WIA, TAA, LX –Utah and West Virginia have been trained LX was implemented in August 2002 LX was implemented in August 2002 Other states are testing WIA Other states are testing WIA

10. 4. Training Regional training sessions are being held in the summer of 2003 for WIA, LX, TAA Regional training sessions are being held in the summer of 2003 for WIA, LX, TAA Other programs – determine training strategy individually Other programs – determine training strategy individually –Tie into national meetings –2-3 sessions per program

11. 5. Technical Assistance Phone and e-mail TA available Phone and e-mail TA available –Installing software –Building and loading extract files –Conducting report validation –Conducting data element validation –Contact information in software user’s guide and help menu of software TA e-mail addresses TA e-mail addresses –For WIA: WIATA@mathematica-mpr.com –For LX: ESTA@mathematica-mpr.com –For TAA: TAATA@mathematica-mpr.com

12. How Data Validation Systems Improve Data Quality Improve communication from ETA to programmers Improve communication from ETA to programmers Provide a blueprint or roadmap to understand reporting and performance measurement Provide a blueprint or roadmap to understand reporting and performance measurement Minimize burden of interpreting specifications Minimize burden of interpreting specifications Provide clear standards for assessing validity Provide clear standards for assessing validity Provide detailed diagnostic data for correcting problems Provide detailed diagnostic data for correcting problems

13. Report Validation Given the data that are stored, is the software generating the correct counts Given the data that are stored, is the software generating the correct counts Develop an audit trail to support the numerators and denominators for each performance outcome Develop an audit trail to support the numerators and denominators for each performance outcome Classifying participant records into performance outcome groups enables non- technical staff to validate and analyze program outcomes Classifying participant records into performance outcome groups enables non- technical staff to validate and analyze program outcomes

14. Data Element Validation Report will not be accurate if the data being used by the software are wrong Report will not be accurate if the data being used by the software are wrong Requires checking data elements against source documentation to verify compliance with federal definitions Requires checking data elements against source documentation to verify compliance with federal definitions Handbooks contain instructions and examples of acceptable source documents for each data element validated Handbooks contain instructions and examples of acceptable source documents for each data element validated –States identify state-specific source documentation to reflect the variability of state MIS systems and state/local documentation standards Self-reported elements such as race, gender, and ethnicity are not validated Self-reported elements such as race, gender, and ethnicity are not validated

16. Data Element Risks Low risk data elements Low risk data elements –Computer generated – wage records –Human input with:  Minimal judgement (e.g. dates)  Low performance impact High risk data elements High risk data elements –Human input with:  Considerable judgement (interpreting rules)  High performance impact – supplemental employment data

17. Software Selects Samples for Data Element Validation Sampled records are displayed on automated worksheets Sampled records are displayed on automated worksheets Participant records with positive outcomes not based on wage records are over-sampled Participant records with positive outcomes not based on wage records are over-sampled Software Software –Adjusts error rates based on weights –Produces a detailed data element validation report with error rates for each data element

18. Data Element Validation by Program For WIA, TAA, LX, MSFW, DINAP, and SCSEP software generates worksheets for sampled records For WIA, TAA, LX, MSFW, DINAP, and SCSEP software generates worksheets for sampled records For WIA and TAA cluster sampling used to reduce the number of offices to be visited For WIA and TAA cluster sampling used to reduce the number of offices to be visited For LX, no data validation against source documents — 25 cases are reviewed to ensure that file was built correctly For LX, no data validation against source documents — 25 cases are reviewed to ensure that file was built correctly

19. Benefits of Performance and Analysis Software Provides technical assistance to states Provides technical assistance to states Reduces burden on local offices and small states Reduces burden on local offices and small states Clear and easy analysis of outcomes Clear and easy analysis of outcomes –For example, impact of zero pre-program earnings Makes underlying performance data accessible to managers Makes underlying performance data accessible to managers Breaks out performance by many factors and checks for errors Breaks out performance by many factors and checks for errors

20. Software Allows for Flexible Data Analysis Software will report by user-selected time period (weekly, monthly, quarterly, annually) Software will report by user-selected time period (weekly, monthly, quarterly, annually) Users can also select reports by state or sub-state breakouts, including WIB, office, or case manager Users can also select reports by state or sub-state breakouts, including WIB, office, or case manager –Not multiple offices per participant unless state loads separate files –Software may be enhanced to allow multiple counts Users can sort participant records by any field within performance outcome groups — will have 3 tiered sort Users can sort participant records by any field within performance outcome groups — will have 3 tiered sort Users can also export participant groups for analysis, local feedback, or WRIS requests Users can also export participant groups for analysis, local feedback, or WRIS requests

21. Reporting of Validation Results Software produces Software produces –Report validation summary –Data element validation summary and analytical reports WIA and LX software creates files with the annual report validation values for upload to ETA WIA and LX software creates files with the annual report validation values for upload to ETA

22. Visual Basic Applications Software requires any Windows operating system Software requires any Windows operating system No other software required No other software required For large files, MS SQL Server is an option if the state has a license (for UI and LX only) For large files, MS SQL Server is an option if the state has a license (for UI and LX only) Front-end edit checks ensure proper format of records Front-end edit checks ensure proper format of records

23. Next Generation Reporting and Performance System In Fall 2004, states may use federal software to: In Fall 2004, states may use federal software to: –Generate reports –Perform and report on data validation –Edit and transmit individual participant records Software likely to be developed as part of new EIMS software development effort Software likely to be developed as part of new EIMS software development effort

24. Web-Based State Internal Audit Tool States want capability to perform data element validation at sub-state level States want capability to perform data element validation at sub-state level Proposed design: Proposed design: –Software would generate samples for any level (WIB, office) upon request from authenticated user (through web) –User can complete worksheets and generate reports on-line –One sample per WIB or office per imported file –Will be able to report multiple offices per participant

25. Benefits of Internal Audit Tool States and federal government are dependent upon data quality at the local level States and federal government are dependent upon data quality at the local level Increase the efficiency and precision of existing state monitoring efforts Increase the efficiency and precision of existing state monitoring efforts Potential cost savings for the system as a whole Potential cost savings for the system as a whole