1. DATA VALIDATION NORTH CAROLINA’S EXPERIENCE

2. NC Division of Employment & Training2 Program Year 2002 Data Validation We contracted with our former Finance Director (retired) to complete PY2002 data validation There were no electronic records provided to the performance or MIS staff after the project was completed No records were sent to DOL A sparse state report was provided and was disseminated to the Local Areas

3. NC Division of Employment & Training3 Program Year 2003 Data Validation Challenges: No raw data from prior year No state policies or procedures No practical experience No training (beyond DOL workshops) No staff

4. NC Division of Employment & Training4 Program Year 2003 Data Validation First Steps: Convene a Data Validation Committee to establish policies, procedures and training materials (August 2004) Committee Membership: DET Staff MIS Super Users Case Managers

5. NC Division of Employment & Training5 Data Validation Committee DET Staff MIS Director, Planners (2), Field Staff (Monitor), Technical Writer, Performance Manager MIS Super Users (7) MIS Super Users are MIS and performance specialists in each Local Area Case Managers (6) For adult, dislocated worker and youth programs

6. NC Division of Employment & Training6 MIS Super Users MIS Super Users are specialists in each Local Area responsible for: Attaining a high-level proficiency with the WorkforcePlus™ system and functioning as the Local Area’s expert user of the participant MIS Attending and actively participating in all MIS User’s Group meetings Training Local Area end users in the proper use of WorkforcePlus™ Providing ongoing technical assistance and remedial training on WorkforcePlus™ issues

7. NC Division of Employment & Training7 MIS Super Users Providing feedback to DET concerning MIS issues Conducting continual review of Local Area data, monitoring the quality of data entry and the proficiency of users and promptly addressing identified problem areas Understanding the relationship between the Local Area’s performance goals, program design, and program outcomes and ensuring the proper recording of program outcomes to maximize performance levels Attaining proficiency in extracting performance and management reports from WorkforcePlus™

8. NC Division of Employment & Training8 Data Validation Committee First Steps: Train DV Committee members Agree on the WorkforcePlus™ source documentation that will be acceptable for NC data validation and amend DOL source list to be NC specific Determine what state policies and procedures need to be put in place specific to data validation

9. NC Division of Employment & Training9 Data Validation Committee Source Documentation Developed a revised DV source document chart, deleting references in DOL’s chart that were not applicable to NC and adding specific WorkforcePlus™ (MIS) references for data entry screens and printouts that could be used for DV documentation A WorkforcePlus™ ‘definitions and instructions’ document that had been drafted by our Super Users group was adopted as state policy

10. NC Division of Employment & Training10 Data Validation Committee State Policies and Procedures Three Local Area Issuances were developed and released in December 2004 Entering Activities and Dates for First WIA Activity Clarified service requirements for initial WIA registration in NC WIA Application Time Limit WIA Data Element Validation and Participant File Structure Formalized a case file structure and content requirements across all Local Areas

11. NC Division of Employment & Training11 Data Validation Policies Entering Activities and Dates for First WIA Activity (Local Area Issuance 2004-15) Established specific employment activities as the initial service to be provided for every WIA participant Provides statewide consistency in setting the WIA registration date The “start date” must equal the “creation date” in WorkforcePlus™ for the initial activities For Adults and Dislocated Workers, the initial enrollment must be “case management” For Youth, the initial enrollment must be “comprehensive guidance and counseling”

12. NC Division of Employment & Training12 Data Validation Policies WIA Application Time Limit (Local Area Issuance 2004-15) Reinforces that individuals must be eligible at the time of enrollment into WIA, not just at application Resolves concerns we had regarding a lengthy time lapse between eligibility determination and enrollment into the WIA program. Requires a new application to be completed if more than 60 days have lapsed since the application date/eligibility date

13. NC Division of Employment & Training13 Data Validation Policies WIA Data Element Validation and Participant File Structure (Local Areas Issuance 2004-17) Intended to improve compliance with data validation requirements Requires Local Areas to develop a formal file structure policy (or amend an existing file structure policy) for all participant files Requires each file to have a file checklist at the front of the file indicating where supporting documentation can be found in the file Requires each DV data element to be labeled within the file

14. NC Division of Employment & Training14 Data Validation Policies File Structure (continued) Each DV data element has a ‘field code’ consisting of a cohort, category, and DOL’s reference number Cohorts: A (adult), DW (dislocated worker), OY (older youth) or YY (younger youth) Category: E – Eligibility/Intake/Application A – Employment Activities XP – Exit and Post-Program Activities DOL Reference Number: corresponds to the elements’ reference numbers in the DOL WIA Data Validation Handbook

15. NC Division of Employment & Training15 Data Validation Policies File Structure (continued) Field codes must be placed by each data elements in the file checklist(s) and the field codes added to the source documentation in the files The policy also requires that certain documents/screen prints also be placed in each file: Signed and dated application 1 st activity enrollment screen Case profiles Case/activity notes Supplemental data screens (where applicable) Leave Program screen Employment Plan screen

16. NC Division of Employment & Training16 Data Validation Policies File Structure (continued) All files for participants active in WIA as of July 1, 2004 are required to be in the file structure format and contain all source documentation for data validation Some Local Areas had file structure ‘parties’ where the case managers met and assembled participants’ files in the new structure Feedback from the case managers has been overwhelmingly positive Most Local Areas use sectioned file folders, although it is not required by the state

17. NC Division of Employment & Training17 Data Validation Committee Training Developed a data validation training outline and materials Created two training manuals For local case managers For state data validation staff Initial DV training for MIS Super Users and select case managers was held over two days in February 2005 Super Users in each Local Area were responsible for training their case managers Quarterly MIS Users Groups meetings provide a forum for continued dialogue between the state and Local Area staff

18. NC Division of Employment & Training18 Data Validation Committee Additional Training Case notes may be used in North Carolina to validate select data elements During Program Year 2003 data validation, it was found that some of the case notes in the files were marginal, at best An RFP was released to solicit bids for the provision of training for data validation specific “Writing Effective Case Notes” We provided 16 training sessions and trained 643 case managers at the Workforce Development Training center Local Areas also arranged for additional “Writing Effective Case Notes” training

19. NC Division of Employment & Training19 Data Validation Committee Additional Training, continued There are plans to offer ongoing data validation training to end users and case managers (after the revised reporting requirements are released by DOL) Data validation training will probably be combined with common measures training One day sessions with half of the day devoted to common measures and the other half devoted to data validation Case Managers tend to want training specific to the individuals they serve (i.e., youth separate from adult and dislocated workers)

20. NC Division of Employment & Training20 Staffing DET has used 4 time-limited/temporary staff to complete data validation for the last 2 years Pay is between $11.18 - $16.62 per hour No fringe benefits Receive reimbursement for mileage, meals and lodging There are two data validation teams – East and West Both teams go as a group to Local Areas with more than 100 files in the sample to minimize disruption to the Local Area staff

21. NC Division of Employment & Training21 Staff Training The DV staff receive a week of training on WIA programs, the NC participant database, and federal and state DV policies and procedures 25 participant files are ‘validated’ in house to allow staff to become familiar with the information they will be seeing in participant file folders and to practice validating the elements Once the files are validated, additional instruction and clarification are provided where needed

22. NC Division of Employment & Training22 Staff Training All validators and the manager go to the first Local Area A select group of files from each cohort at the Local Area (adult, dislocated worker, older youth, and younger youth) is reviewed by all 5 individuals DV results for all of these select files are compared and discussed The file(s) are re-reviewed by the group to determine if a questioned data element is supported by source documentation

23. NC Division of Employment & Training23 Actual Data Validation North Carolina has an official sample and an unofficial sample Every Local Area has a data validation visit The data validation schedule is arranged by the manager The validators are required to call Local Area contacts the week prior to the visit to confirm and to inform them when they will arrive (day and time) Validators hold entrance and exit conferences with the Local Area staff Validators complete hard copy worksheets at the Local Area. For every data element that is failed, the validator has to document why the element was failed

24. NC Division of Employment & Training24 Actual Data Validation Upon return to the state office : Each validator writes an individual report about his/her general findings at the Local Area and summarizes recommendations for the Local Area Validators enter DV findings into the DOL software Validators create a comment sheet for each Local Area containing the individuals’ names, data elements failed and why the elements failed (by cohort) State staff compiles reports for each Local Area Numerical report Narrative report (including comments on all failed data elements

25. NC Division of Employment & Training25 Actual Data Validation After the completion of all data validation for the state Individual reports are completed for each Local Area DOL “WIA Data Element Validation Summary and Analytical Report” DET Report Narrative report (including comments on all failed data elements

26. NC Division of Employment & Training26 Example 1 of Local Area Report LAXXX - Dislocated Worker (DOL Report) Wrong SSN0Invalid Records0 Missing Records0Unable to Locate Records0 # of Errors Overall Error Rate Reported Data Error Rate 02 Date of Birth1.002.30% 03 Disability0.000.00% 04 Veteran2.004.60%100.00% 06 UC Claimant5.0010.65%17.46% 08 TANF0.000.00% 09 Cash Assistance0.000.00% 10 Displaced Homemaker0.000.00% 11 Dislocation Date3.006.90%13.32% 18 WIA IB Registration Date5.0011.50% 19 WIA Exit Date35.0086.20% 21 Intensive Service Date5.0011.50% 22 Training Service Date4.009.20%14.02%

27. NC Division of Employment & Training27 Example 2 of Local Area Report LAXXX - Dislocated Worker (DET Report) Wrong SSN0Invalid Records 0 Missing Records0Unable to Locate Records 0 # Elements Selected # Pass # Fail % Pass % Fail 02 Date of Birth4140197.56%2.44% 03 Disability0000.00% 04 Veteran2020.00%100.00% 06 UC Claimant2520580.00%20.00% 08 TANF0000.00% 09 Cash Assistance0000.00% 10 Displaced Homemaker0000.00% 11 Dislocation Date2118385.71%14.29% 18 WIA IB Registration Date4136587.80%12.20% 19 WIA Exit Date4163514.63%85.37% 21 Intensive Service Date4136587.80%12.20% 22 Training Service Date2723485.19%14.81%

28. NC Division of Employment & Training28 Reports DOL’s DV report may tell them what they want to know, but it is confusing to the Local Areas How many of you know (without looking) what ‘Overall Error Rate’ and ‘Reported Data Error Rate’ mean? How many of your Local Areas’ staff understand the meaning of the error rates? The DET DV report puts information in perspective for the Local Area: how many elements were selected for validation and how many passed or failed?

29. NC Division of Employment & Training29 What Have We Learned? Local Areas want training, structure and guidance For the most part, case managers like the file structure requirements (once they got their files in order with all requisite information) Program Year 2004 data looks better than prior years’ data We anticipate that as the exit date range for DV catches up with our DV policies, our data validation performance will improve significantly.

30. NC Division of Employment & Training30 QUESTIONS? NC VALIDATION EXPERIENCE

31. NC Division of Employment & Training31 Contacts and References Martha Cranford NC Division of Employment & Training 313 Chapanoke Road, Suite 120 4316 Mail Service Center Raleigh, NC 27699-4316 Phone:919-329-5290 Fax:919-661-4770 mcranford@nccommerce.com NC DET State Policies www.ncdet.com Case Notes Training Mary Ann Lawrence (Center for Workforce Learning) MALaw528@aol.com www.mylearningchain.com