Discretionary Grants: Data Collection, Processing & Reporting 1 Back-to-Basics - Discretionary Grants Data Collection, Processing & Reporting Employment and Training Administration U.S. Department of Labor Frances Perkins Building 200 Constitution Avenue, NW Washington, DC

Discretionary Grants: Data Collection, Processing & Reporting 2 Data Collection Data Processing Reporting Requirements Session Objectives Note: The discretionary grantees that are the focus of this session are those subject to USDOL’s common measures (e.g., High-Growth and Community-Based Job Training Grantees)

Discretionary Grants: Data Collection, Processing & Reporting 3 Reports and Information Data Collection Data Processing “What matters in the end is Completion. Performance. Results. Not just making promises, but making good on promises. - President’s Management Agenda - Performance Management See Handouts 1 and 2

Discretionary Grants: Data Collection, Processing & Reporting 4 29 CFR Parts 95 & CFR Part 37 Public Law Grant Agreement Federal Requirements Data Collection & Reporting Requirements

Discretionary Grants: Data Collection, Processing & Reporting 5 Data Collection—Introduction  Process used to gather and collect information for the management of the project  To develop a tracking mechanism, must address four fundamental questions:  What information needs to be collected and in what format?  Where does the grantee obtain the information?  When does the information need to be collected?  How does the grantee know the information is accurate/valid?

Discretionary Grants: Data Collection, Processing & Reporting 6 Data Collection—What Information To Collect? What information is needed?  Start with grant proposal and grant award documents  Review goals, objectives, & expected outcomes for your grant  Review Work Plan, Service Flow Chart, Performance Measures, Project Deliverables, etc.  Involve internal staff and external partners in the process of identifying data collection elements  They know what format it comes in and when the information is collected

Discretionary Grants: Data Collection, Processing & Reporting 7 Case Management Cycle Application/ Intake/Eligibility Data Collection— Service Flow Service Delivery Job Development Exit & Follow-Up Service Planning Assessment See Handout 3

Discretionary Grants: Data Collection, Processing & Reporting 8 Data Collection— Participant Characteristics Types of individual characteristic information to collect includes:  Participant contact and emergency contact information (e.g., name, address, telephone)  Information to assess a person’s eligibility for the project (tailored to each grantee based on definition/information from the proposal)  EEO information (e.g., age, sex, ethnicity, race, and disability status)  Veteran’s characteristics and status Continued

Discretionary Grants: Data Collection, Processing & Reporting 9 Data Collection— Participant Characteristics Continued from previous slide  Education information (e.g., highest school grade completed)  Employment status at enrollment and past/current employment information  Information about supportive service needs and additional reportable characteristics (e.g., offender, runaway, low income, single parent, etc.)

Discretionary Grants: Data Collection, Processing & Reporting 10 Data Collection— Participant Services Collect information about the services provided to individual participants  Types of services in which the participant is enrolled  Some examples include career guidance services, job search assistance activities, supportive services, customized training, on-the-job training, apprenticeship training, occupational skills training, Adult Basic Education/ English Second Language training  Participation or service enrollment dates  Service completion dates  Training provider information

Discretionary Grants: Data Collection, Processing & Reporting 11 Data Collection— Participant Outcomes  Common Measures Initiative  Why use common measures?  Ability to describe in a similar manner the core purposes of the workforce system—how many people found jobs? Did they stay employed? What did they earn?  Enhance ability for different programs to collaborate by reducing multiple reporting requirements  Ability to analyze program outcomes and develop effective program strategies  Ability to compare results of similar government programs

Discretionary Grants: Data Collection, Processing & Reporting 12 DOL Common Measures Policy  Common measures is required for ETA formula- funded grants and some discretionary grants  High growth and community-based grantees are subject to the common measures policy  Training and Employment Guidance Letter (TEGL) No , dated February 17, 2006 details DOL policy for common measures See TEGL 17-05

Discretionary Grants: Data Collection, Processing & Reporting 13 What Common Measures Apply to Your Grant?  Adult Measures  Entered Employment  Employment Retention  Average Earnings  Youth Measures  Placement in Employment or Education  Attainment of a Degree or Certificate  Literacy or Numeracy Gain

Discretionary Grants: Data Collection, Processing & Reporting 14 Required Definitions— Participant  Participant = An individual determined eligible to participate in the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies  You need to determine what services trigger participation in your grant  You may need to capture information about services provided remotely

Discretionary Grants: Data Collection, Processing & Reporting 15 Required Definitions—Other Terms Related to Participation  Participation Date: Date of first program-funded service  It’s possible to record an earlier date if services were being provided by another partner at the time of participation  Discretionary grantees, however, cannot report a participation date that begins before the actual period of performance in the grant award document  Participation Quarter: Calendar quarter containing the participation date (e.g., If 1 st service is 4/22/07 then participation quarter is 2 nd CY quarter)  Participant Cohort: Group of individuals who share the same participation quarter

Discretionary Grants: Data Collection, Processing & Reporting 16 Required Definitions—Exit Exit: A participant who hasn’t received a program or partner- funded service for 90 consecutive days and no future services are scheduled has exited the program 1. Hasn’t received a service 2. No service received for 90 consecutive days 3. No future services scheduled (does not include any follow-up services or circumstances where the participant voluntarily withdraws or drops out of the grant)  For example, if a participant completes training on 2/13/2007 and planned to take one additional specialty course but then failed to do so, the exit date would be retroactive to 2/13/2007. Discretionary grantees cannot report an exit date that extends beyond the actual period of performance in the grant award document

Discretionary Grants: Data Collection, Processing & Reporting 17 Required Definitions— Other Terms Related to Exit  Exit Quarter: Calendar quarter containing the exit date  For example, the exit quarter for individuals exiting on 2/13/2007 would be Q1 of 2007  Exit Cohort: Group of individuals who exit during the same calendar quarter.  For example, all participants that exit between 1/01/07 and 3/31/07 are in the same exit cohort 17

Discretionary Grants: Data Collection, Processing & Reporting 18 Last Service Date Triggers Exit Date Date of 1 st Service Triggers Participation Date Gap in Services Allowed (see TEGL Page 22) No Services Received for 90 Consecutive Days Follow-up Discussed in TEGL (see TEGL Pages 20-21) Service Delivery Eligible & Receives Grant Services Participant Receiving Grant Services Last Service Received & No Future Services Scheduled Participant Receives Follow-up Services

Discretionary Grants: Data Collection, Processing & Reporting 19 Basic Common Measurement — Rules and Concepts  Only participants are included in performance calculations  Specific conditions exclude participants from calculations, such as being employed on the date of participation means excluded from the entered employment measure  Global exclusions if participant becomes (1) institutionalized, (2) invalid or missing SSN, (3) terminates due to health/medical or family care, (4) dies, (5) is a reservist called to active duty, and (6) relocates to a residential or non-residential program (applies to youth).  Calculations are based on exiters  Exception is Youth Literacy/Numeracy because based on one year from participation date  Each measure is based on a specific cohort  For example, a person who was unemployed on their participation date and exits on 4/28/07 would be in the entered employment rate cohort of Quarter 3 of 2007 (July 1-September 30, 2007)  Specific data sources must demonstrate the outcomes

Discretionary Grants: Data Collection, Processing & Reporting 20 Exit Cohorts Oct 1 - Dec 31 Jan 1 – Mar 31 Apr 1 – Jun 30 Jul 1 – Sep 30 Jan 1 – Mar 31 Apr 1 – Jun 30 Jul 1 - Sep 30 Oct 1 - Dec 31 Apr 1 – Jun 30 Jul 1 – Sep 30 Oct 1 – Dec 31 Jan 1 – Mar 31 Jul 1 – Sep 30 Oct 1 - Dec 31 Jan 1 - Mar 31 Apr 1 – June 30 ExitParticipant Exit Quarter Q1 1 st Quarter After Exit Quarter Q2 2 nd Quarter After Exit Quarter Q3 3 rd Quarter After Exit Quarter

Discretionary Grants: Data Collection, Processing & Reporting 21  Pay stubs  Progress Reports  Surveys  Self-attestation forms  Copy of Diploma  Training Certificates  Interviews  Public Agency Records  Student ID  Social Security Card  Driver’s License/ID Card  Hospital Records  Intake/Eligibility Forms  Attendance Sheets  Sign-In Sheets  School Records  Activity Forms  Assessment Results Data Collection— Where to Get the Information? Source Documentation

Discretionary Grants: Data Collection, Processing & Reporting 22 Wage Records—Data Source for Outcomes Measures  Discretionary grantees that are State and local workforce investment boards or one-stop providers will most likely have access to wage record data  Employers are required to send confidential quarterly reports of wage earners and their wages for the respective calendar quarter  The majority of discretionary grantees, however, will not have access to Unemployment Insurance (UI) wage records  The following websites provide fee-based access to employment verification and wage information, but not all employers provide information to these sources: (1) (2) and (3) See Handout 6

Discretionary Grants: Data Collection, Processing & Reporting 23 Wage Records—Data Source for Outcomes Measures (cont’d)  Wage records are the preferred source of information for calculating common measures  The proposed reporting package for HG/CB job training grantees would require grantees to submit quarterly participant records for each exiter to ETA containing 4 data elements  ETA would then use these records to obtain wage record data and calculate the common measures for grantees  This is proposed! See Handout 6

Discretionary Grants: Data Collection, Processing & Reporting 24 Supplemental Data Sources  Most discretionary grantees will need to use supplemental data sources to verify employment and wages  Copy of a participant’s pay stub  Employer contact—telephone or written verification  Participant contact—telephone, participant surveys or written self-attestation form  Case management notes  All supplemental data sources must be documented in participant case files and are subject to audit See Handout 6

Discretionary Grants: Data Collection, Processing & Reporting 25 Data Sources for Measures Not Related to Employment Pertains to all three youth measures  Youth Placement in Employment or Education  Administrative records utilized for placement in education (wage records or supplemental data for employment portion)  Attainment of Degree/Certificate  Administrative records utilized  Literacy and Numeracy Gains  Assessment instrument 25

Discretionary Grants: Data Collection, Processing & Reporting 26 Entered Employment Of those who are not employed at the date of participation: Number of adult participants who are employed in the 1 st quarter after the exit quarter Number of adult participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting 27 Entered Employment Employed at Participation Date? Employed in 1 st quarter after exit? Excluded from Measure Numerator Denominator YES NO YES

Discretionary Grants: Data Collection, Processing & Reporting 28 Employment Retention Of those who are employed in the first quarter after the exit quarter: Number of adult participants who are employed in both the second and third quarters after the exit quarter Number of adult participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting 29 Employment Retention Employed in 1 st quarter after exit quarter? Employed in 2 nd and 3 rd quarters after exit quarter? Excluded from Measure Numerator Denominator NO YES NO YES

Discretionary Grants: Data Collection, Processing & Reporting 30 Earnings Measure Of those adult participants who are employed in the first, second and third quarters after the exit quarter: Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter Number of adult participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting 31 Average Earnings Employed in 1 st quarter after exit quarter? Employed in 2 nd and 3 rd quarters after exit quarter? Excluded from Measure Numerator = Wages from quarters 2 & 3, post-exit included Denominator = Number of adult participants who exit during the quarter No Yes No Yes

Discretionary Grants: Data Collection, Processing & Reporting 32 Youth Placement in Employment or Education Of those who are not in post-secondary education or employment (including the military) at the date of participation: Number of youth participants who are employed (including military) or enrolled in post-secondary education and/or advanced training/occupational skills training in the first quarter after the exit quarter Number of youth participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting 33 Youth Attainment of a Degree or Certificate Of those who are enrolled in education--at the date of participation or at any point during the program: Number of youth participants who attain a diploma, GED, or certificate by the end of the third quarter after the exit quarter Number of youth participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting 34 Youth Literacy and Numeracy Gains Of those out-of-school youth who are basic skills deficient: Number of youth participants who increase one or more educational functioning levels Number of youth participants who have completed one year of participation in the youth program (one year from date of first program service) plus the number of youth participants who exit before completing a year in the youth program

Discretionary Grants: Data Collection, Processing & Reporting 35 Data Collection— Is That It?  While you are required to track information to report on the common measures, other management information continues to be necessary  Collect information relevant to the day-to-day operations of your project (e.g., information about enrollments and counts on various service dimensions)  May collect information to help case managers, job developers, managers, etc. make programmatic decisions

Discretionary Grants: Data Collection, Processing & Reporting 36 Data Collection—When to Collect the Information? Important to develop an information management plan addressing:  How will data be collected?  Information collected verbally, with paper documents or electronically  Who is responsible for the data collection?  Assign responsibility for data collection to specific staff to ensure that expectations are clearly understood  In what format will data be collected?  MS Excel spreadsheet, MS Access, Proprietary system  When will data be collected?  Incorporate deadlines & establish timeframes for all collection tasks

Discretionary Grants: Data Collection, Processing & Reporting 37 Data Collection— Is the Information Reliable?  Ensure data elements are well defined and understood by project staff—this includes contractors  Train all staff on definitions and processing requirements  Note that original source of information more reliable than secondary information  Ensure duplication of effort does not occur in the collection of data  Costly  Lack of Consistency

Discretionary Grants: Data Collection, Processing & Reporting 38 Data Processing—Introduction  Process used to input data and information into a MIS, find and correct errors in the data, and compile and aggregate the information into a user-friendly format  Grantees must address these fundamental questions:  Who has responsibility for data entry, compilation and processing?  How should they validate the data and ensure its integrity?  What management information system will be used to maintain the data?

Discretionary Grants: Data Collection, Processing & Reporting 39 Data Processing— Who Has Responsibility?  The grantee is responsible for ensuring that a system is in place to track participant characteristics, services and outcomes  It is highly recommended that grantees maintain access to data processing and reporting at all times  You need to know what is going on with your grant!  The grantee may contract out for services, but should provide input on how the data is maintained and gathered

Discretionary Grants: Data Collection, Processing & Reporting 40 Data Processing— Who Has Responsibility?  All staff, including sub-recipients staff, need to clearly understand their role in creating, collecting, maintaining, and reporting program data  Understand interrelationship of staff member duties  Understand consequences of missing deadlines, not sharing information, etc.  Reduce errors by training staff and making sure they meet specified qualifications

Discretionary Grants: Data Collection, Processing & Reporting 41 Data Processing— Is the Data Reliable?  Data needs to be reliable…accurate, complete and consistent  Factors affecting data reliability:  Lack of data collection and data processing policies and procedures  Inaccurate and incomplete data  Insufficient staff training  Differences in definitions  Insufficient system controls

Discretionary Grants: Data Collection, Processing & Reporting 42 Data Processing—Is the Data Reliable?  Data errors can occur at any point in record keeping. To help maintain data quality, consider the following:  Assign ranges of possible values to each data element  A person’s age could range from , anything outside the range would be flagged by the system  Check consistency across data elements  Same identification number but different race or sex  Look for missing data  Conduct periodic reviews of information keyed into MIS compared with original sources (i.e., sample of case files and MIS cases)  Remember data must be auditable—must be able to track back to original source

Discretionary Grants: Data Collection, Processing & Reporting 43 Data Processing—What System Should We Use?  How sophisticated or elaborate does the grantee’s MIS need to be?  It varies, but high-performing organizations have an MIS that produces information/reports to assist staff in addressing issues and improving performance  Examples include MS Access, MS Excel, or a proprietary system such as Client Tracking System  Contract with local workforce investment areas to process and aggregate data  What must grantee MIS be able to do?  At a minimum, capture all required data elements, perform any necessary calculations and report information to the grantee and its partners  In some cases, ETA may provide the required data collection instruments and a system for collecting and reporting the information See Handout 5

Discretionary Grants: Data Collection, Processing & Reporting 44 Data Processing—What System Should We Use?  Key principles of an effective MIS:  Enter information only once  Enter information in a timely manner  Enter information at the most cost-effective level of the organization (e.g., line staff versus technical staff)  Information available on demand  Access available to all levels in the organization, as needed

Discretionary Grants: Data Collection, Processing & Reporting 45 Reports and Information  Ability of grantee to use data processing and processing techniques to develop reports  Need for an information system that can give you reports that tell a story about your grant  What other data help tell your story?  What about customer satisfaction data?  Don’t collect data simply to ‘feed the Federal data machine,’ but to inform program design and management

Discretionary Grants: Data Collection, Processing & Reporting 46 Reports and Information— Federal Requirements  Reporting requirements based on Uniform Administrative Requirements in 29 CFR Parts and  Comparison of actual accomplishments with goals and objectives for the project  Reasons why established goals were not met and corrective action being taken  Other pertinent information, including technical assistance needs, best practices or any promising approaches  In some cases, ETA has a required report format for your use with established due dates

Discretionary Grants: Data Collection, Processing & Reporting 47 Reports and Information— Common Measures  Common measures are based on specific cohorts  Cohorts based on calendar quarters  January 1 to March 31, 2007 = CY 2007 Q1  April 1 to June 30, 2007 = CY 2007 Q2  July 1 to September 30, 2007 = CY 2007 Q3  October 1 to December 31, 2007 = CY 2007 Q4  If using wages to calculate performance on employment, earnings, and retention there will be a lag in when the wage information is available

Discretionary Grants: Data Collection, Processing & Reporting 48 Reports and Information— Table of Reporting Periods Reporting Time Periods Oct 1 - Dec 31 Jan 1 – Mar 31 Jul 1 - Sep 30 Apr 1 – Jun 30 Oct 1 – Dec 31 Jul 1 – Sep 30 Jan 1 - Mar 31 ExitParticipant Exit Quarter Q1 1 st Quarter After Exit Quarter Q2 2 nd Quarter After Exit Quarter Q3 3 rd Quarter After Exit Quarter

Discretionary Grants: Data Collection, Processing & Reporting 49 Reports and Information— Is the Information Reliable?  Employ mechanism(s) to ensure reports generated and outcomes reported are calculated correctly  Useful to have a data entry/MIS handbook  Reference tool for system users that details data collection protocols, definitions of terms, coding assignments, acceptable data sources, operational rules for data entry, available edit checks and report operations

Discretionary Grants: Data Collection, Processing & Reporting 50 Data Collection Data Processing Information & Reports Conclusion Measuring performance can be a powerful mechanism for managing performance!  Internal management  Quarterly ETA reports  Stakeholder reports  Customer satisfaction  Evaluation reports  Contractor reports  Internal management  Quarterly ETA reports  Stakeholder reports  Customer satisfaction  Evaluation reports  Contractor reports  Data Entry  Organizing data  Compiling data  Aggregating data  MIS  Excel spreadsheets  Access databases  Analyzing data  Data Entry  Organizing data  Compiling data  Aggregating data  MIS  Excel spreadsheets  Access databases  Analyzing data  Intake/Eligibility forms  Attendance sheets  Activity forms  Progress reports  Records of completion  Exit & follow-up forms  Satisfaction surveys  Intake/Eligibility forms  Attendance sheets  Activity forms  Progress reports  Records of completion  Exit & follow-up forms  Satisfaction surveys

Discretionary Grants: Data Collection, Processing & Reporting 51 Are there any questions? THANK YOU