Presented by: ETA/OWI Division of Strategic Investments Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant Program ROUND 1 COHORT REFRESH Webinar Date: July 26, 2013 Presented by: ETA/OWI Division of Strategic Investments U.S. Department of Labor Employment and Training Administration

Enter your location in the Chat window – lower left of screen Where are you? Enter your location in the Chat window – lower left of screen #

Presenters Presenter: Tom Hooper Title: Program Manager Organization: Division of Strategic Investments Presenter: Charlotte Harris Title: Workforce Analyst Organization: Division of Strategic Investments Presenter: Kristen Milstead Title: Workforce Analyst Organization: Division of Strategic Investments #

A Brief Summary of the Cohort Requirement Agenda A Brief Summary of the Cohort Requirement What You Must Do to Ensure Your Cohorts Follow Reporting Requirements Things to Know: Using Your Data for Continuous Improvement #

A Brief Summary of the Cohort Requirement #

OMB Package Instructions The OMB Package explains what you need to do in order to be compliant when it comes to cohort reporting Generally on p. 15-16 #

Each program of study that is funded by the grant should have its own set of cohorts. A program of study is defined in the reporting handbook as “an educational program in which a degree or certificate is granted.” Background: ETA leadership wanted all programs to have cohorts, rather than just a sample. Implications: Grantees will have many participant and comparison cohort sets. Data for each set is reported on a separate copy of Table 2 for the Annual Performance Report (APR). Table 2 may be submitted several times in the reporting system: once for each set of cohorts. #

A Representation of the Cohort Reporting Requirement PARTICIPANT UNIVERSE Everyone enrolled in a TAACCCT- funded class, course or program MATCHED NON-PARTICIPANT UNIVERSE PARTICIPANT COHORT COMPARISON COHORT PARTICIPANT COHORT MATCHED PARTICIPANT COHORT COMPARISON COHORT #

An Exception Grantees may combine programs of study into broader programs or industries, or into programs with the same credential type. EXAMPLE: Grantee B is planning to build or expand a program in Solar Photovoltaic Installation and a program in Wind Turbine Service Technician Training. Participants from each could be combined into one “renewable energy” program of study participant cohort. #

Acceptable Programs of Study: Guidelines Strongest Participant Cohort  No combining (one program of study) Strong Participant Cohort Different programs of study in the same industry or discipline with the same credential type/level May only be used if a stronger participant cohort cannot be selected. Acceptable participant cohort, but not ideal  Different programs of study in different industries or disciplines with the same credential duration. #

Gender and Age The percent male and average age of the comparison cohort must match the percent male and average age of the participant cohort. Purpose: To assist with ensuring the comparisons made between the two groups are valid. Exception: A “margin of discrepancy” between the two groups on either or both characteristics is allowed due to circumstances beyond a grantee’s control #

Year 1 Comparison Cohort Pilot Study Requirement ETA required the reporting of a “test” cohort during Year 1 to ensure that grantees were prepared to identify a cohort and report on it in preparation for the second annual report. Data was only collected for the test cohort for Year 1. The exception was if participants had enrolled and grantees had selected students for their participant cohort. In that case, actual participant and comparison cohort data should have been reported instead. Participant cohort is not enrolled in Year 1 Participants enrolled during Year 1 Grantees must identify and report on comparison cohort pilot study Grantees must report on the participant cohort and the comparison cohort in lieu of a comparison cohort pilot study #

IMPORTANT NOTE: YEAR 1 Whether you launched all of your programs… Whether you combined programs… YOU SHOULD HAVE REPORTED IN TABLE 2 OF THE APR FOR EVERY PROGRAM/COMBINED PROGRAM (EITHER PILOT STUDY DATA OR ACTUAL COHORT DATA, DEPENDING ON WHETHER PARTICIPANTS HAD ENROLLED IN THE PROGRAM OR NOT). #

Your Grant’s Review is Coming Soon Your FPO may have contacted you or will be contacting you soon about issues with your Year 1 APR cohort submission #

OMB Reporting Package Instructions for Year 2 “During year two, grantees should report on the actual comparison cohorts that they have established based on the parameters for matching to the actual enrollments that comprise the participant cohorts, and discontinue reporting on the test comparison cohort. The test cohort established in year one should not be used as the actual comparison cohort” (p. 16, Forms and Instructions). #

YEAR TWO ENDS SEPTEMBER 30, 2013. IMPORTANT NOTE: YEAR 2 YOU MUST REPORT ACTUAL PARTICIPANT AND COHORT DATA ON EVERY PROGRAM OF STUDY OR COMBINED PROGRAM OF STUDY. IF YOU HAVE NOT BEGUN COLLECTING COHORT DATA ON ALL PROGRAMS OR COMBINED PROGRAMS, NOW IS THE TIME TO START. YEAR TWO ENDS SEPTEMBER 30, 2013. #

What You Must Do to Ensure Your Cohorts and Table 2 Reporting Follow Reporting Requirements #

Issue #1 Number of programs to offer as stated in the QNPR exceeded number of cohorts reported on in the APR. If you combined programs, there would have been fewer. Please follow-up with your FPO to confirm whether this is the case and receive further instructions on how to resolve. #

Grant Name Grant Number Expected number of programs to offer: 2 Number of cohort data records: 2 Participant cohort started training? Y N Tracking basic skills deficiency? A_1A_Number Enrolled with basic skills deficiency- C 4 A_1A_Number Enrolled with basic skills deficiency- P   A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P B1_PROGRAM Business Administration Energy B2_COLLEGES_OFFERING 3 5 B3_PROGRAM_LENGTH 2 years 1 semester B4_COHORT_START DATE 01/01/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C 19 40 B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 17 B6_NUMBER OF STUDENTS_COMPLETING- C 20 #

Grant Name Grant Number Expected number of programs to offer: 27 Number of cohort data records: 1 Participant cohort started training? N Tracking basic skills deficiency? Y A_1A_Number Enrolled with basic skills deficiency- C 1 A_1A_Number Enrolled with basic skills deficiency- P   A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P B1_PROGRAM Criminal Justice B2_COLLEGES_OFFERING 7 B3_PROGRAM_LENGTH 2 years B4_COHORT_START DATE 01/01/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C 22 B5_NUMBER OF STUDENTS_IN_PARTICIPANT P B6_NUMBER OF STUDENTS_COMPLETING- C #

Issue #2 Number of cohorts reported on in the APR exceeded number of programs to offer as provided in the QNPR. Follow up with your FPO to clarify. #

Grant Name Grant Number Expected number of programs to offer: 10 Number of cohort data records: 39 Participant cohort started training? Y Tracking basic skills deficiency? A_1A_Number Enrolled with basic skills deficiency- C 19 A_1A_Number Enrolled with basic skills deficiency- P 2 A_1B_Number with demonstrated skills gains-C 15 A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C 11 12 A_1C Number with success in credit-bearing courses-P B1_PROGRAM Accounting Business Administration B2_COLLEGES_OFFERING N/A B3_PROGRAM_LENGTH 10 mo B4_COHORT_START DATE 08/06/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C B5_NUMBER OF STUDENTS_IN_PARTICIPANT P B6_NUMBER OF STUDENTS_COMPLETING- C B6_NUMBER OF STUDENTS_COMPLETING- P #

Issue #3 The pilot study for a program of study contained only zeroes. In other words, a pilot study was not actually conducted. There should at least be a number of students in the cohort and their demographic information (e.g., number of males and females, etc.). Follow up with your FPO to clarify. #

Grant Name Grant Number Participant cohort started training? N Tracking basic skills deficiency? Y A_1A_Number Enrolled with basic skills deficiency- C A_1A_Number Enrolled with basic skills deficiency- P   A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P B1_PROGRAM Building Information Technology GIS B2_COLLEGES_OFFERING 1 B3_PROGRAM_LENGTH 24 months B4_COHORT_START DATE 05/01/2013 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C B5_NUMBER OF STUDENTS_IN_PARTICIPANT P B6_NUMBER OF STUDENTS_COMPLETING- C B6_NUMBER OF STUDENTS_COMPLETING- P #

Issue #4 Participant data was included, but narrative information indicates the comparison cohort data was test data. The Table 2 for the program of study should contain either pilot study data for the comparison cohort or real data for both the participant and comparison cohort. Follow up with your FPO to confirm which type of data is contained in the report. #

Grant Name Grant Number B1_PROGRAM Pilot Cohort B2_COLLEGES_OFFERING 1 B3_PROGRAM_LENGTH 2 years B4_COHORT_START DATE 04/16/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C 44 B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 31 B6_NUMBER OF STUDENTS_COMPLETING- C B6_NUMBER OF STUDENTS_COMPLETING- P B6_INCUMBENT WORKERS COMPLETING-C B6_INCUMBENT WORKERS COMPLETING-P B7 STUDENTS RETAINED IN PROGRAM-C 38 B7 STUDENTS RETAINED IN PROGRAM-P 24 D1_COMPARISON COHORT DESCRIPTION The consortium conducted a pilot comparison study involving participants in grant-funded Programs of Study compared to students with similar characteristics (per matching guidelines provided by DOL) who participated in similar non-grant funded Programs of Study... #

Issue #5 Grantee states the participant cohort has started training, but only pilot study data for the comparison cohort was included. Follow up with your FPO to confirm which type of data is contained in the report. #

Grant Name Grant Program Participant cohort started training? Y Tracking basic skills deficiency? A_1A_Number Enrolled with basic skills deficiency- C A_1A_Number Enrolled with basic skills deficiency- P A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P B1_PROGRAM Associate Degree in Office Systems Associate Degree in Business Administration B2_COLLEGES_OFFERING -- B3_PROGRAM_LENGTH 3 years 3 years B4_COHORT_START DATE 08/01/2010 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C 38 82 B5_NUMBER OF STUDENTS_IN_PARTICIPANT P B6_NUMBER OF STUDENTS_COMPLETING- C #

Section A. Acceleration of Progress for Low-skilled and Other Workers “Grantees are required to submit information for this section only if their Statements of Work focus on Priority 1 (“Accelerate progress for low-skilled and other workers”)…” (p. 16). #

Issue #6 Grantee has indicated that it is focusing on Priority 1 in the SGA, however, none of the cohort data for any programs of study contain any participants who are identified as basic skills deficient. Follow up with your FPO to clarify. #

Grant Name Grant Number Participant cohort started training? Y Tracking basic skills deficiency? A_1A_Number Enrolled with basic skills deficiency- C A_1A_Number Enrolled with basic skills deficiency- P A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P #

Issue #7 Programs of Study If a degree or certificate is not granted, cohort reporting is not required for that grant-funded program, nor is it required in the reporting package. A grantee may, if desired, report on a program that is not considered a program of study, however, this does not count in lieu of the required reporting on programs of study. #

Developmental English and Math Grant Name Grant Number Participant cohort started training? N Tracking basic skills deficiency? Y A_1A_Number Enrolled with basic skills deficiency- C 239 A_1A_Number Enrolled with basic skills deficiency- P   A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C A_1C Number with success in credit-bearing courses-P B1_PROGRAM Developmental English and Math B2_COLLEGES_OFFERING All 7 campuses B3_PROGRAM_LENGTH approximately 3 years B4_COHORT_START DATE 08/20/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C #

Things to Know: Using Your Cohort Data for Continuous Improvement #

A Reminder: Purpose of the Cohort Requirement Allows for an analysis between the group that received something extra and the group that did not Did the “something extra” make a difference? The “something extra” in this is TAACCCT funding, which encompasses the sum total of all of the innovations that could not have been done without the funding If executed properly, the cohort requirement may allow you to use the data to improve your programs #

When Cohorts Are Treated Differently Ideally, the only difference between those in the participant cohort and those in the comparison cohort would be that those in the former were in programs of study that were TAACCCT-funded and those in the latter were not. Not always realistic All of the recommendations offered were to help you to keep the two as pure as possible #

Quiz You’re reporting on the participant cohort, but you haven’t selected your comparison cohort yet. Will this affect your results at the end of the grant period of performance? A. Yes, it biases the results in favor of the participant cohort. B. Yes, it biases the results in favor of the comparison cohort. C. Yes, it makes it difficult to determine whether my innovations had an impact. D. A and C E. No, it has no effect on the results of my cohort data. #

Time is On Our Side… Or Is It? If the participant cohort started earlier than the comparison cohort… At any point during the period of performance, when you compare the outcomes between the two, the participant cohort will always be “ahead” Be cautious about drawing conclusions about the effectiveness of your program if your comparison cohort starts later. #

Quiz True or False: It doesn’t really matter how many people are in my cohorts. True False #

Cohort Size Matters Part I With 2 people in your participant cohort, the results for entered employment will either be… 2 people enter employment… (100%) 1 person enters employment… (50%) 0 people enter employment… (0%) Be cautious about drawing conclusions about the effectiveness of your program if you have too few people in either or both cohorts. #

Cohort Size Matters Part II If you have significantly more students in your one of your cohorts for a particular program of study… Participant Cohort Comparison Cohort Number of Participants 500 1000 Entered Employment 250 400 Entered Employment Rate 50% 40% Be cautious about drawing conclusions about the effectiveness of your program if you have too few people in either or both cohorts. #

It’s Not a Match Made in Heaven Using the same comparison cohort for seemingly unlike programs of study (e.g., Welding and Health Care Aid) Using an unlike program of study for the comparison cohort from the participant cohort will yield results that are not comparable by educational program of study. #

Grant Name Grant Number Participant cohort started training? Y Tracking basic skills deficiency? A_1A_Number Enrolled with basic skills deficiency- C 443 A_1A_Number Enrolled with basic skills deficiency- P 7 11 A_1B_Number with demonstrated skills gains-C A_1B_Number with demonstrated skills gains-P A_1C Number with success in credit-bearing courses-C 233 A_1C Number with success in credit-bearing courses-P B1_PROGRAM Welding Health Care Aid B2_COLLEGES_OFFERING N/A B3_PROGRAM_LENGTH 2 years B4_COHORT_START DATE 09/04/2012 00:00:00 B5_NUMBER OF STUDENTS_IN_COMPARISON C 804 B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 16 17 B6_NUMBER OF STUDENTS_COMPLETING- C 204 #

In Conclusion Your cohorts must be reported in certain ways, as described in the reporting package Your FPO will contact you to go over any issues with your Year 1 cohort reporting like those described in this Webinar In Year 2, you must report actual participant and comparison cohort data for each program of study or combined program of study Be aware of the issues that may affect your ability to draw certain conclusions from your data #

Resources: Community of Practice http://etagrantees.workforce3one.org Available online: Reporting Templates/ Forms FAQs Data Elements Reporting System User Guide and Tutorial Reporting Schedules Newsletters TA Materials #

Please enter your questions in the Chat Room! #

Please send to the TAACCCT mailbox at taaccct@dol.gov Follow-Up Questions Please send to the TAACCCT mailbox at taaccct@dol.gov #

Look for upcoming TAACCCT Round 1 Webinars! SAVE DATE the TAACCCT Back to School September 12, 2013 TAACCCT APR Table 1 YEAR 2 September 26, 2013 #

Find resources for workforce system success at: Thank You! Find resources for workforce system success at: www.workforce3one.org #