1. Presented by: ETA/OWI Division of Strategic Investments
Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant Program ROUND 1 COHORT REFRESH
Webinar Date: July 26, 2013
Presented by: ETA/OWI Division of Strategic Investments
U.S. Department of Labor
Employment and Training Administration

2. Enter your location in the Chat window – lower left of screen
Where are you?
Enter your location in the Chat window – lower left of screen #

3. Presenters Presenter: Tom Hooper Title: Program Manager
Organization: Division of Strategic Investments
Presenter: Charlotte Harris
Title: Workforce Analyst
Organization: Division of Strategic Investments
Presenter: Kristen Milstead
Title: Workforce Analyst
Organization: Division of Strategic Investments #

4. A Brief Summary of the Cohort Requirement
Agenda
A Brief Summary of the Cohort Requirement
What You Must Do to Ensure Your Cohorts Follow Reporting Requirements
Things to Know: Using Your Data for Continuous Improvement #

5. A Brief Summary of the Cohort Requirement#

6. OMB Package Instructions
The OMB Package explains what you need to do in order to be compliant when it comes to cohort reporting
Generally on p #

7. Each program of study that is funded by the grant should have its own set of cohorts.
A program of study is defined in the reporting handbook as “an educational program in which a degree or certificate is granted.”
Background: ETA leadership wanted all programs to have cohorts, rather than just a sample.
Implications: Grantees will have many participant and comparison cohort sets. Data for each set is reported on a separate copy of Table 2 for the Annual Performance Report (APR).
Table 2 may be submitted several times in the reporting system: once for each set of cohorts. #

8. A Representation of the Cohort Reporting Requirement
PARTICIPANT UNIVERSE
Everyone enrolled in a TAACCCT- funded class,
course or program
MATCHED
NON-PARTICIPANT UNIVERSE
PARTICIPANT COHORT
COMPARISON COHORT
PARTICIPANT COHORT
MATCHED
PARTICIPANT COHORT
COMPARISON COHORT #

9. An Exception
Grantees may combine programs of study into broader programs or industries, or into programs with the same credential type.
EXAMPLE: Grantee B is planning to build or expand a program in Solar Photovoltaic Installation and a program in Wind Turbine Service Technician Training. Participants from each could be combined into one “renewable energy” program of study participant cohort. #

10. Acceptable Programs of Study: Guidelines
Strongest Participant Cohort 
No combining (one program of study)
Strong Participant Cohort
Different programs of study in the same industry or discipline with the same credential type/level
May only be used if a stronger participant cohort cannot be selected.
Acceptable participant cohort, but not ideal
 Different programs of study in different industries or disciplines with the same credential duration. #

11. Gender and Age
The percent male and average age of the comparison cohort must match the percent male and average age of the participant cohort.
Purpose: To assist with ensuring the comparisons made between the two groups are valid.
Exception: A “margin of discrepancy” between the two groups on either or both characteristics is allowed due to circumstances beyond a grantee’s control #

12. Year 1 Comparison Cohort Pilot Study Requirement
ETA required the reporting of a “test” cohort during Year 1 to ensure that grantees were prepared to identify a cohort and report on it in preparation for the second annual report.
Data was only collected for the test cohort for Year 1.
The exception was if participants had enrolled and grantees had selected students for their participant cohort. In that case, actual participant and comparison cohort data should have been reported instead.
Participant cohort is not enrolled in Year 1
Participants enrolled during Year 1
Grantees must identify and report on comparison cohort pilot study
Grantees must report on the participant cohort and the comparison cohort in lieu of a comparison cohort pilot study #

13. IMPORTANT NOTE: YEAR 1 Whether you launched all of your programs…
Whether you combined programs…
YOU SHOULD HAVE REPORTED IN TABLE 2 OF THE APR FOR EVERY PROGRAM/COMBINED PROGRAM (EITHER PILOT STUDY DATA OR ACTUAL COHORT DATA, DEPENDING ON WHETHER PARTICIPANTS HAD ENROLLED IN THE PROGRAM OR NOT). #

14. Your Grant’s Review is Coming Soon
Your FPO may have contacted you or will be contacting you soon about issues with your Year 1 APR cohort submission #

15. OMB Reporting Package Instructions for Year 2
“During year two, grantees should report on the actual comparison cohorts that they have established based on the parameters for matching to the actual enrollments that comprise the participant cohorts, and discontinue reporting on the test comparison cohort. The test cohort established in year one should not be used as the actual comparison cohort” (p. 16, Forms and Instructions). #

16. YEAR TWO ENDS SEPTEMBER 30, 2013.
IMPORTANT NOTE: YEAR 2
YOU MUST REPORT ACTUAL PARTICIPANT AND COHORT DATA ON EVERY PROGRAM OF STUDY OR COMBINED PROGRAM OF STUDY. IF YOU HAVE NOT BEGUN COLLECTING COHORT DATA ON ALL PROGRAMS OR COMBINED PROGRAMS, NOW IS THE TIME TO START.
YEAR TWO ENDS SEPTEMBER 30, 2013. #

17. What You Must Do to Ensure Your Cohorts and Table 2 Reporting Follow Reporting Requirements#

18. Issue #1
Number of programs to offer as stated in the QNPR exceeded number of cohorts reported on in the APR.
If you combined programs, there would have been fewer.
Please follow-up with your FPO to confirm whether this is the case and receive further instructions on how to resolve. #

19. Grant Name Grant Number
Expected number of programs to offer: 2 Number of cohort data records: 2
Participant cohort started training? Y N
Tracking basic skills deficiency?
A_1A_Number Enrolled with basic skills deficiency- C 4
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Business Administration
Energy
B2_COLLEGES_OFFERING 3 5
B3_PROGRAM_LENGTH
2 years
1 semester
B4_COHORT_START DATE
01/01/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C 19 40
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 17
B6_NUMBER OF STUDENTS_COMPLETING- C 20 #

20. Grant Name Grant Number Expected number of programs
to offer: 27 Number of cohort data records: 1
Participant cohort started training? N
Tracking basic skills deficiency? Y
A_1A_Number Enrolled with basic skills deficiency- C 1
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Criminal Justice
B2_COLLEGES_OFFERING 7
B3_PROGRAM_LENGTH
2 years
B4_COHORT_START DATE
01/01/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C 22
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P
B6_NUMBER OF STUDENTS_COMPLETING- C #

21. Issue #2
Number of cohorts reported on in the APR exceeded number of programs to offer as provided in the QNPR.
Follow up with your FPO to clarify. #

22. Grant Name Grant Number
Expected number of programs to offer: 10 Number of cohort data records: 39
Participant cohort started training? Y
Tracking basic skills deficiency?
A_1A_Number Enrolled with basic skills deficiency- C 19
A_1A_Number Enrolled with basic skills deficiency- P 2
A_1B_Number with demonstrated skills gains-C 15
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C 11 12
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Accounting
Business Administration
B2_COLLEGES_OFFERING
N/A
B3_PROGRAM_LENGTH
10 mo
B4_COHORT_START DATE
08/06/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P
B6_NUMBER OF STUDENTS_COMPLETING- C
B6_NUMBER OF STUDENTS_COMPLETING- P #

23. Issue #3
The pilot study for a program of study contained only zeroes. In other words, a pilot study was not actually conducted.
There should at least be a number of students in the cohort and their demographic information (e.g., number of males and females, etc.).
Follow up with your FPO to clarify. #

24. Grant Name Grant Number Participant cohort started training? N
Tracking basic skills deficiency? Y
A_1A_Number Enrolled with basic skills deficiency- C
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Building Information Technology
GIS
B2_COLLEGES_OFFERING 1
B3_PROGRAM_LENGTH
24 months
B4_COHORT_START DATE
05/01/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P
B6_NUMBER OF STUDENTS_COMPLETING- C
B6_NUMBER OF STUDENTS_COMPLETING- P #

25. Issue #4
Participant data was included, but narrative information indicates the comparison cohort data was test data.
The Table 2 for the program of study should contain either pilot study data for the comparison cohort or real data for both the participant and comparison cohort.
Follow up with your FPO to confirm which type of data is contained in the report. #

26. Grant Name Grant Number B1_PROGRAM Pilot Cohort B2_COLLEGES_OFFERING 1
B3_PROGRAM_LENGTH
2 years
B4_COHORT_START DATE
04/16/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C 44
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 31
B6_NUMBER OF STUDENTS_COMPLETING- C
B6_NUMBER OF STUDENTS_COMPLETING- P
B6_INCUMBENT WORKERS COMPLETING-C
B6_INCUMBENT WORKERS COMPLETING-P
B7 STUDENTS RETAINED IN PROGRAM-C 38
B7 STUDENTS RETAINED IN PROGRAM-P 24
D1_COMPARISON COHORT DESCRIPTION
The consortium conducted a pilot comparison study involving participants in grant-funded Programs of Study compared to students with similar characteristics (per matching guidelines provided by DOL) who participated in similar non-grant funded Programs of Study... #

27. Issue #5
Grantee states the participant cohort has started training, but only pilot study data for the comparison cohort was included.
Follow up with your FPO to confirm which type of data is contained in the report. #

28. Grant Name Grant Program Participant cohort started training? Y
Tracking basic skills deficiency?
A_1A_Number Enrolled with basic skills deficiency- C
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Associate Degree in Office Systems
Associate Degree in Business Administration
B2_COLLEGES_OFFERING --
B3_PROGRAM_LENGTH
3 years
3 years
B4_COHORT_START DATE
08/01/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C 38 82
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P
B6_NUMBER OF STUDENTS_COMPLETING- C #

29. Section A. Acceleration of Progress for Low-skilled and Other Workers
“Grantees are required to submit information for this section only if their Statements of Work focus on Priority 1 (“Accelerate progress for low-skilled and other workers”)…” (p. 16). #

30. Issue #6
Grantee has indicated that it is focusing on Priority 1 in the SGA, however, none of the cohort data for any programs of study contain any participants who are identified as basic skills deficient.
Follow up with your FPO to clarify. #

31. Grant Name Grant Number Participant cohort started training? Y
Tracking basic skills deficiency?
A_1A_Number Enrolled with basic skills deficiency- C
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P #

32. Issue #7 Programs of Study
If a degree or certificate is not granted, cohort reporting is not required for that grant-funded program, nor is it required in the reporting package.
A grantee may, if desired, report on a program that is not considered a program of study, however, this does not count in lieu of the required reporting on programs of study. #

33. Developmental English and Math
Grant Name
Grant Number
Participant cohort started training? N
Tracking basic skills deficiency? Y
A_1A_Number Enrolled with basic skills deficiency- C
239
A_1A_Number Enrolled with basic skills deficiency- P
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Developmental English and Math
B2_COLLEGES_OFFERING
All 7 campuses
B3_PROGRAM_LENGTH
approximately 3 years
B4_COHORT_START DATE
08/20/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C #

34. Things to Know: Using Your Cohort Data for Continuous Improvement#

35. A Reminder: Purpose of the Cohort Requirement
Allows for an analysis between the group that received something extra and the group that did not
Did the “something extra” make a difference?
The “something extra” in this is TAACCCT funding, which encompasses the sum total of all of the innovations that could not have been done without the funding
If executed properly, the cohort requirement may allow you to use the data to improve your programs #

36. When Cohorts Are Treated Differently
Ideally, the only difference between those in the participant cohort and those in the comparison cohort would be that those in the former were in programs of study that were TAACCCT-funded and those in the latter were not.
Not always realistic
All of the recommendations offered were to help you to keep the two as pure as possible #

37. Quiz
You’re reporting on the participant cohort, but you haven’t selected your comparison cohort yet. Will this affect your results at the end of the grant period of performance?
A. Yes, it biases the results in favor of the participant cohort.
B. Yes, it biases the results in favor of the comparison cohort.
C. Yes, it makes it difficult to determine whether my innovations had an impact.
D. A and C
E. No, it has no effect on the results of my cohort data. #

38. Time is On Our Side… Or Is It?
If the participant cohort started earlier than the comparison cohort…
At any point during the period of performance, when you compare the outcomes between the two, the participant cohort will always be “ahead”
Be cautious about drawing conclusions about the effectiveness of your program if your comparison cohort starts later. #

39. Quiz True or False: It doesn’t really matter how many
people are in my cohorts.
True
False #

40. Cohort Size Matters Part I
With 2 people in your participant cohort, the results for entered employment will either be…
2 people enter employment… (100%)
1 person enters employment… (50%)
0 people enter employment… (0%)
Be cautious about drawing conclusions about the effectiveness of your program if you have too few people in either or both cohorts. #

41. Cohort Size Matters Part II
If you have significantly more students in your one of your cohorts for a particular program of study…
Participant Cohort
Comparison Cohort
Number of Participants
500
1000
Entered Employment
250
400
Entered Employment Rate
50%
40%
Be cautious about drawing conclusions about the effectiveness of your program if you have too few people in either or both cohorts. #

42. It’s Not a Match Made in Heaven
Using the same comparison cohort for seemingly unlike programs of study (e.g., Welding and Health Care Aid)
Using an unlike program of study for the comparison cohort from the participant cohort will yield results that are not comparable by educational program of study. #

43. Grant Name Grant Number Participant cohort started training? Y
Tracking basic skills deficiency?
A_1A_Number Enrolled with basic skills deficiency- C
443
A_1A_Number Enrolled with basic skills deficiency- P 7 11
A_1B_Number with demonstrated skills gains-C
A_1B_Number with demonstrated skills gains-P
A_1C Number with success in credit-bearing courses-C
233
A_1C Number with success in credit-bearing courses-P
B1_PROGRAM
Welding
Health Care Aid
B2_COLLEGES_OFFERING
N/A
B3_PROGRAM_LENGTH
2 years
B4_COHORT_START DATE
09/04/ :00:00
B5_NUMBER OF STUDENTS_IN_COMPARISON C
804
B5_NUMBER OF STUDENTS_IN_PARTICIPANT P 16 17
B6_NUMBER OF STUDENTS_COMPLETING- C
204 #

44. In Conclusion
Your cohorts must be reported in certain ways, as described in the reporting package
Your FPO will contact you to go over any issues with your Year 1 cohort reporting like those described in this Webinar
In Year 2, you must report actual participant and comparison cohort data for each program of study or combined program of study
Be aware of the issues that may affect your ability to draw certain conclusions from your data #

45. Resources: Community of Practice
Available online:
Reporting Templates/ Forms
FAQs
Data Elements
Reporting System User Guide and Tutorial
Reporting Schedules
Newsletters
TA Materials #

46. Please enter your questions in the Chat Room!#

47. Please send to the TAACCCT mailbox at taaccct@dol.gov
Follow-Up Questions
Please send to the TAACCCT mailbox at #

48. Look for upcoming TAACCCT Round 1 Webinars!
SAVE DATE
the
TAACCCT Back to School September 12, 2013
TAACCCT APR Table 1 YEAR 2
September 26, 2013 #

49. Find resources for workforce system success at:
Thank You!
Find resources for workforce system success at: #