Kimberlee Bryant, Technical Coordinator July 1, 2018 The National Reporting System for Adult Education (NRS): Outcomes and Reporting Kimberlee Bryant, Technical Coordinator July 1, 2018

Training Objectives Reinforce the Importance of Data Quality Review of the Intake Assessment Form Understand Periods of Participation Help you master Measurable Skill Gains Give the opportunity for questions and clarification

Why do we collect data? Adult Education services are provided through a combination of federal, state, and local funds. Federal funds require that student demographics, progress, and outcomes be collected and reported. The National Reporting System for Adult Education (NRS) ensures that all states report data in the same way. States report NRS data annually (on October 1st) to the Office of Career, Technical, and Adult Education (OCTAE), U.S. Department of Education, using data tables based on the fiscal year (July 1-June 30). States also report in tandem with state WIOA partners using a new Joint Report. OCTAE uses the states’ data to report about the program to Congress, other Federal agencies, the States and the general public. OCTAE also uses the data to set state performance levels (targets). Bullet 1 - Local funds (from the local organization, CLCP, other grants, contracts) Bullet 2 - Examples of student demographic measures – race/ethnicity, age, and gender. Bullet 2 - Examples of progress and outcomes - measurable skill gains (MSG), employment indicator, GED. Bullet 4 – Emphasize that a FY is from July 1 – June 30

Intake Assessment Form Primary tool for initial data collection Completed each time a student begins a new enrollment period. This is a change from prior years. Includes important data elements for federal, state, and programmatic reporting Key areas include: Highest School Grade Completed Highest Educational Certificate, Diploma, Degree Labor Force Status Special Status Populations (Barriers to Employment) Student data are collected through student self-reporting and staff observation (in some cases) Bullet 1 – Items with an asterisk on the IAF indicate required fields. Bullet 2 – Keep enrollment period on your mind. I’ll further explain this in a few minutes. Bullet 4 - Highest School Grade Completed and Highest Educational Certificate, Diploma, Degree: The highest number of years of formal schooling the learner has completed AND the highest credential or degree the learner has achieved. Schooling in the U.S. or abroad is included. Labor Force Status is where students identify if they are employed, unemployed or not in the labor force at the time of entry into the adult education program. Special Status Populations (Barriers to Employment) is a list of categories on page 2 of the IAF: low income, displaced homemaker, single parent, dislocated worker, homeless or runaway youth, ex offender, foster care, farm worker and cultural barriers. Emphasize: The importance of collection of SSN for matching purposes.

Intake Assessment Recommendations If you are a new employee, complete the FY19 Intake Assessment Form training, located at: http://literacy.coe.uga.edu/ops-inst-services/technicaltraining/index.html. Annually, review the current year Intake Assessment Form and the Directions and Definitions to make sure you are up-to-date. Seek guidance from experienced staff members: If a student refuses to give you information If you think something isn’t right If you don’t know how to explain part of the form to a student If you are unsure about the intake process in your program Trainee Check: Any questions about the intake process or the IAF? Optional (if time permits): Presenter could ask for a show of hands of employees who had completed IAF training. Presenter could ask if attendees know who to go to for IAF questions/problems.

Enrollment Period Starts with a student’s first day of recorded attendance (Program Entry) Ends the last date on which the participant received services and doesn’t receive services for at least 90 days. (The exact exit date cannot be determined until at least 90 days have elapsed since the last attendance date.) Programs do not have to track the exit date of each enrollment period, GALIS determines the exit date based on the last date of attendance. An enrollment period becomes a Period of Participation (PoP) when the student has a valid assessment and earns at least 12 hours. Program Entry Bullet 3 - Exit date is the last day of service. Services do not include follow-up telephone calls, emails, or text messages. Bullet 4 – PoP is a minimum of 12 hours and an assessment.

Enrollment Periods - Details Participants with more than one entry have multiple periods of enrollment in a program year. Every period of enrollment is a new service period and is treated as if the participant is a new participant (intake, MSG, follow-up, etc.). A new intake process is required for each new enrollment period, whether it becomes a PoP or not. Enrollment periods can continue across fiscal years. Emphasize: Every period of participation is treated as a separate event for a participant and performance indicators apply separately to each period. Point 4 – We will look at an example of this later in the training.

The New Table 4 First Period of Participation All Periods of Participation The New Table 4 Before we do an activity, let me show you where we report Periods of Participation. One of the most useful NRS Tables, Table 4, has been changed to show outcomes by Periods of Participation. We’ll go over Table 4 in more detail later, but for now, note that a student’s first Period of Participation of the year is reported on the left side of the table and the total number of PoPs, and whether those PoPs had positive outcomes, are reported on the right side. Column B is where we report a single count of students based upon their first achievement of a PoP – 12 hours and an assessment Column I is a duplicated count of students based upon their number of PoPs

Activity 1: Periods of Participation Before we begin the activity, What are the two key factors for an enrollment period to become a Period of Participation? How are PoPs recorded on Table 4? Question: “What are the two key factors for an enrollment period to become a Period of Participation?” Answer: A student has to attend the program at least 12 hours and have a valid assessment. Question: How are PoPs recorded on Table 4?: Answer: The First PoP – left side; All PoPs – right side

Activity 1: Periods of Participation Is it a PoP or not? Barbara enrolls in ABE2 in July. She attends 10 hours, exits the program and doesn’t return this fiscal year. Bao enrolls in ESL3 in August, stays for 8 hours, and leaves in September. He returns in October and stays for 20 hours. He exits at the end of December and does not return. Beatriz joins the IELCE program (ESL4) in January. She attends two days (4 hours) before leaving to take care of a family member. She re-enrolls in May and completes 65 hours in class and online before exiting at the end of June. Bryan enrolls in the GED preparation program (ABE5) in June and racks up 12 hours of attendance before June 30. He finishes the program in August when he gets 6 more hours of attendance and earns his GED® credential. Burt joins the ABE program in August at the ABE3 level. He attends 13 hours over the next two weeks and then exits. He returns in January and stays 26 hours before leaving the program again. Based on those parameters (12 hours and an assessment), for each of these examples, Is it a PoP or not? Barbara – One period of enrollment; PoP? No Bao – One period of enrollment; PoP? Yes Beatriz – Two periods of enrollment; First one a PoP? No; Second one a PoP? Yes Bryan – One period of enrollment that goes across two fiscal years; First one a PoP? Yes, because he had 12 hours of attendance before the new fiscal year began. He would be reported as a participant in the current fiscal year (without a GED) and reported as a participant in the new fiscal year (with a GED). His participant status carrries with him into the new fiscal year. Burt – Two periods of enrollment; First one a PoP? Yes: Second one a PoP? Yes

Activity 1: Periods of Participation Before we begin the next part of the activity, How are PoPs recorded on NRS Table 4? Before we begin the next part of the activity, How are PoPs recorded on NRS Table 4? First PoP – left side; All PoPs – right side

The New Table 4 First Period of Participation All Periods of Participation The New Table 4 Let’s now use the students from Activity 1 to populate their numbers in Columns B and I. Please look at Slide 10 and take out one of your copies of Table 4. First we have Barbara – How would Barbara be placed on Table 4?

The New Table 4 First Period of Participation All Periods of Participation Barbara (No PoP) The New Table 4 First we have Barbara – How would Barbara be placed on Table 4? Answer: Although Barbara was assessed on and received a level placement, she is not reported on Table 4 because she only attended 10 hours; What is the number of minimum hours a student has to attend? Answer: 12 hours Next, we have Bao – We agreed that he had 1 PoP. What was his entry level placement? How would he be represented in Column B on ESL3?

The New Table 4 First Period of Participation All Periods of Participation Barbara (No PoP) The New Table 4 Bao 1 Bao 1 Next, we have Bao – We agreed that he had 1 PoP. What was his entry level placement? Answer: ESL3; How would he be represented in Column B on ESL3? Answer: 1; How would he be reflected in Column I? Answer: 1; Why? Answer: Because he only had 1 period of enrollment that made 1 PoP Now let’s look at Beatriz – What was her placement level? How many PoPs did she have?

The New Table 4 First Period of Participation All Periods of Participation Barbara (No PoP) The New Table 4 Bao 1 Bao 1 Beatriz 1 Beatriz 1 Now let’s look at Beatriz – What was her placement level? Answer: ESL4; How many PoPs did she have? Answer: 1; Why? Answer: Because her first enrollment period was not a PoP; How would she reported in Column B? Answer: 1; How would she be reported in Column I? Answer: 1 Bryan is the next student – What was Bryan’s placement level? How many PoPs?

(Also reported as a participant next year and GED completion) First Period of Participation All Periods of Participation Barbara (No PoP) Bryan (Also reported as a participant next year and GED completion) The New Table 4 Bryan 1 Bryan 1 Bao 1 Bao 1 Beatriz 1 Beatriz 1 Bryan is the next student – What was Bryan’s placement level? Answer: ABE5; How many PoPs? Answer: 1 Why? His attendance is technically one period of participation, but since it continues across a fiscal year break, it is reported as two PoPs (one in each fiscal year). For this FY, he appears once in column B and once in column I. His GED, which was earned in the next fiscal year, will be reported, along with him, in the next fiscal year. Finally, we have Burt – What was Burt’s placement level? How many period of enrollment did he have?

(Also reported as a participant next year) First Period of Participation All Periods of Participation Barbara (No PoP) Bryan (Also reported as a participant next year) The New Table 4 Burt 1 Burt 2 Bryan 1 Bryan 1 Bao 1 Bao 1 Beatriz 1 Beatriz 1 Finally, we have Burt – What was Burt’s placement level? Answer: ABE3 How many period of enrollment did he have? Answer: 2; What number would be reported in Column B? Answer: 1; because this column is a single count of his first PoP; How would he be reported in Column I? Answer: 2, because he had 2 PoPs. Hopefully this gives you a better understanding of how students are reported based upon PoPs. Now we’re going to move on and discuss how student gains are reported.

Measurable Skill Gains (MSG) Educational Functioning Level Gain (1) Pre- and Post-Test (2) Program exit and entry into postsecondary education or training by June 30 (3) Secondary Diploma (GED) by June 30 Under the Workforce Investment Act (WIA) (old legislation), student completions were only reported via pre and post-testing and it was called EFL gain. Now, under our new legislation, WIOA, we have Measurable Skill Gains, which expands the opportunities for student completions/gains to be reported. Essentially, we have 2 types of gain opportunities EFL Gain – EFL gain is still pre and post-testing, but it has been expanded to include Postsecondary Enrollment. Please note that for Postsecondary to be counted as a completion, a student must exit the program (90 days) and enroll in Postsecondary education or training by the end of the FY, June 30th. Secondary Diploma achievement by June 30 – In Georgia that would be completion of the GED by June 30, regardless of program exit date. So, we have two types of MSG, but 3 ways to get MSG; let’s look at those 3 ways again (next slide).

Three ways to get MSG in Adult Education Comparing the participant’s pretest with the participant’s posttest, using an NRS approved test Enrollment in postsecondary education or training after exit, but by June 30th of the fiscal year from any level Earning a secondary diploma (GED) by June 30th of the fiscal year from any level This adds some text to the last slide’s illustration. Use this slide to Emphasize the details: Assessment must be NRS approved Student must exit and enroll in PS education or training by 6/30 GED must be earned by 6/30 PS and GED gains can come from any Educational Functioning Level (EFL) ESL1-6, ABE1-6

MSG and Reporting Students are placed in an EFL by their lowest pre-test score. All of their achievements are reported on this row of Table 4. MSG is reported on the new NRS Table 4 in two categories: EFL Gain (Pre/Post Gains and Enter Postsecondary by 6/30) GED® Credential Attainment by 6/30 Only one gain is reported per participant per PoP (the last one achieved by date or GED) In some situations, achievements made in a second PoP can be counted back in the first PoP. MSGs are counted to give the most benefit to the local program. Bullet 3 – This is a big change from our old legislation, WIA. Under that legislation we reported completions based upon the first completion in the lowest subject area or area(s). Please know that GALIS was modified to reflect this change during FY17. We’ll discuss the reporting of the last achievement in a few more slides.

The New Table 4 First Period of Participation All Periods of Participation The New Table 4 Use this slide to show the EFL levels and where the MSG types are recorded. Question: Does anyone remember what the two types of EFL gain are? Answer: Pre- and post-testing or Program Exit and Postsecondary education or training Enrollment by June 30. Column D is used to report EFL Gain. Column E is used to report GED gains by June 30. Column J is used to report the total number PoPs with Measurable Skill Gains.

MSG and Data Matching GED® Testing – Data match to determine achievement of a GED credential Monthly (15th of the month) Visible in GALIS on the Student Goals tab Postsecondary – Data match to determine enrollment Monthly with TCSG colleges Quarterly with the National Student Clearing House (national postsecondary database) Exit from Adult Education is required for this to count as MSG Not visible on the Student Goals tab Emphasize: The importance of collection of SSNs for this process.

General Rules about MSG Multiple Measurable Skill Gains (MSG) can happen during the enrollment period, but only the last one achieved (by date) will be reported on the NRS Tables. The exception to this rule is the GED credential, it is reported on the NRS Tables no matter when it was achieved during the fiscal year. (In prior trainings we have called this the royal flush.) Postsecondary enrollment can only be reported in one PoP. GED credential achievement can be reported in all PoPs for that fiscal year. In some situations, a post-testing achievement in a second PoP will be applied back to the first PoP. GALIS makes this determination behind the scenes. Question: What are the two conditions that must happen for a student to have a Postsecondary gain? Answer: 1) student must have exited the program; 2) student must enroll in Postsecondary by June 30th. Question: How many PoPs can a student have MSG through achievement of a GED? Answer: All PoPs for the fiscal year in which it was earned.

Enrollment Periods and Fiscal Years Enrollment Periods used to end at the conclusion of a fiscal year (June 30). An Enrollment Period can now span fiscal years, if there is continuous enrollment without an exit. But…since reporting is still done by fiscal year, the activity that happens on each side will be reported in its correlating year. Note to Trainer: Read bullet points first and then use the diagram as an example. Another example: If a student was enrolled in a fiscal year and continued into the new fiscal year, and let’s say earned their GED in July, which year is the MSG reported or can it be reported in both? Answer: 2nd FY; in order to be counted as MSG in the first, the student would have had to complete the GED by June 30th. GED obtainment can only be reported in the FY earned.

PoPs and Measurable Skill Gains Here’s another example of an enrollment period that goes across fiscal years. Emphasize: Our federal office, OCTAE refers to a fiscal year as a Program Year. So, our fiscal year 2017 is OCTAE’s Program Year 2016. We see clearly that the student enrolled early in the first fiscal year and had a POP. This means he had at least 12 hours and an assessment. The student then returned back to the program after at least a 90 day break, which created a new enrollment period. Question: When the student returned the second time, what must the program do? (Hint: We talked about this early in the presentation) Answer: Have them update their intake information for the new PoP. Now, let’s look closely at the second enrollment period. Question; Did the student stay at least 12 hours before June 30th? Answer: Yes; Question: What does that mean? Answer: He has enough hours for a new PoP. Note to Trainer: This is also a good point to emphasize that because the student had enough hours in both PoPs during PY2016 (FY2017), the program would be evaluated based on whether the student made 2 MSGs. OCTAE uses Program Year (PY) instead of Fiscal Year (FY). PY 2016 is FY 2017, etc.

The New Table 4 First Period of Participation All Periods of Participation The New Table 4 Let’s now look back at Table 4. We’ve already discussed Columns B and I as they relate to student count. What does B represent? Answer: The total of non-duplicated student count based upon the first PoP. What does Column I reflect? Answer: A duplicated count of students based on the total number of PoPs. Then we also talked about Columns D and E. What are those? Answer: The columns where MSG is reported for the first POP. Column D is EFL gain. What are the two ways a student may make EFL gain? Answer: 1) completion through pre and post-test, 2) Program exit and Postsecondary enrollment by June 30 And we’ve talked about Column J, which is where we report MSG based upon PoPs. For example, if you have a student with 2 PoPs, the student would be a count of two in Column I and then Column J number would be populated based on whether or not the student made a completion in each PoP. If the student only made 1 MSG in either the first or second PoP, then a 1 would go in Column J and when divided by Column I, the result would be 50% in Column K. If the student made an MSG in both PoPs, then 2 would be reported in Column J and when divided by Column I, the result would be 100% in Column K. Emphasize: Column K is what the feds look at to see how we perform as a state in relation to our negotiated percentages. Therefore, this is also the percentages that we look at for local programs.

Activity # 2: PoPs and MSG Read each student scenario and determine: How many MSGs did the student earn? How many PoPs did the student have? Using the NRS Table 4 provided, plot how the student would appear in each column. Use this slide to introduce the Activity. You will be given several student scenarios. For each one, you will need to answer these questions. (Read slide.)

Activity #2 examples: PoPs and MSG Celia enrolled in the ESL program in August. Her pre-test placed her on ESL3. She attended 65 hours and post-tested in December to an ESL4. She did not attend the remainder of the fiscal year. Chaz wants to earn a GED, so he joined the ABE program in October. His pre-test placed him at ABE5. He attended 25 hours and then stopped attending in December. He returned in May and stayed for 14 hours. In June, he took the GED Test, earned his credential, and exited the program. Clyde has a goal to improve his reading from his pre-test level of ABE2. Since joining the program in October he has attended 125 hours and post-tested twice without moving to the next level. It’s now March, and Clyde says he has to leave the program for seasonal employment. He says he will be back in 6 months. Please take out another copy of Table 4. (Give attendees a minute before each scenario to read and think about the answers.)

First Period of Participation All Periods of Participation Celia 1 65 1 0 0 0 100% 1 1 100% Looking at Celia, how many PoPs? One How many MSGs and in which PoPs? One MSG in One PoP Let’s plot her on NRS Table 4. Which row in Column A? ESL3 Column B - 1 Column C – 65 hours Column D - 1 Column E - 0 Column F - 0 Column G - 0 Column H – 100% Column I - 1 Column J - 1 Column K – 100%

First Period of Participation All Periods of Participation Chaz 1 39 0 1 0 0 100% 2 2 100% Celia 1 65 1 0 0 0 100% 1 1 100% Looking at Chaz, how many PoPs? Two How many MSGs and in which PoPs? Two MSG in Two PoPs Let’s plot him on NRS Table 4. Which row in Column A? ABE5 Column B - 1 Column C – 39 Column D - 0 Column E - 1 Column F - 0 Column G - 0 Column H – 100% Column I - 2 Column J - 2 Column K – 100%

First Period of Participation All Periods of Participation Clyde 1 125 0 0 1 0 0% 1 0 0% Chaz 1 39 0 1 0 0 100% 2 2 100% Celia 1 65 1 0 0 0 100% 1 1 100% Looking at Clyde, how many PoPs? One How many MSGs and in which PoPs? Zero MSG in One PoP Let’s plot him on NRS Table 4. Which row in Column A? ABE2 Column B - 1 Column C - 125 Column D - 0 Column E - 0 Column F - 1 Column G – 0 Column H – 0% Column I - 1 Column J - 0 Column K – 0%

Keep in Mind: MSG and Assessments Students must have a valid assessment for each enrollment period. The last test(s) from a prior enrollment period becomes the starting point for the next enrollment period as long as they are no older than 270 calendar days. Required hours between TABE tests is set by the entry Educational Functioning Level of the student (initial placement); If a student enters as ABE, he/she has to have a minimum of 40 hours between tests, if a student enters as ASE, he/she has to have a minimum of 30 hours between tests. This will update the next FY. The next three slides give some other pieces of information about PoPs, MSG, and GALIS.

Keep in Mind: MSG and TABE Under WIOA, each level completion by subtest area counts as a Measurable Skill Gain. Because of this, the subtest area that is reported in one PoP may not be the subtest area that is reported in the next PoP. Ex. PoP1: Reading level increased from ABE2 to ABE3. Math stayed the same at ABE2. Pop2: Math level increased from ABE2 to ABE3. Students can now transition from an ESL assessment to TABE within the fiscal year. Enrique enrolls in the ESL program at ESL5 and post-tests 60 hours later into ESLX (two MSG). He then transitions to the ABE program where he tests into ABE 1. After 45 hours he post-tests into ABE2 (one MSG). Keep in mind that Enrique would have his TABE gains reported in his assigned EFL row on Table 4, ESL5.

Keep in Mind: GALIS and Enrollment Periods The Georgia Adult Learners Information System (GALIS) has been modified to keep up with student enrollment periods and achievements. Students that are starting a new enrollment period and need an updated intake form will appear on the front page of GALIS. Multiple enrollment periods have complicated the way that GALIS counts MSG, but OAE has programmed GALIS to take advantage of every allowable way to count student outcomes.

FY19 Measurable Skill Gains Targets Negotiated with OCTAE Based on Past Performance Each EFL has its own target Aligns with column K on Table 4 FY19 Georgia Measurable Skill Gains (MSG)* Targets Entering Educational Functioning Level FY18 percentages FY19 percentages ABE Level 1 58 59 ABE Level 2 55 54 ABE Level 3 53 52 ABE Level 4 48 ABE Level 5 57 66 ABE Level 6 NA 56   ESL Level 1 60 ESL Level 2 62 ESL Level 3 61 ESL Level 4 ESL Level 5 51 ESL Level 6 Source: Georgia TCSG/OAE Negotiated Percentages FY18 – June 2016. These are the MSG targets that each local program has agreed to meet for FY19. FY18 percentages are used for comparison. MSG Target achievement is checked by looking at column K on Table 4. Please note this is the first year we have been given a target for ABE6.

Questions?

State-level contact information