LEAP Again: Updates and Training of Trainers on WIOA Accountability Larry Condelli & Amanda Duffy American Institutes for Research National Reporting System Support Project Presented at the Adult Education State Directors’ Meeting June 27, 2017

Questions? Write questions on post-its. We’ll collect and respond.

Purpose of This Session Review and clarify key accountability topics. Present updates to performance reporting requirements. Practice responding to challenging questions about National Reporting System (NRS) table changes. States and local programs have many questions as they begin to implement any new policy. WIOA is no exception. Over the past few month OCTAE has clarified policy and procedures in response to questions from state and local staff and has reviewed comments from a formal information request. As a result, OCTAE has made some necessary tweaks to the policy. It also has made change to NRS reporting tables. This is an opportunity to learn the most recent changes to WIOA. You’ll also have the chance to test your knowledge of key components to the updated NRS tables as a result of WIOA requirements. Objectives: Review and clarify key accountability topics Explain changes to NRS table requirements

Updates and Clarifications Periods of Participation (PoPs) and Assessment: Review procedures. Carry over across PoPs and program years. Implications regarding PoPs. Placement and posttest gains. Table updates and reminders: Table 4, Contact hours. Table 4b, Participants who are pre- and post-tested.

Periods of Participation (PoPs) and Assessment

PoPs –Reminder of Policy A Period of Participation (PoP) begins when an individual enters the program. A PoP ends at program exit (90 days without service and no scheduled services). A PoP does not end when a program year ends.

Periods of Participation (POP) Follow-Up Indicators Example This example shows the basic concept of Periods of Participation. There are 2 separate program entries followed by corresponding program exits. There is a PoP that continues across program years.

PoPs –Implications Participant Status An individual becomes a participant once he/she has attained 12 hours of instruction within a period of participation (PoP). Participant status is retained until an exit occurs. Individuals must achieve participant status each time a new PoP begins. Here we want to clarify a few interactions between participants and periods of participation. We will show this visually in an upcoming slide.

PoPs –Implications Participant Status (continued) Participants who remain continuously enrolled across multiple program years do not need to requalify as a participant in new program years. Measurable Skill Gain (MSG) is reported at least once for every participant per program year. Since participation status ends only when someone exits the program, those participants who continue receiving services across program years do not need to requalify as a participant in the subsequent program years.

Periods of Participation (POP) MSG Indicator Example This illustration shows how participation continues across program years.

Changes to Reporting MSG in a Prior PoP Purpose: To ensure PoPs do not have an impact on proper testing practices, the following clarifications have been made. Educational Functioning Level (EFL) placement may be carried over from a previous PoP, if the state accounts for this practice in the state assessment policy and the test is still valid according to the test publisher’s guidelines. EFL gain via post-testing may be achieved in a PoP based on post- testing in a subsequent PoP, assuming the test scores are still valid according to test publisher guidelines.

Reporting MSG in a Prior PoP: Scenario 1 MSG may be applied to a previous PoP under the following scenario: A participant exits, has enough hours to posttest (according to state policy and test guidelines), but does not complete a posttest. The individual reenters the program (PoP2) and is tested at entry. Since the participant had already qualified to posttest based on state policy and test guidelines, the test given upon reentry may be used as the pretest for PoP2 and EFL gain may be counted for PoP1, if a gain is achieved.

Reporting MSG in a Prior PoP: Scenario 1 (continued) This illustration shows a posttest occurring at the beginning of a new PoP.

Reporting MSG in a Prior PoP: Scenario 2 EFL gain may also be applied to a previous PoP under this scenario: A participant exits, does not have enough hours to posttest (according to state policy and test guidelines), and is not post-tested. The individual returns to the program (PoP2), is tested after achieving enough instructional hours (combining hours in PoP1 and PoP2), and achieves an EFL gain. This test in PoP2 serves as the pretest for PoP2 and as a posttest for PoP1. The EFL gain may be counted for PoP1. EFL gain in PoP2 would require another test after the participant receives sufficient instructional hours for a posttest.

Reporting MSG in a Prior PoP: Scenario 2 (continued) This illustration shows a posttest occurring after the appropriate contact hours have been achieved across 2 PoPs.

Reporting MSG in Prior PoP: Requirements In order to apply MSG across PoPs via pre-/post-testing: The participant must have sufficient hours to posttest according to state policy and test guidelines from either PoP1 or combined PoP1+PoP2. The PoPs must be in the same program year. MSG cannot be retroactively applied to a prior program year.

Activity With a partner, write a scenario and response that may happen in a program around periods of participation and assessment. Use the information learned on slides 12 and 14 to support your writing. Quiz another team with your scenario. Have fun with this! Be creative! This will help you respond to locals’ questions.

Questions? (Section 1)

Assessment: Placement and EFL Gain

Assessment and EFL: Setting Initial Placement As of July 1, 2017: Initial Placement is the first student placement assigned during a program year. Initial placement for a program year may be set using any subject area of pre-tests given. NRS will no longer indicate placement in the lowest subject area of all tests. The EFL for initial placement should be locked in upon assignment to ensure accurate data reporting. This is more explicitly noted now, but it was allowed under WIA. It is no longer articulated that a participant must be placed at the lowest level.

Assessment and EFL: Reporting based on Initial Placement All participant data reported on NRS Table 4 will be reported on the row of the initial placement for a given Program Year. Only columns I and J will be updated on NRS Table 4 for subsequent PoPs. Include total attendance hours for all PoPs in column C. We will show an example for reporting on NRS Table 4 in an upcoming slide.

Assessment: Changing How EFL Gains Can Be Made

Assessment and EFL Gain: Example: Jim

Reporting Gain on NRS Table 4: Jim’s Example This is an example of reporting multiple gains across multiple PoPs on NRS Table 4. Jim has 2 PoPs. The secondary credential attainment was the most recent gain and can be applied as an MSG to both PoPs. Only 1 gain may be counted per PoP, so PoP1 (columns B-H) shows the secondary credential gain, since it was the most recent gain. The second PoP shows the secondary credential gain as an MSG as well.

Guiding Questions At your table, discuss one of the following questions: How might the changes discussed affect program practice? Positively or negatively? Why? Do you have concerns about accurate testing or misuse of tests? What are they and why? What affect might the changes have on data collection or data systems? On performance? We can assign questions. Include a report out and group discussion

Questions? (Section 2)

Credential Attainment Rate

Credential Attainment Rate (continued)

Credential Attainment Indicator-Calculations Numerator Denominator

Credential Attainment Indicator-Calculations (continued) Numerator Denominator

NRS Tables 4C and 5A Participant data for all periods of participation are included on distance education tables, if the participant met the state’s definition of a distance education participant during any POP during the program year.

Reporting Timeline 2017-2020 October 15, 2017, is a Sunday. The reporting date will slide to October 16, 2017.

NRS Table Questions

Table Activity Tables will be given a question/scenario based on new reporting requirements. As a group they will provide an answer to the question. One person should be assigned the note taker and another to report out. Each group will share their question (we can put on slides too) and response. Everyone will have a red and green card and after response is given, will hold up green if they agree, red if they disagree. The ones that disagree will have to provide their rationale and their correct answer. Group will discuss. Another round of questions will be shared based on time left. Questions will be pulled from Alan’s list and written responses will be provided to participants. [Kelly is in the process of putting together the document for the questions]

Wrap-up

NRS Regional Training, Fall 2017 WIOA Accountability: Year One Data Check-Up September 13- 15, Los Angeles, CA October 11-13, Bethesda, MD October 24-26, Austin, TX Pre-Training Activities: As you review your year end data, identify “problems” and write 1-3 questions that need answers. Example: How do we provide better follow-up? Consider your current data flow. Check your system for validation rules and error checks. Goals of the Training Review WIOA data collection requirements for performance indicators, focusing on challenges to data collection and reporting based on their first year of implementation and successful ways to address them. Review data flow from data collection to reporting to identify effective methods Explore methods for improving data quality and how to implement them Identify data monitoring methods, including effective reports. Develop a plan for enhancing data quality through improving data monitoring and training to local staff Pre-Training Activities: We’re telling you about this now because it’s more than we’ve asked in the past. The training will not provide an excessive amount of new content. Rather it will be tailored to your current systems and how you can better make it work for your state. We need you to have clear questions and understanding of what is currently working and not working. We will provide templates and guides for these pretraining activities on Moodle once it opens. As you review your year end data, identify “problems” and write 1-3 questions that need answers (example: how do we provide better follow-up?) Consider your current data flow Check your system for validation rules and error checks Need to be able to tell them when the registration will be sent.

Additional Questions? Please contact us at: Alan Tucker alan.tucker@ed.gov Larry Condelli LCondelli@air.org Jay LeMaster john.lemaster@ed.gov