Literacy and Numeracy

2 Agenda Background Key Variables for Calculating Literacy and Numeracy Rate –Definition of a Youth –Out-of-School –Basic Literacy Skills Deficiency –Educational Functioning Level –Program Years –Reasons For Exit –Participation Years Testing Requirements for Youth

3 Background Target Population – More than 40 million adults, or approximately 21 percent of the adult population of the United States, possess limited literacy capability—that is, they have not completed a high school diploma or equivalent.

4 Target Population Statistics 1 37 % (or about 8 % of the total adult population) have completed eight or fewer years of education 11.3 % - four or fewer years of schooling 33 % English is their second language. 1

5 Background TEGL 17-05, Common Performance Measures Policy for the Employment and Training Accountability System TEGL 3-04 New Strategic Vision for the Delivery of Youth Services under the Workforce Investment System National Reporting System (NRS) for Adult Education – U.S. Department of Education

6 ETA & NRS Implementation Notes ETA is using the NRS EFL Structure EFL Implementation –ETA allows for reporting on various EFL –NRS (Adult ED) reports on lowest EFL Calculations –ETA will use a different set of calculations Data Quality –Both ETA & NRS are focused on data integrity and reporting

7 Literacy or Numeracy Gains Of those out-of-school youth who are basic skills deficient: # of youth participants who increase one or more educational functioning levels # of youth participants who have completed a year in the program* plus the # of youth participants who exit before completing a year in the youth program *(i.e. one year from the date of first youth program service)

8 Key Variables for Calculating Literacy and Numeracy Rate

9 Definition of a Youth There are two WISARD layouts –OMB approved layout –ETA software layout Differences –Fields 306 & 307 have been changed ItemOMB ApprovedETA Software 306Youth (Local)Date of First Youth Service 307Youth [Statewide (15%) Activities] Youth (Statewide 15% Activities)

10 Definition of a Youth (cont.) ETA software uses WIASRD as the basis of its record layout. We are using the WIASRD that we expect to be approved by OMB for the October 2006 WIASRD submission.

11 Calculation of Age for Youth Date of First Youth Service minus Date of Birth –Item 306 – Item 102 = Age Age >= 14 and Age <= 21 There is no younger or older youth differentiation

12 Out-of-School Out-of-School is defined as: –An eligible youth who is a school dropout, or who has received a secondary school diploma or its equivalent but is basic skills deficient, unemployed, or underemployed (WIA section 101(33)). –For reporting purposes, this term includes all youth except: (i) those who are attending any school and have not received a secondary school diploma or its recognized equivalent, or (ii) those who are attending post-secondary school and are not basic skills deficient.

13 Understanding Out-of-School Youth For Reporting Purposes: 1.Not in school 2.In post-secondary education & basic skills deficient We could be looking potentially at anyone who is not in secondary school and is basic skills deficient

14 Basic Literacy Skills Deficient Is defined as: –The individual computes or solves problems, reads, writes, or speaks English at or below the eighth grade level or is unable to compute or solve problems, read, write, or speak English at a level necessary to function on the job, in the individual’s family, or in society.

15 Basic Literacy Skills Deficient For the WISARD – During Year 1 –Basic Literacy Skills Deficient, Item 130 = 1 –Initial Pre-test score, EFL and test date must be recorded For WISARD – During Subsequent Years –Based on EFL (Year #1, #2, etc.), Items 709, 712, etc. <= 4

16 Educational Functioning Levels Based on the National Reporting Systems EFL Table Educational Functioning Levels are based on skill sets – a measure of adult educational progress They are equal to approximately 2 grade levels

17 Educational Functioning Levels Educational Gain – –At post-test, participant completes or advances one or more educational functioning levels from the starting level measured on entry into the program (pre-test) – for Year 1. –At post-test, participant completes or advances one or more educational functioning levels from the post-test level measured from the previous year’s latest post-test – for Year 2 and beyond.

18 EFL Calculation To increase an EFL: EFL (based on test) during the current participation year minus the prior EFL (based on prior test) (either the pre-test or one of the post-tests) > 0. If EFL is either 0 or -1 then there is a no success. An increase of one EFL in any functional area during a participation year is considered a success for that participation year.

19 EFL Calculation Success for year one is – –Educational Functioning Level Item 709 > than original EFL Level, Item 706 –Date must be within or equal to 1 year anniversary of Date of First Youth Service Success for subsequent years –In the PRESENT year, the Educational Functioning Level > the EFL for the previous year

20 NO Literacy or Numeracy Gain – 1 st Yr. Has increased one EFL within 1 year of First Youth Service Date? NO YES Excluded YES Numerator Denominator 20 YES Is Out-of-School? YES Is Basic Skills Deficient? Is Youth? NO

21 First Year Notes Initial test must be given within 60 days of Date of First Youth Service. If a previous assessment was conducted (i.e. by a partner program) within six months prior to Date of First Youth Service, then that test score can be used. Participant is included in the measure even if they exit prior to end of the first year.

22 1 st Program Youth Program Post-test must be given 1 1 Exit can occur at anytime, but post-test must be given prior to exit 2 Post-test must be given prior to or on the anniversary date Date of Participation Anniversary Date 2 Date of First Youth Service Pre-test given within 6 months can be used* Pre-test must be given within 60 days 1 year First Year

23 First Year Notes (cont.) Post-test can be anytime within the first year, but must be given prior to anniversary date. Failure to test within first year or prior to exit will result in a negative outcome If a participant exits for an exclusion (i.e. global exclusion) and is not a success, the participant is excluded from the measure.

24 Basic Literacy Skills Deficient Revisited R = Reading/Writing Proficiency M = Math Proficiency BSP = Basic Skills Proficiency Not R OR Not M = Not BSP Not R AND Not M = Not BSP Not R = Not BSP Not M = Not BSP R AND M = BSP

25 NO Literacy or Numeracy Gain – 2 nd Yr. & Each Year Thereafter Has increased one EFL within last year based on latest post-test score? NO YES Excluded YES Numerator Denominator 25 YES Has completed another full year based on Anniversary Date*? Is Basic Skills Deficient? NO *Anniversary date is based on Date of First Youth Service and Exit Date >= Anniversary Date

26 Second Year and Beyond Notes Test can be given anytime between anniversary dates For subsequent years in the program, if the participant exits prior to the anniversary of the participation year, then the participant is excluded from the measure whether or not the participant completed the appropriate post-test.

27 Post-test must be given 1 1 If exit occurs prior to anniversary date, participant is excluded from the measure 2 Post-test must be given prior to or on the anniversary date Anniversary Date 2 Anniversary Date Second Year and Beyond

28 Two Methods for Calculations Option A – States who are implementing in first year –The participant is included in the measure the moment the individual has an outcome – a successful post-test, exit, or the anniversary of the participation year Option B – Used by States in all subsequent years –Includes the participant in the measure only on the anniversary of the individuals participation year

29 1 st Program Youth Program * Option A ** Option B Date of Participation Anniversary Dates Date of First Youth Service Year 1 * Two Methods for Calculations Year 2 ** Date of Exit Common Participation Period 90 calendar days of no service

30 Example Participant A enters the program on 10/01/2006 Pre-test given on 10/02/2006 Takes post-test and increases 1 EFL on 10/05/2006 Option A –Success as of 10/05/2006, the date of EFL increase Option B –Success on 10/01/2007, the anniversary date of the Date of the First Youth Service

31 Exclusions Social Security Number – –Can impact the ability of states to receive administrative records from partner programs Technical and community college that use SSN as primary identifier Exit due to exclusion causes all outcomes to be lost –Participant had a success in the 1 st year and then they were excluded in the 2 nd year Options 1.Positive or negative outcome would be lost 2.No impact to real-time outcomes More clarification to come

32 Test Requirements

33 Why Test? Test or assessment has three purposes 1 : –It provides diagnostic information (formative assessments) –Evaluates student progress (summative assessment) –Evaluates the overall performance of an entity (e.g., class, program, state). 1 Developing Performance Assessments

34 Testing Requirements Pre-test – A test used to assess a participant’s basic literacy skills, which is administered to a participant up to six months prior to the date of participation, if such pre-test scores are available, or within 60 days following the date of first youth service. Post-test – A test administered to a participant at regular intervals during the program.

35 Testing Requirements All assessment tests must crosswalk to the NRS Programs must use test outline in Attachment C – NRS EFL Table States, grantees, or contractors are not required to use the same assessment tool throughout their jurisdictions.

36 Testing Requirements Programs must adhere to the following in choosing an assessment tool: 1.The same assessment tool is administered to the participant for pre-testing and post-testing; 2.The assessment tool and its scores must crosswalk directly to the educational functioning levels so that educational gains can be reported in terms of increase in one or more ABE or ESL levels; and 3.Tests must be administered in a standardized manner throughout the jurisdiction (i.e., used consistently and reliably across programs and produce observable results).

37 Understanding NRS Table The NRS provides test benchmarks for EFL. Indicate how students functioning at each level would perform on the tests. Inclusion of tests does not imply that they are equivalent or the basis for assessment. Tests do not necessarily measure the same skills.

38 Testing Requirements Intervals –Participants must be post-tested at least once by the end of one year following the individual’s date of first youth program service. –If more than one assessment is administered after the initial test, the latest assessment should be used Individuals who remain basic skills deficient and continue to participate, must continue to receive basic skills remediation services.

39 Testing Youth with Disabilities When administering assessment tools, individuals with disabilities are to be provided with reasonable accommodations: –“[m]odifications or adjustments,” made on a case-by-case basis, “that enable a qualified individual with a disability... to receive aid, benefits, services, or training equal to that provided to qualified individuals without disabilities.”* *29 CFR 37.4

40 Testing Youth with Disabilities Reasonable Accommodations are to be provided in accordance with: –Section 188 of the WIA –Guidelines associated with the assessment test –State laws or policy Youth with disabilities are expected to achieve the same gains as other youth –Accommodations provide a “level playing field.”

41 Testing Youth with Disabilities For youth with one or more severe disabilities –The use of these testing instruments, even with appropriate accommodations, may not provide a valid or reliable evaluation of the literacy and numeracy skills Service providers and grant recipients may use alternate assessment tools to measure gains –State has deemed alternate assessments valid and reliable indicators of information –Comparable to the information provided through the educational functioning levels

42 Categories of Accommodations Presentation – Braille or oral reading of questions Response – having test taker point to response Setting – taken in small setting Timing/Schedule – extending testing times. State participation and accommodation policies for students with disabilities: 1999 update (Synthesis Report No. 33)

43 Documentation of Accommodation Diagnosis of the learning disability or disabilities or, diagnosis of ADHD Statement of how this disability substantially limits candidate’s activity Recommendation for accommodation(s) appropriate for the individual’s educational needs

44 Further Information ETA’s Performance and Results Website ETA’s Office of Workforce Investment – Youth Services National Reporting System for Adult Education Website

45 Further Information The Iowa Literacy Resource Center GED Testing Accommodations American Council on Education – GED Testing Accommodations &TEMPLATE=/CM/ContentDisplay.cfm&CONTENTID=12223 Synthesis Report No tml

46 Special Thanks Mathematica Policy Research, Inc. for their assistance in preparing technical documentation for this presentation.