Strengths & Weaknesses of IALS as an Assessment Framework: A Longitudinal Perspective Stephen Reder Portland State University I IALS, Its Meaning and Impact for Policy and Practice The Centre for Literacy Fall Institute Banff, Alberta - October 24, 2011

Where I’m Coming From Literacy researcher interested in how the IALS Framework affects policies and programs in adult basic education Some of my researcher and practitioner colleagues believe that IALS is not relevant to their programs and students Although I believe there is a disconnect between the IALS Framework and the needs of adult education, we can fix it !

Strengths of IALS Framework (for adult education) Contextualizes literacy in terms of important social & economic indicators Provides compelling snapshots of adult literacy at given points in time Builds technically sound assessment instruments and procedures Pays careful attention to maintaining comparability of large-scale assessments over time and space

Weaknesses of IALS Framework (for adult education) Supports increased literacy proficiency as a policy goal but… …does not provide useful tools for assessing program impact on literacy …is of limited use for short-term accountability regimes & program improvement efforts

The Longitudinal Study of Adult Learning (LSAL) funded by U.S. Department of Education and National Institute for Literacy Portland State University

Longitudinal Study of Adult Learning Decade-long panel study of Portland high school dropouts, age at the beginning of the study Representative sample of ~1,000 drawn from local rather than national population of dropouts Periodic in-home interviews and literacy assessments and SSN-linked administrative data (with individuals’ permission) Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, literacy practices & technology use over time Smaller-scale qualitative components

LSAL Timeline wave 1 wave – 1999 wave 2 wave – 2000 wave 3 wave – 2001 wave 4 wave – 2003 wave 5 wave – 2005 wave 6 wave – 2007

Literacy Measures in LSAL Measures of Proficiency Repeated measures of TALS Document Literacy SSN-matched GED test scores Measures of Practices Repeated measures of literacy practices Repeated self-reported changes from wave-to-wave in reading, writing and math Measures of Component Skills Oral vocabulary Word recognition Fluency Holistic writing

Some LSAL Demographics Average age is 28 (at Wave 1) 50 % female and male 35 % minority 9 % foreign-born 9 % foreign-born 34 % live in poverty 29 % report a learning disability 34 % took special education Broad range of assessed basic skills

Up Close and Personal Looking at Things Seen in IALS in the LSAL Panel Study Ageing Ageing Proficiency & economic indicators Proficiency & economic indicators

NALS Document Literacy by Age (1992 national data corrected for education & disabilities)

UK Document Literacy by Age (IALS data, corrected for education)

Employment and Earnings Data Using individuals’ quarterly administrative employment & wage data reported by employers for unemployment insurance Covers more than 90% of labor activity Examining (7 years / 28 qrtrs) Major recession in Portland

Wages by Proficiency,

Earnings by Proficiency Growth

So What’s The Problem? Longitudinal data seem consistent with IALS snapshots and allow us to look more closely at the individual level of the large-scale phenomena Controlled comparisons of adults who participate in basic skills programs with similar adults who do not participate show there is no significant impact of programs on proficiency one or two years following participation Does this mean programs are not effective? NO! But it illustrates the practical disconnect of IALS from programs

Repeated Measures of Engagement in Literacy Practices

Program Impact Literacy proficiency growth over relatively short periods of time is not affected by program participation Accountability data show proficiency gains but do not contrast participants’ gains with those of comparable non-participants; LSAL indicates their gains are equivalent Growth in literacy practices over short time periods is, on the other hand, directly impacted by programs These findings are consistent with cross-sectional research (e.g., Smith & Sheehan-Holt) and classroom studies (e.g., Purcell-Gates, Jacobson & Degener) Disconnect: Proficiency measures do not reflect the impact that programs have in the short-term nor can support evidence-based program improvement processes

Repairing the Disconnect: Practice-Engagement Theory Predicts that engagement in literacy practices over time leads to proficiency growth Proposed by Reder from ethnographic work and by Smith & colleagues in analyses of NALS LSAL confirms through longitudinal modeling: Adults with the same proficiency level and a higher level of engagement in literacy practices at one point in time have significantly higher levels of literacy proficiency 6 years later Adults with the same proficiency level and a higher level of engagement in literacy practices at one point in time have significantly higher levels of literacy proficiency 6 years later

What Should We Do? Stop relying on only IALS (or other) proficiency measures for short-term accountability & program improvement Use literacy practices measures along with proficiency measures in large-scale assessments and for short-term program accountability and improvement Use proficiency measures as long-term (5-6 year) follow-up outcome measures

Contact information Steve Reder Professor, Chair – Department of Applied Linguistics Portland State University (503) For more information and references to the research covered in the Powerpoint presentation, please see: Longitudinal Study of Adult Learning Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development