IALS Measurement Validity, Program Impact and Logic Models for Policy Development Stephen Reder Portland State University I IALS, Its Meaning and Impact for Policy and Practice The Centre for Literacy Fall Institute Banff, Alberta – October 25, 2011

The Longitudinal Study of Adult Learning (LSAL) funded by U.S. Department of Education and National Institute for Literacy Portland State University

Periods of Participation in Basic Skills Programs Since Leaving School

Self-Study in the LSAL Defined as “studying on your own to improve your reading, writing or math skills or prepare for the GED” Probes distinguished such self-study from activity conducted in school or as part of a basic skills or GED class In-depth qualitative interviews confirmed the validity of these self-reports

Self-Study by Literacy Proficiency (wave 1)

Self-Study: Skills “Practiced” (Wave 3)

Self-Study: Materials Used (Wave 3) By Wave 5, computer use for self-study rose to 51%

Self-Study and Participation (between leaving school and Wave 4) self- study & adult ed self- study only adult ed only neither 64% have self-studied 60% have taken adult ed

Self-Reported Change in Literacy Practices by Participation & Self-Study

Percent GED Attainment by Participation & Self-Study

Some Implications We need a broader construct of participation that Includes programs, self-study & other learning activities Includes programs, self-study & other learning activities Sees self-study in a continuum of provision strategies along with various types of classroom and tutoring programs Sees self-study in a continuum of provision strategies along with various types of classroom and tutoring programs Conceives of learners as actively deploying resources as well as programs delivering services Conceives of learners as actively deploying resources as well as programs delivering services

Some Implications (con’t) We need broader programmatic structures that support & coordinate different modes of participation Technologies may be particularly helpful here but need to provide more than just direct instruction (e.g., support goal setting & plan development, counselling, etc.)

Modeling Lifespan Literacy Development Program participation and self-study have positive, short-term effects on engagement in literacy practices Engagement in literacy practices over time has cumulative effects on the growth of literacy proficiency This leads to a broader concept of participation that involes both programs and self-study Self-study bridges between periods of program participation and helps broaden persistence of attendance into persistence of learning

Programs as Practice-Building Contexts Programs that build engagement in authentic literacy practices may lead to long-term proficiency growth that is not evident in short-term pre-post testing Programs that build engagement in authentic literacy practices may lead to long-term proficiency growth that is not evident in short-term pre-post testing

Programs as Facilitated Learning Contexts Programs often blend time in classroom instruction with time in learning centers Programs often blend time in classroom instruction with time in learning centers There may be a continuum of facilitated learning with classroom activities on one end and self-study on the other end, with facilitated use of learning centers and tutoring in-between There may be a continuum of facilitated learning with classroom activities on one end and self-study on the other end, with facilitated use of learning centers and tutoring in-between

Redefining Program, Participation and Persistence Focus on engagement in literacy practices rather than on persistence in programs Broaden definition of program to include facilitated engagement in literacy practices Technology has broader role here than just delivering distance learning

Effects of Instruction Strongest evidence of program impact: Controlled comparison of participants’ and non-participants’ gains Next strongest: Covariation of hours of instruction with observed gains My analysis of CASAS proficiency gains in several states’ NRS data indicates the gains attributable to instructional hours are greatly attenuated (if present at all) when test experience is controlled

Program Participation....is often fragmented by life circumstances..is better thought of as a busy intersection among life and learning trajectories rather than as a parking lot..should support students’ trajectories not just their activities in the intersections..is expanded by a learning plan that learners can follow and that services can be wrapped around

Learning Support Systems Adults need learning support systems that provide portable, personalized learning plans they can follow to reach their goals Learning support systems include a focus on building engagement in literacy practices Learning support systems provide flexible blends of online and offline local resources to support learning The technology supports learners directly as they follow a learning plan and indirectly by coordinating the work of those who support them Communities design and implement local learning support systems, utilizing the technology to facilitate collaboration and information-sharing among the various organizations working with learners

A New Logic Model for Program Impact Practice-engagement theory Busy intersection vs. parking lot Learning support system

Research Recommendations Improve ways of measuring engagement in literacy practices Conduct interventions that stimulate increased practice- engagement Need long-term longitudinal studies with multiple repeated literacy outcome measures Need long-term longitudinal studies with multiple repeated literacy outcome measures Note that National Reading Panel found little evidence indicating that programs which aim to increase independent reading in schoolchildren lead to increased proficiency Note that National Reading Panel found little evidence indicating that programs which aim to increase independent reading in schoolchildren lead to increased proficiency Examine literacy changes (including proficiency loss) in older adults and interventions to maintain literacy capabilities in older adults

Contact information Steve Reder Professor, Chair – Department of Applied Linguistics Portland State University (503) For more information and references to the research covered in the Powerpoint presentation, please see: Longitudinal Study of Adult Learning Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development