Larry Condelli Stephanie Cronen American Institutes for Research, USA LESLLA Sixth Annual Symposium Cologne, Germany August 26,

Overview of Presentation Summary of Study Purpose and Design Impact of the Intervention Content of Instruction Student Attendance Discussion and Implications of Findings 2

Evaluation of an ESL Literacy Intervention: Sam and Pat Structured language approach Adapted from Wilson Reading System Never before evaluated for ESL Literacy activities organized around basal reader/workbook Heavily phonics-based 3

Sam and Pat: Instructional Approach Direct instruction, transparent Rules explained, modeled, practiced Controlled text, vocabulary and grammar Words match phonics already learned Sequential Easy to hard in defined steps ESL instruction to support literacy 4

Sam and Pat: Literacy and Language Skills Covered Phonics for reading and writing Sight words Oral reading for accuracy and fluency Reading comprehension Vocabulary Speaking and listening Grammar 5

Research Questions How effective is instruction using the intervention in improving the English reading, speaking and listening skills of low- literate adult ESL learners? Is the intervention more effective for certain groups of students (e.g., language, literacy level)? Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts? 6

Study Design 10 adult ESOL centers across USA Paired intervention and “normal” ESL literacy classes (34 total) Random assignment of students and teachers 1,344 students participated for one term Minimum 5 hours/week weeks instruction with approach Other instruction also provided Each class conducted twice over a year 7

Student Flow in the Study 1. Student applies to center 2. Assessed for NLL 3. Recruited into study 4. Gives informed consent 5. Random assignment to class 6. Pretests administered 7. Instruction 8. Posttests administered 8 Intake NL Literacy Post-test Random Assignment Recruited into Study Informed consent Pre-test Instruction

Students Assessments Phonics and decoding Word attack Letter/Word ID Reading comprehension Vocabulary (ROWPVT) Listening, oral expression 9

Sam And Pat Teachers in the Study All teachers randomly assigned 3-day teacher training on curriculum Follow-up visits by trainers Classroom observations to monitor fidelity Refresher webinar at start of second term 10

Students in the Study: “True” Literacy (LESLLA) Little or no literacy in native language Limited oral English Education: 0-3 years Languages: Haitian-Creole Spanish Burmese Others 11

Students in the Study: Non-Roman Alphabet Literate Some Literacy in native language with non-Roman script Mean education: 6 and more years Limited oral English Languages: Armenian Arabic Farsi Chinese Others 12

Student Test Results 13

Main Impact Analyses OutcomeTreatmentControlDifferenceEffect SizeP-value WJID WJWA SARA Dec WJPC OWLS ROWPVT WJPVT Sample size: 1137 students 14 No significant impacts on literacy or language outcomes for full sample (below) and subsamples But all students gained on pre-post tests

Pre-post Differences, All Students OutcomeMean Pretest Mean Posttest DifferenceP-value WJID * WJWA * WJPC * OWLS * ROWPVT * N=1,113 Sam & Pat=567 Control=546 15

Lack of Impact of Sam and Pat The study took place in a challenging environment that makes having/finding an impact difficult, but that represents reality Low exposure and instructional time in adult ESL class Short class duration Limited instructional time and irregular attendance Training teachers Lack of specialized training in literacy Short training time available 16

Instruction: What is taught? 17

Classroom Observations Literacy development instruction: Pre-literacy Phonics Fluency Reading strategies & comprehension Writing ESL Instruction: Oral language development Grammar, etc. Vocabulary Socio-cultural knowledge Functional literacy 18

Instructional Content by Interval (%) Content Sam and Pat Mean Control MeanDifferenceP-value Reading Development ESL Acquisition Other Content Link to Outside Use of NL Sample Size Intervals Observations

Instruction in Literacy Classes In control (“normal”) classes, little literacy taught- Why? Importance of oral language for daily life Student expressed needs Lack of teaching on how to teach LESLLA students More diversity of instruction in control classes More NL use and bringing in outside Less constrained by curriculum and study needs 20

Attendance 21

Attendance and Outcomes OutcomeRegression Coefficient P=value WJID WJWA WJPC OWLS ROWPVT WJPVT Mean attendance hours (p= n.s.) Sam & Pat : 79.4Control: 71.9 N=1,137, Sam & Pat=587, Control=557 22

Effect of Attendance on Test Scores Instruction and outcomes correlated Instruction appears to have an effect on learning Relationship is weak Larger effect on reading outcomes May mean literacy gains more sensitive to gain, may be testing artifact 23

Implications of Possible Findings for Practice and Research for LESLLA Students 24

Implications for LESLLA Practice Sam & Pat is no more effective than other types of ESL literacy instruction Under what conditions, then, would Sam & Pat be a good choice for instruction? Attendance relates to instruction, weakly Is this what you expect? How could it be increased? Teachers of literacy classes teach little literacy How can we improve this --through curriculum and teacher professional development? What are some potential explanations for these results from your perspective? 25

Thank you! Contacts Enjoy Koeln and LESLLA 26