1. 1 Adult ESL Literacy Impact Study: Description and Implications Larry Condelli Stephanie Cronen American Institutes for Research, USA LESLLA Fifth Annual Symposium Banff, AB, Canada September 29, 2009

2. 2 Overview of Presentation Summary of Purpose, Research Questions and Design Summary of Purpose, Research Questions and Design Present descriptive data on students and teachers Present descriptive data on students and teachers Discussion and implications of potential findings Discussion and implications of potential findings

3. 3 Evaluation of an ESL Literacy Intervention: Sam and Pat Structured language approach Structured language approach Adapted from Wilson Reading System Adapted from Wilson Reading System Never before evaluated for ESL Never before evaluated for ESL Literacy activities organized Literacy activities organized around basal reader/workbook Heavily phonics-based Heavily phonics-based

4. 4 Sam and Pat: Instructional Approach Direct instruction, transparent Direct instruction, transparent Rules explained, modeled, practiced Rules explained, modeled, practiced Controlled text, vocabulary and grammar Controlled text, vocabulary and grammar Words match phonics already learned Words match phonics already learned Sequential Sequential Easy to hard in defined steps Easy to hard in defined steps Multi-sensory, kinesthetic Multi-sensory, kinesthetic ESL instruction to support literacy ESL instruction to support literacy

5. 5 Sam and Pat: Literacy and Language Skills Covered Phonics for reading and writing Phonics for reading and writing Sight words Sight words Oral reading for Oral reading for accuracy and fluency accuracy and fluency Reading comprehension Reading comprehension Vocabulary Vocabulary Speaking and listening Speaking and listening Grammar Grammar

6. 6 Structured Two Part Lesson Part 1: Part 1: Rereading familiar text Rereading familiar text Pre-reading (picture story) Pre-reading (picture story) Letters and phonics Letters and phonics Vocabulary and grammar Vocabulary and grammar Sight words Sight words Conversation Conversation Part 2: Part 2: Review of Part 1 content Read story Written exercises

7. 7 Research Questions How effective is instruction using the intervention in improving the English reading, speaking and listening skills of low-literate adult ESL learners? How effective is instruction using the intervention in improving the English reading, speaking and listening skills of low-literate adult ESL learners? Is the intervention more effective for certain groups of students (e.g., language, literacy level)? Is the intervention more effective for certain groups of students (e.g., language, literacy level)? Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts? Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts?

8. 8 Study Design 10 adult ESOL centers across USA 10 adult ESOL centers across USA Paired intervention and “normal” ESL literacy classes (34 total) Paired intervention and “normal” ESL literacy classes (34 total) Random assignment of students and teachers Random assignment of students and teachers Minimum 5 hours/week 10-12 weeks instruction with approach Minimum 5 hours/week 10-12 weeks instruction with approach Other instruction also Other instruction also provided (5-10 hours/week) Each class conducted Each class conducted twice over a year twice over a year

9. 9 Student Flow in the Study 1. Student applies to center 2. Assessed for NLL 3. Recruited into study 4. Gives informed consent 5. Random assignment to class 6. Pretests administered 7. Instruction 8. Posttests administered Intake NL Literacy Post-test Random Assignment Recruited into Study Informed consent Pre-test Instruction

10. 10 Students Assessments Phonics and decoding Phonics and decoding Word attack Word attack Letter/Word ID Letter/Word ID Reading comprehension Reading comprehension Vocabulary (ROWPVT) Vocabulary (ROWPVT) Listening, oral expression Listening, oral expression

11. 11 Sam And Pat Teachers in the Study All teachers randomly assigned All teachers randomly assigned 3-day teacher training on curriculum 3-day teacher training on curriculum Follow-up visits by trainers Follow-up visits by trainers Classroom observations Classroom observations to monitor fidelity Refresher webinar at Refresher webinar at start of second term

12. 12 Classroom Observations Literacy development instruction: Literacy development instruction: Pre-literacy Pre-literacy Phonics Phonics Fluency Fluency Reading strategies & comprehension Reading strategies & comprehension Writing Writing ESL Instruction: ESL Instruction: Oral language development Grammar, etc. Vocabulary Socio-cultural knowledge Functional literacy

13. 13 Students in the Study “True” Literacy (LESLLA) Little or no literacy in native language Little or no literacy in native language Limited oral English Limited oral English Education: 0-6 years Education: 0-6 years Languages: Languages: Haitian-Creole Haitian-Creole Spanish Spanish Burmese Burmese Others Others

14. 14 Students in the Study Non-Roman Alphabet Literate Some Literacy in native language with non- Roman script Some Literacy in native language with non- Roman script Mean education: 6 and more years Mean education: 6 and more years Limited oral English Limited oral English Languages: Languages: Armenian Armenian Arabic Arabic Farsi Farsi Chinese Chinese Others Others

15. Description of Sites Ten sites participated: Ten sites participated: 1 site in San Francisco CA area 1 site in San Francisco CA area 1 site in Los Angeles CA area 1 site in Los Angeles CA area 3 sites in Miami FL area 3 sites in Miami FL area 2 sites in Chicago IL area 2 sites in Chicago IL area 3 sites in Houston TX area 3 sites in Houston TX area Sites included adult ed centers, high schools, and community centers Sites included adult ed centers, high schools, and community centers Type of programs included…?? Type of programs included…?? 15

16. Description of Students 1,344 students participated for one term 1,344 students participated for one term They attended a range of low-level classes in the study, with ESOL Literacy being the most common: They attended a range of low-level classes in the study, with ESOL Literacy being the most common: ESOL Literacy ESOL Literacy Low Beginning ESOL Low Beginning ESOL Beginning ESOL (combined w/Literacy) Beginning ESOL (combined w/Literacy) Students in the sample had been in the U.S. for an average of 4.7 years Students in the sample had been in the U.S. for an average of 4.7 years 16

17. Description of Students Spanish was the most common language, followed by Armenian Spanish was the most common language, followed by Armenian 17

18. Description of Students, Cont. Average years of education Average years of education 6.7 years of education outside the U.S. 6.7 years of education outside the U.S. 7.7 years of education total 7.7 years of education total Why so high? Why so high? As a group, only Haitian students were “true literacy” As a group, only Haitian students were “true literacy” Spanish-speakers ranged from true literacy to beginning ESL level Spanish-speakers ranged from true literacy to beginning ESL level Non-Roman alphabet students ranged from beginning ESL level to much higher Non-Roman alphabet students ranged from beginning ESL level to much higher 18

19. Description of Students, Cont. Over half (58%) of Ss were female Over half (58%) of Ss were female Student ages ranged from 18 to 84 Student ages ranged from 18 to 84 19

20. Teachers in the Study 33 teachers participated in the study in Fall, and 31 participated in Winter/Spring 33 teachers participated in the study in Fall, and 31 participated in Winter/Spring 1 class pair/2 teachers were dropped before the second term due to insufficient numbers of new students enrolling in their classes 1 class pair/2 teachers were dropped before the second term due to insufficient numbers of new students enrolling in their classes 1 teacher taught 2 treatment classes both terms 1 teacher taught 2 treatment classes both terms Teachers had 7.5 years of experience teaching adult ESOL, on average Teachers had 7.5 years of experience teaching adult ESOL, on average 5.3 years of experience teaching literacy-level 5.3 years of experience teaching literacy-level 20

21. Teachers in the Study The majority of teachers (85%) had some form of certification The majority of teachers (85%) had some form of certification 21

22. Teachers in the Study Most teachers’ highest degree received was either a Bachelor’s (42%) or a Master’s (48%) Most teachers’ highest degree received was either a Bachelor’s (42%) or a Master’s (48%) 61% of teachers were male 61% of teachers were male Race/ethnicity varied Race/ethnicity varied 40% white, nonHispanic 40% white, nonHispanic 30% Black or African American 30% Black or African American 24% Hispanic or Latino 24% Hispanic or Latino 6% Asian or American Indian 6% Asian or American Indian 22

23. Teachers in the Study Most teachers spoke a language other than English Most teachers spoke a language other than English 23

24. Findings and Impacts…. 24

25. SORRY!!! We’re not sure yet (and couldn’t tell you now anyway….) 25

26. 26 Observations and Discussion Implications of Possible Findings for Practice and Research for LESLLA Students

27. 27 Implications for LESLLA Practice Let’s talk about the 3 most likely results scenarios: Let’s talk about the 3 most likely results scenarios: Positive impact on decoding; no other impacts Positive impact on decoding; no other impacts No impacts on any outcomes No impacts on any outcomes Negative impacts on some ESL outcomes Negative impacts on some ESL outcomes

28. 28 Implications for LESLLA Practice Positive impact on decoding; no other impacts. Given Sam & Pat strong phonics focus this is the expected and most likely scenario. Positive impact on decoding; no other impacts. Given Sam & Pat strong phonics focus this is the expected and most likely scenario. But is improving only decoding after 3 months of instruction enough to warrant using it? But is improving only decoding after 3 months of instruction enough to warrant using it? Would the impact differ by type of student? Would the impact differ by type of student? What would the implications of these results be for teachers considering Sam & Pat or a similar approach? What would the implications of these results be for teachers considering Sam & Pat or a similar approach?

29. 29 Implications for LESLLA Practice Gains for all but no differences between groups Gains for all but no differences between groups Sam & Pat is no more effective than other types of ESL literacy instruction Sam & Pat is no more effective than other types of ESL literacy instruction Under what conditions, then, would Sam & Pat be a good choice for instruction? Under what conditions, then, would Sam & Pat be a good choice for instruction? What are some potential explanations for these results from your perspective? What are some potential explanations for these results from your perspective?

30. 30 Implications for LESLLA Practice Negative impacts on some ESL outcomes Negative impacts on some ESL outcomes Some teachers expressed concern about taking 5 hours of instruction away from ESL activities, so we could see a negative impact on vocab or listening tests Some teachers expressed concern about taking 5 hours of instruction away from ESL activities, so we could see a negative impact on vocab or listening tests How likely is this scenario? How likely is this scenario? What other explanations could we investigate in our analyses? What other explanations could we investigate in our analyses?

31. 31 Context of LESLLA Research The study took place in a challenging environment that makes having/finding an impact difficult, but that represents reality The study took place in a challenging environment that makes having/finding an impact difficult, but that represents reality Low exposure and instructional time in adult ESL class Low exposure and instructional time in adult ESL class Short class duration Short class duration Limited instructional time and irregular attendance Limited instructional time and irregular attendance Training teachers Training teachers Lack of specialized training in literacy Lack of specialized training in literacy Short training time available Short training time available

32. 32 Thank you! Contacts Contacts lcondelli@air.org lcondelli@air.org scronen@air.org scronen@air.org Enjoy Banff and LESLLA Enjoy Banff and LESLLA