1. Evaluating Outcomes of the English for Speakers of Other Languages (ESOL) Program in Secondary Schools: Methodological Advance and Strategy to Incorporate a Quasi-experimental Design Helen Wang, Ph.D. & Shahpar Modarresi, Ph.D. Montgomery County Public Schools Rockville, Maryland Rockville, Maryland Helen_wangyq@mcpsmd.org http://www.montgomeryschoolsmd.org/departments/ sharedaccountability/

2. Program Overview Purpose: Help ESOL students function linguistically and culturally in regular classrooms Participants: Participants: Students speaking a language other than standard American English Length: Throughout school year Length: Throughout school year Curricula: Curricula: Rigorous standard-based ESOL curriculum Improvement Outcomes: Improvement and Attainment of English language proficiency and achievement in content areas

3. Evaluation Questions 1.To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency (the percentage of students gaining a 15-point or more increase annually in overall scores of Language Assessment System Links or LAS-Links)? 2. To what extent did secondary ESOL students attain English language proficiency toward the state target (the percentage of students achieving the advanced level in overall LAS-Links and the proficient or advanced level in subtests)? 3. How did Grade 8 ESOL students perform on Measures of Academic Progress– Reading (MAP-R)? 4. How did Grade 8 ESOL students perform on Maryland School Assessments (MSAs) in reading and mathematics? 5. Were students receiving higher-level ESOL instruction more likely to pass High School Assessments (HSAs) than those receiving lower-level ESOL instruction?

4. Evaluation Design Incorporated Quasi-experimental design Specified samples across years to examine trends of student performance Specified outcome measures and analytical procedures pertaining to research questions

5. Outcome Measures: Standardized Assessments Language Assessment System Links (LAS-Links) overall and subtest scale scores and proficient levels Measures of Academic Progress Assessment in Reading Rasch Unit (MAP-R RIT) scores Maryland School Assessment (MSA) reading and mathematics scale scores High School Assessment (HSA) passing status in algebra, English, biology, and government

6. Study Samples for Evaluation Questions Q1: Grade 6-12 ESOL students with LAS-Links scores in any two adjacent years of most recent three school years. Q2: Grade 6-12 ESOL students with LAS-Links scores in the most recent two school years Q3: Grade 8 students with MAP-R RIT scores in most recent three school years Q4: Grade 8 students with MSA reading or math scale scores in most recent three school years Q5: English learners from two most recent graduating classes who received ESOL instruction at the time they maximized an HSA score.

7. Analyses for Question 1: Analyses for Question 1: To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency? Performed descriptive procedures on LAS-Links for the three selected years: Computed numbers and percentages of students who gained 15 or more points on the overall score from the previous year, by grade level, ESOL instructional level, demographics, and service receipt measure. Compared the percentages with state targets which increased over the years

8. Analyses for Question 2: Analyses for Question 2: To what extent did secondary ESOL students attain English language proficiency toward the state target? Performed descriptive procedures on LAS-Links for the two selected years: Computed numbers and percentages of students who achieved the advanced level overall and proficient or advanced level for speaking, listening, reading, and writing subtests, by grade level, ESOL instructional level, demographics, and service rec Compared the percentages with state targets which increased over the years

9. Analyses for Questions 3 and 4: 3. Analyses for Questions 3 and 4: 3. How did Grade 8 ESOL students perform on MAP-R? 4. How did Grade 8 ESOL students perform on MSA reading and math? Performed analyses of covariance (ANCOVA) for cross-year comparisons on MAP-R RIT scores and MSA reading and mathematics scale scores for the three selected years; c Performed analyses of covariance (ANCOVA) for cross-year comparisons on MAP-R RIT scores and MSA reading and mathematics scale scores for the three selected years; computed and used propensity scores as a covariate 20072008 cohort 2007─2008 cohort(Baseline) 20082009 cohort 2008─2009 cohort 20072008 cohort 2007─2008 cohort(Baseline) 20092010 cohort 2009─2010 cohort Adjusted Group Mean Scores Adjusted Group Mean Difference (p value) Adjusted Group Mean Difference (p value) Effect Size

10. Analyses for Question 5: Analyses for Question 5: How did Grade 8 ESOL students perform on MSA reading and math? Performed Logistic regressions to compare likelihoods of passing HSA subjects (algebra, English, biology, & government) by ESOL instructional levels for the two graduating classes combined, controlling for background variables Students at Higher ESOL Instructional Level Students at Lower ESOL Instructional Level Odds of Passing an HSA Odds Ratio (Comparing Probabilities of Passing an HSA) P Value and Effect Size

11. Findings for Question 1: Findings for Question 1: To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency ? Most secondary grades and subgroups made state-required annual progress in English acquisition (percentage of students having a 15-point or more annual increase in LAS-Links overall scores). Most secondary grades and subgroups made state-required annual progress in English acquisition (percentage of students having a 15-point or more annual increase in LAS-Links overall scores). Grade 9 and special education students in all grades were most likely to fall below the state target. Grade 9 and special education students in all grades were most likely to fall below the state target. More lower-level ESOL students made the annual progress in English language proficiency than higher-level ESOL students did. More lower-level ESOL students made the annual progress in English language proficiency than higher-level ESOL students did. Differences in annual progress among racial/ethnic and service groups became smaller, negligible, none, or reversed across cohorts in most secondary grades. Differences in annual progress among racial/ethnic and service groups became smaller, negligible, none, or reversed across cohorts in most secondary grades.

12. Findings for Question 2: Findings for Question 2: To what extent did secondary ESOL students attain English language proficiency toward the state target? All secondary grades and most subgroups attained the state- required English proficient level (percentage of students at the advanced level in overall LAS-Links and the proficient or advanced level in all subtests). All secondary grades and most subgroups attained the state- required English proficient level (percentage of students at the advanced level in overall LAS-Links and the proficient or advanced level in all subtests). Special education students in most grades and some racial/ethnic groups in Grade 9 did not meet the state target. Special education students in most grades and some racial/ethnic groups in Grade 9 did not meet the state target. More higher-level ESOL students attained the advanced level in English language proficiency than lower-level ESOL students did. More higher-level ESOL students attained the advanced level in English language proficiency than lower-level ESOL students did. White students led other racial/ethnic groups in attaining the target proficient level in most grades White students led other racial/ethnic groups in attaining the target proficient level in most grades

13. Findings for Question 3: Findings for Question 3: How did Grade 8 ESOL students perform on MAP-R? Yearly achievement changes were found insignificant for Grade 8 MAP-R RIT scores.

14. Findings for Question 4: Findings for Question 4: How did Grade 8 ESOL students perform on MSA reading and mathematics? Yearly achievement growth was found significant for MSA reading and mathematics between cohorts 2008 and 2009. Outcome Measure Adjusted MeansYearly Achievement 2007–2008 cohort (Baseline) 2008–2009 cohort Mean DifferenceStd. Error MeanN N MSA Reading 377.25350382.353475.1*2.07 MSA Math 392.26483399.073526.80*2.58 Note. For both MSA reading and mathematics: p <.05 and effect size =.18 Adjusted Means and Mean Differences Comparing 2007–2008 with 2008–2009 MSA

15. Findings for Question 4 (continued): Findings for Question 4 (continued): How did Grade 8 ESOL students perform on MSA reading and mathematics? Findings about achievement growth for MSA reading and mathematics were mixed between cohorts 2008 and 2010 Adjusted Means and Mean Differences Comparing 2007–2008 with 2009–2010 MSA Adjusted Means and Mean Differences Comparing 2007–2008 with 2009–2010 MSA Outcome Measure Adjusted MeansYearly Achievement 2007–2008 cohort (Baseline)2009–2010 cohort Mean DifferenceStd. Error MeanN N MSA Reading 377.25350383.353226.29*2.37 MSA Math 391.87483389.53444-2.342.60 Note. For MSA reading: p.05 & ES = -0.06

16. Findings for Question 5: Findings for Question 5: Were students receiving higher-level ESOL instruction more likely to pass HSA subjects than those receiving lower-level ESOL instruction? Students who received higher-level ESOL instruction had a significantly higher probability of passing an HSA than their lower-level peers, across all HSA subjects. NOdds RatioP value Effect Size HSA Biology Instructional Level Level 5 615 2.120.0000.41 Level 4 373 Level 4 373 1.990.0090.38 Level 3 79 Level 5 615 2.420.0000.49 Levels 4+3 452 Odds of Passing HSA Biology by ESOL Instructional Level