Different Skills? Identifying Differentially Effective Teachers of English Language Learners Ben Master, Susanna Loeb, Camille Whitney, James Wyckoff 5 rd Annual CALDER Conference January 27 th, 2012

English Language Learners More than 5 million public school students (~10%) are currently designated as ELLs Typically take 3-5 years to attain oral English proficiency Low overall achievement. e.g. 22% in lowest percentile compared with 3% in highest in 2011 math

Teaching of ELLs Many schools, districts and states seek to improve the outcomes for ELLs in particular Teachers often provide few accommodations for ELLS Teacher preparation programs often provide little opportunity to learn about teaching ELLs

Teachers Clearly important for student achievement Some evidence that effective teaching can be observed Some evidence that effective teachers, on average have differential knowledge (general and specific) Evidence that teachers improve with experience Some evidence that pre-service and in-service programs can improve teaching

Motivation Little corresponding evidence on what teacher characteristics or training are differentially or specifically beneficial for English language learners

Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs?

own test performanceteaching experience

Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs?

teaching experience with ELLs pre-service preparation addressing skills for teaching ELLs in-service preparation addressing skills for teaching ELLs ESL Certification

Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? 3. Do theorized teacher characteristics predict differential effectiveness with ELLs?

ability to speak Spanish fluently, either native or learned

Today’s study 1. Do teacher characteristics that predict growth for non-ELLs also predict growth for ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? 3. Do theorized teacher characteristics predict differential effectiveness with ELLs?

New York City (NYC) public school system through Administrative data: student exams and student characteristics; teacher background characteristics and certification information; school characteristics Survey of first year teachers in 2005: pre-service and in-service experiences, additional background characteristics Data

SOME DESCRIPTIVES

Percentage of students who are ELLs and standardized math test scores in New York City, by year Percentage of students who are ELLs11.30% Percentage of ELLs in each grade of study Grade % Grade % Grade % Grade % Grade % Standardized math test scores-0.63

Race/ethnicity and Free or reduced price lunch, by ELL status ELLsNon-ELLs Race/ethnicity White7%15% Black6%36% Hispanic70%35% Asian18%13% Other race/ethnicity0%1% Free or reduced price lunch76%68% Home language is English0%63%

Basic distribution of ELLs across classrooms % ELLs in the class Mean across all classrooms11 Percentiles 25th0 50th3 75th9 90th40 95th94

Characteristics of teachers serving ELL and non-ELL students, district-wide % Taught by Teachers with 1 or 2 Years of Experience Initial LAST Scores (std. dev.) Mean across NYC (30.2) Mean for ELLs (33.8) Mean for non-ELLs (29.6)

Responses and # of respondents for first year teacher survey questions of interest % YesTotal N 1.Fluency in languages other than English: % that self-identified as fluent in Spanish? Pre-service opportunity to learn instructional strategies for teaching ELLs: % that “explored in some depth” or “extensively”? By mid-year, # of hours of in-service PD focused on ELL instruction: % reporting “>9,” “>17,” or “>33 hours” so far this year?

Approach Option 1 Compare the learning of ELLs to non-ELLs with one type of teacher vs. another Sorting of teachers and students into school may bias estimates (type 1 teachers sort to better schools) Sorting of teachers to students within schools may bias estimates (easier students sort to type 1 teachers) not particularly believable

Option 2 Compare the learning of ELL students with one type of teacher to others within the same school Sorting of teachers and students into school largely accounted for Sorting of teachers to students within schools may bias estimates (easier students sort to type 1 teachers) Better

Option 3 Compare the learning of ELL students to learning of non-ELL students within the same classroom Sorting of teachers and students into school accounted for Sorting of teachers to students within schools taken care Better: but can’t see overall effect, just differential

1. Do teacher characteristics that predict growth for non-ELLs also predict growth for ELLs? teachers' own test performance teachers' teaching experience Results

Approach 2 (within school): About the same effect of experience Effect of tests only for non-ELLs ELLsNon-ELLs initially failed LAST *** 2 nd year teaching in NYC 0.056***0.057*** 3 rd year teaching in NYC 0.091***0.076*** 4 th year teaching in NYC 0.089***0.093*** 6 th year teaching in NYC 0.086***0.097*** 7 th year teaching in NYC 0.079***0.094***

Approach 3 (within teacher): No evident difference ELL versus Non-ELL Achievement Gap failed LAST exam x ELLs nd year x ELLs rd year x ELLs th year x ELLs th year x ELLs th year x ELLs th year x ELLs-0.005

2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? past teaching experience with ELLs pre-service addressing skills for ELL teaching in-service addressing skills for ELL teaching ESL certification

Approach 2 (within school): Experience with ELLs substantially more important for ELLs ELLs All2 nd year3+ years >6 ELLs last year0.024**0.069**0.015~ Non-ELLs All2 nd year3+ years >6 ELLs last year0.010* ~

Approach 3 (within teacher): Similar Results ELL versus Non-ELL Achievement Gap 2nd year nd year x ELL-0.012~ >6 ELLs in prior year >6 ELLs in prior year x ELLs0.031**

Approach 2 (within school): Training substantially more important for ELLs ELLs ELL-specific pre-service0.090*0.093*0.027 ELL-specific in-service ELL-specific PD (in ‘05) 0.294*** Non-ELLs ELL-specific pre-service ELL-specific in-service-0.120*** ELL-specific PD (in ‘05)-0.112* all differences significant **

Approach 3 (within teacher): Similar PreIn (same year) ELL pre-service X ELL0.091* ELL in-service X ELL ELL PD (in ‘05) X ELL 0.226* all differences significant **

Approach 3 (within teacher): Similar all differences significant ** All teachersNovices (<=3yrs) Any ESL Cert X ELL ** All teachersNovices (<=3yrs) ELL Any ESL Certification0.057**0.069* Non-ELL Any ESL Certification ~ Approach 2 (within school): ESL Certification more Important for ELLs

3. Do theorized teacher characteristics predict differential effectiveness with ELLs? ability to speak Spanish fluently

Approach 2 (within school): No Evidence of Effect of Language Hispanic ELLs All Other Students Native fluency in Spanish Non-native fluency in Spanish Approach 3 (within teacher): Similar, though positive Hispanic ELLs Native fluency in Spanish X Hispanic ELL Non-native in Spanish X Hispanic ELL0.034

Summary ELLDifferential Test performancenomaybe Experienceno Experience with ELLsyes Pre-sevice ELLyes In-service ELyes ESL certificationyes Spanish fluency?no

Conclusions Just a start Indicates Improvement with both experience and education Broadly - indicates differential effectiveness Blunt measures Imperfect identification Highlights importance of opportunities Highlights importance of matching

Different Skills? Identifying Differentially Effective Teachers of English Language Learners Ben Master, Susanna Loeb, Camille Whitney, James Wyckoff 5 rd Annual CALDER Conference January 27 th, 2012