1. Methodological Challenges in School Effectiveness Research
What’s the Matter?
Methodological Challenges in School Effectiveness Research

2. Methodological Challenges for School Effectiveness
Session Objectives
Why is school effectiveness hard?
Causality, complex structure, change, measurement ...
What models/methods are available?
Multi-level, structural equation, multiple membership, MCMC ...
Conclusions
Evidence about school effectiveness, future direction
Methodological Challenges for School Effectiveness

3. Great Scientific Advances
What ?
How ?
Why ?
School Effectiveness ( )
The Double Helix ( )
One Giant Step for
Mankind ( )
Coleman et al. ( 1966); (Strand 2007)
Methodological Challenges for School Effectiveness

4. School Performance is Dynamic
Parents want to know about future performance; school league tables record the past
Time-School-Pupil: a 3 level model
Effectiveness changes; so does the definition of “effective”
Leckie and Goldstein (2009)
Methodological Challenges for School Effectiveness

5. School Pupils are Dynamic
The education system is dynamic
43% of primary pupils move between schools
League tables credit achievement to last school
Needs multiple membership and cross-classification
Pro rata apportionment corrects school effects by 17%
Creemers, Kyriakides and Sammons (2010); Goldstein, Burgess and McConnell (2007); Leckie (2008)
Methodological Challenges for School Effectiveness

6. Attainment and Attenuation
Prior attainment is hugely important in CVA
Standard 2-level models omit measurement error
Unaccounted measurement error attenuates the estimated slope at the pupil level
Raudenbush and Bryk (2002); Woodhouse, Yang, Goldstein and Rasbash (1996); Hutchison (2007)
Methodological Challenges for School Effectiveness

7. Phantom Contextual Effects
The estimate at the school level is less attenuated
The CVA model has a different parameterization
Where
Differential attenuation = Phantom contextual effect
Woodhouse, Yang, Goldstein and Rasbash (1996); Harker and Tymms (2004)
Methodological Challenges for School Effectiveness

8. Measurement and School Effects
School effects are assessed using variance components
Phantom effects carry through to residual variation
We under-estimate VPC unless we omit the context
Methodological Challenges for School Effectiveness

9. Methodological Challenges for School Effectiveness
Poverty of Data
Low socio-economic status (SES) pupils make slower academic progress but SES is hard to collect
So DCSF uses free school meals (FSM)
FSM has a number of weaknesses
Only one component of SES
Dichotomous
Missing data
High mis-classification rates
Gorard (2009); Kounali, Goldstein and Lauder (2007)
Methodological Challenges for School Effectiveness

10. Measuring Dichotomously
Methodological Challenges for School Effectiveness

11. Measuring Dichotomously
... Badly
Expected
Value
Actual
Observed
16/17.2
40%
17.2%
16%
FSM=1
24%
.64
1.2%
60%
82.8%
FSM=0
24/82.8
58.8%
Kounali, Goldstein and Lauder (2007); Goldstein, Kounali, and Robinson (2007)
Methodological Challenges for School Effectiveness

12. Methodological Challenges for School Effectiveness
FSM and School Effects
Unreliability due to dichotomous measurement and mis-classification compound
CVA omits a contextual term for FSM
Only observing a fraction of FSM pupils may cause us to further overestimate school effects
Goldstein, Kounali, and Robinson (2007)
Methodological Challenges for School Effectiveness

13. Manifestations of the Phantom
The phantom effect manifests on compositional variables
“... differences in the composition of schools’ intake achievements ... appears to be a major driver of between- school differences in GCSE scores.”
“school level coefficients which show strong differential effects for prior achievement, FSM and SEN.”
Leckie and Goldstein (2009); Leckie (2008)
Methodological Challenges for School Effectiveness

14. Discriminating Among Schools
Discrimination between schools is (largely) a function of school size and VPC
Coleman (1966) estimated VPC at around 10%
“ ... adjusting for these [compositional] variables halves the between-school variance; the VPC drops from 10.4% to 7.2%” Leckie and Goldstein (2009)
To improve discrimination, we need to allow for measurement error
Methodological Challenges for School Effectiveness

15. Dealing with Unreliability
Methods to deal with unreliability need to :
Work in conjunction with complex, multi-level methods
Handle complex, multi-level error and categorical variables
Some competing methodologies:
Regression discontinuity
Multiple measures in MLSEM
Item Response Theory (IRT)
Sensitivity analysis approach (MCMC)
Luyten (2006); Luyten and Tymms (2010 ); Marsh et al. (2009); Fox (2004 ); Ecob and Goldstein (1983); Goldstein, Kounali, and Robinson (2007)
Methodological Challenges for School Effectiveness

16. Methodological Challenges for School Effectiveness
Summary / Conclusions
Phantom effects are ubiquitous in school effectiveness research
FSM is deeply flawed as a proxy for SES and makes interpretation of results very problematical
Merely including cluster-level aggregates will tend to understate school-level residual variance
SER needs complex ML methods in combination with sophisticated tools to model unreliability
Methodological Challenges for School Effectiveness

17. Methodological Challenges for School Effectiveness
Bibliography
Equality of Educational Opportunity (Coleman et al. 1966) Minority Ethnic Pupils in the Longitudinal Study of Young People in England (Strand 2007) The estimation of school effects (Raudenbush and Willms 1995) Methodological Advances in Educational Effectiveness Research (Creemers, Kyriakides and Sammons 2010) The limitations of using school league tables to inform school choice (Leckie and Goldstein 2009) Incorporating Student Mobility in Achievement Growth Modeling: A Cross-Classified Multiple Membership Growth Curve Model (Grady and Beretvas 2010) Modelling The Effect Of Pupil Mobility On School Differences In Educational Achievement (Goldstein, Burgess and McConnell 2007) Effects of Pupil Mobility and Neighbourhood on School Differences in Attainment (Leckie 2008) School Value Added Measures in England (Ray 2006) The Effects of Student Composition on School Outcomes (Harker and Tymms 2004) Modelling Response Error in School Effectiveness Research (Fox 2004) Multilevel, Fully Latent-variable Models of Contextual Effects ... (Marsh et al., 2009) Instrumental variable methods for the estimation of test score reliability (Ecob and Goldstein 1983) Modelling measurement errors and category misclassifications in multilevel models (Goldstein, Kounali, and Robinson 2007) An empirical assessment of the absolute effect of schooling: regression discontinuity applied to TIMSS-95 (Luyten 2006) The probity of free school meals as a proxy measure for disadvantage (Kounali, Goldstein and Lauder 2007) Serious doubts about school effectiveness (Gorard 2009)
Methodological Challenges for School Effectiveness