How Big was Response Bias in England to PISA 2003? John Micklewright & Sylke V. Schnepf July 2008

2 Motivation PISA – Programme of International Student Assessment Reports from OECD for 2003 exclude the UK due to perceived non-response bias in England Simon Briscoe, Economics Editor at The Financial Times: the exclusion is among the ‘Top 20’ recent threats to public confidence in official statistics in the UK Presentation draws on: Response Bias in England in PISA 2000 and 2003, DfES Research Report 771

3 PISA response rates (%): England

4 ‘Bias’ in what? Mean Variance % beneath a given threshold

5 Five groups of 15 year olds i.all pupils in England (less permitted exclusions from the target population) ii.all pupils in sampled schools (initial, 1 st repl., 2 nd repl.) iii.all pupils in responding schools iv.sampled pupils in responding schools v.responding pupils

6 What we do Show how (i) mean, (ii) variance, and (iii) % < thresholds of KS3 and KS4 scores change from group to group Estimate biases in these 3 measures of PISA scores in group v, focusing on problem of pupil response Look at both 2000 and 2003

7 KS4 v. PISA scores - respondents R = 0.78 KS4 score PISA maths score

8 Means and SDs of KS4 scores, 2003

9 % below or above KS4 thresholds, 2003

10 Creating response weights Model probability of pupil response for sampled pupils in responding schools (group iv) Prob(response) i = F(domestic test score i, other characteristics i ) Predicted probability of response KS4 score

11 Bias due to pupil response Extent of bias, calculated as number of PISA score points (PISA international mean = 500, international SD = 100 Compare with the estimated standard errors (SEs) of the mean and SD for England in 2003: SE (mean) = 2.9 (reading and maths) and 3.0 for science SE (SD) = 1.6 (reading and maths) and 1.7 for science

12 However… Adjusting the mean by the estimated bias of about 6 points moves England one place in the country rankings ‘Post-stratification’ weights are used in many surveys – they could be provided with the data for England Think of possible biases in countries just above the 80% student response threshold (Australia, Austria, Canada, Ireland, Poland and the USA)

13 Conclusions 1.It is possible to assess reliably both the direction and magnitude of response biases in England in 2003, and hence compare UK scores with those for other countries 2.The extent of the bias in means and SDs of scores is equal to about double the SEs, but the impact on ‘league table’ rankings is small 3.Biases were similar in 2000, when pupil response in England just reached the required threshold