Labour Market Knowledge and A-level Choices: A Blinded Cluster Randomized Controlled Trial with Linked Administrative Data Neil Davies, Peter Davies, Tian Qiu Randomised Controlled Trials in the Social Sciences Eleventh Annual Conference 09/09/16 1

Policy context English school students specialise at an early age Students chose A-level subjects age 15/16 These subject affect university subjects Degree subject is associated with labour market outcomes, such as wages Research hypothesis Does providing labour market information to students affect their choices and beliefs? 2

3

4

5 Background

6

Methods Cluster randomised controlled trial Intervention compared to control lesson 50 schools (20 private and 30 state) 5,593 students Two surveys Baseline Follow-up Linkage to National Pupil Database Actual A-level choices returned by schools Primary outcome: Actual A-level subject choices 7

Timeline of experiment 8 50 eligible schools randomly sampled Schools randomly allocated to intervention or control arm Week 0 Baseline survey Intervention or control lesson Follow-up survey Linkage to National Pupil Database Actual A- level choices reported Week 2 Week 4 Week 6

Baseline survey Questions Socioeconomic demographics Expected GCSE grades Prior intentions to study each subject at A-level (Likert scale) Measures of cultural capital (number of books) Intentions and motivations towards university study (1 to 5) Expected effect of attending university on salary (seven point scale) Expected effect of each degree on salary (seven point scale) 9

Intervention and control lessons Structured lessons Intervention lesson Information on the graduate premium for 10 subjects: Business, Education, Engineering, History, Languages, Law, Mathematics, Politics, Psychology, and Science Control lesson Based on advice from public available websites Advice from Russell group of universities Subject difficultly No information on relative wages 10

11

Statistical methods 1 12

Statistical Methods 2 Sampling Age range Have at least 100 students in their ‘sixth forms’ (students in the academic year between 16 and 18). Stratified sample to include 20 private schools and 30 state schools. Sampled schools were randomly sampled from list of eligible schools. Blinding Participating schools blinded to their allocation. Deviations from study protocol Six schools (571 students) allocated to the treatment arm did not take part in the intervention. Four schools (487 students) allocated to the control arm did not participate in the second round of questions. 13

Statistical methods 3 All analysis conducted using intention-to-treat. Primary outcome Logistic regression of intervention on subject choice Raw and adjusted for baseline variables (including prior intentions) Secondary outcomes Earnings expectation (average and own graduate salaries) Multivariate logistic regression of intervention on intention Raw and adjusted for baseline variables (including prior intentions) All standard errors and statistical tests allow for clustering by school, permutation p-values also reported in paper. 14

Motivations for attending university 15

Motivation for choosing subject 16

17

Proportion of students taking each subject in intervention and control arms 18

Effects of the intervention on subject choices: unadjusted 19 Less likely to takeMore likely to take

Effects of the intervention on subject choices: adjusted 20 Less likely to takeMore likely to take

Effects of the intervention on income expectations. 21

Strengths and limitations Strengths Cluster randomized Administrative data on educational attainment, and data on hard choices Large sample Active lesson and engagement rather than passive online website Limitations Multiple testing Power Missing data Deviations from study protocol Representativeness 22

Conclusions and future work Students’ educational choices were affected by the intervention Potentially cheap intervention Some evidence that students beliefs are not accurate Little evidence of interactions by school type or gender Benefit of linkage for randomised trials in social science Replication needed - Ioannidis (2014) Future – linked NPD and HESA data Compare with randomly non-sampled schools “untreated control” 23

24

References Walker, I. & Zhu, Y. (2011). Differences by degree: Evidence of the net financial return to undergraduate study for England and Wales, Economics of Education Review, 30, pp O’Leary NC, Sloane PJ. The wage premium for university education in Great Britain during a decade of change: wage premium for university education. The Manchester School 2011;79:740–64. doi: /j x Britton J, Dearden L, Shephard N, et al. How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic background McGuigan, M. McNally, S. & Wyness, G. (2012). Student Awareness of Costs and Benefits of Educational Decisions: Effects of an Information Campaign. Centre for Economics of Education Working Paper CEE DP 139. (London, London School of Economics). Manski, C. (2004). Measuring expectations, Econometrica, 72, 5, pp Manski (1993) Adolescent econometricians: How do youth infer the returns to schooling? in Studies of supply and demand in higher education (University of Chicago Press, 1993; pp. 43–60. Jensen, R. (2010). The (Perceived) returns to education and the demand for schooling. Quarterly Journal of Economics, 125, No.2: Jerrim. J. (2011). Do UK Higher Education Students Overestimate their starting salary? Fiscal Studies, 32, 4, pp Wood A.M., White, I. R. & Thompson, S. G. (2004). Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clinical Trials. 1(4):368–76. Ioannidis J.P.A. (2014). How to Make More Published Research True. PLoS Medicine. 21;11(10):e