Evaluating WP initiatives - overcoming the challenges (Randomised Controlled) Trials and Tribulations The case of a programme to widen postgraduate participation Sally Hancock & Paul Wakeling Centre for Research into Education and Social Justice Department of Education Evaluating WP initiatives - overcoming the challenges The Open University, 28th February 2019

Outline Aims and trial design Implementation and challenges Implications for future research

While there have been some improvements in overall university access for the poorest students, there remain huge challenges, particularly in our leading universities and at postgraduate level… Action [is] needed ensure that postgraduate study is open to all those with the ability to contribute to the advancement of knowledge and understanding. Sir Peter Lampl, Sutton Trust (2013)

Two strand intervention to widen participation to postgraduate study Final year undergraduates from POLAR 1 & 2 neighbourhoods Structured programme of information, advice and guidance (Autumn 2017) Baseline and post survey to monitor change in plans, knowledge, confidence, destinations PGT offer holders from Black, Asian & Minority ethnic groups Online pre-enrolment course covering advice and study skills (Summer 2018) Monitor engagement with course, and conversion from offer to enrolment

Why RCT? ‘Gold standard’ for measuring interventions (Pocock 1983) Increasingly used in educational evaluations; typically in compulsory education Participants randomly assigned to the intervention or control group Groups are assumed to be identical at the baseline measure; therefore any change in outcome is attributed to the intervention Can help to identify efficacy and potential harm

Trial design Institution A B C D E Biology I Economics English European Languages Law Mathematics C= Control; I= Intervention

Implementation and challenges

Administrative Multi-institution working Coordination and institutional differences (fidelity) GDPR Industrial action

‘Intention to treat’ Disappointing levels of engagement Self-selection bias (example) Contradiction: enforcing voluntary interventions …also: master’s loans

Strand 1 baseline I intend to study at PG level Control (22.9%) Intervention (24.9%) Engaged intervention (30.3%) I have already applied for PG study Control (7.8%) Intervention (11.0%) Engaged intervention (11.9%) I am not considering PG study Control (29.1%) Intervention (25.1%) Engaged intervention (18.4%)

Effects of the intervention - mixed results Diplomacy of handling disappointing results – direction of change not as hoped on key variables In our case, doesn’t give much insight into why we observe these trends

Strand 1 follow-up I intend to study at PG level Control (25.2%) Intervention (24.7%) Engaged intervention (28.0%) -0.2% I have already applied for PG study Control (31.9%) Intervention (19.6%) Engaged intervention (35.0%) +24.1% 18.6%   +23.1% I am not considering PG study Control (30.2%) Intervention (31.2%) Engaged intervention (24.8%)   +1.1% +6.1% +6.7%

Implications for future research and practice

Caveat indagator 1 Gold is expensive! Nature of PG WP work with undergraduates means it is hard to get a clear ‘signal’ (Military-level!) control over the process needed Most EEF trials don’t work (Lortie-Forgues & Inglis, forthcoming) BUT difficulties with RCTs do not remove the need for the counterfactual 1Latin for ‘researcher beware’

sally.hancock@york.ac.uk paul.wakeling@york.ac.uk Thank you sally.hancock@york.ac.uk paul.wakeling@york.ac.uk