1. Evaluating WP initiatives - overcoming the challenges
(Randomised Controlled) Trials and Tribulations The case of a programme to widen postgraduate participation
Sally Hancock & Paul Wakeling Centre for Research into Education and Social Justice Department of Education
Evaluating WP initiatives - overcoming the challenges
The Open University, 28th February 2019

2. Outline Aims and trial design Implementation and challenges
Implications for future research

3. While there have been some improvements in overall university access for the poorest students, there remain huge challenges, particularly in our leading universities and at postgraduate level…
Action [is] needed ensure that postgraduate study is open to all those with the ability to contribute to the advancement of knowledge and understanding.
Sir Peter Lampl, Sutton Trust (2013)

4. Two strand intervention to widen participation to postgraduate study
Final year undergraduates from POLAR 1 & 2 neighbourhoods
Structured programme of information, advice and guidance (Autumn 2017)
Baseline and post survey to monitor change in plans, knowledge, confidence, destinations
PGT offer holders from Black, Asian & Minority ethnic groups
Online pre-enrolment course covering advice and study skills (Summer 2018)
Monitor engagement with course, and conversion from offer to enrolment

5. Why RCT? ‘Gold standard’ for measuring interventions (Pocock 1983)
Increasingly used in educational evaluations; typically in compulsory education
Participants randomly assigned to the intervention or control group
Groups are assumed to be identical at the baseline measure; therefore any change in outcome is attributed to the intervention
Can help to identify efficacy and potential harm

6. Trial design Institution A B C D E Biology I Economics English
European Languages
Law
Mathematics
C= Control; I= Intervention

7. Implementation and challenges

8. Administrative Multi-institution working
Coordination and institutional differences (fidelity)
GDPR
Industrial action

9. ‘Intention to treat’ Disappointing levels of engagement
Self-selection bias (example)
Contradiction: enforcing voluntary interventions
…also: master’s loans

10. Strand 1 baseline I intend to study at PG level Control (22.9%)
Intervention (24.9%)
Engaged intervention (30.3%)
I have already applied for PG study
Control (7.8%)
Intervention (11.0%)
Engaged intervention (11.9%)
I am not considering PG study
Control (29.1%)
Intervention (25.1%)
Engaged intervention (18.4%)

11. Effects of the intervention - mixed results
Diplomacy of handling disappointing results – direction of change not as hoped on key variables
In our case, doesn’t give much insight into why we observe these trends

12. Strand 1 follow-up I intend to study at PG level Control (25.2%)
Intervention (24.7%)
Engaged intervention (28.0%)
-0.2%
I have already applied for PG study
Control (31.9%)
Intervention (19.6%)
Engaged intervention (35.0%)
+24.1%
18.6%
+23.1%
I am not considering PG study
Control (30.2%)
Intervention (31.2%)
Engaged intervention (24.8%)
+1.1%
+6.1%
+6.7%

13. Implications for future research and practice

14. Caveat indagator 1 Gold is expensive!
Nature of PG WP work with undergraduates means it is hard to get a clear ‘signal’
(Military-level!) control over the process needed
Most EEF trials don’t work (Lortie-Forgues & Inglis, forthcoming)
BUT difficulties with RCTs do not remove the need for the counterfactual
1Latin for ‘researcher beware’

15. sally.hancock@york.ac.uk paul.wakeling@york.ac.uk
Thank you