To RCT, or not RCT: that is the question Mandy Crow, Lucy Lernelius, Alex Webb Widening Participation and Outreach Team, The University of Manchester

“Craft a patchwork of evaluation methodologies”* WP programmes aim to change the lives of children and young people by intervening in complex systems Complexity aware evaluation “positions evaluation as an internal, team function integrated into action and ongoing interpretive processes” (Douthwaite et al, 2017) Harrison and Waller (2017) recommend a ‘realist small steps approach’ whereby evidence is built incrementally, based upon an evolving theory of change framework *Connelly, 2018

Manchester Access Programme (MAP) MAP is a structured scheme for local post-16 students from less advantaged backgrounds

Manchester Access Programme (MAP) Decision Manchester Opportunity for students to get advice and an early decision on their application to The University of Manchester, before submitting their UCAS application Progress and Options workshops – to discuss higher education options with a member of university staff Support with writing UCAS Personal Statement Submission of pre-UCAS form and feedback from admission decision-makers Interview skills workshops

Post programme questionnaire 2018 (E2019) As a result of completing MAP do you feel more confident about your university applications? 94% Yes, I have gained so many invaluable skills from MAP as well as application support and guidance, especially with writing a personal statement …it has given me so much confidence to progress to the next stage of my life. It was a privilege to be part of it and I feel the lower offer has given me a chance to achieve my goals. Yes, it is because of MAP that I am more confident in applying for University

Pre-UCAS applications – Decision Manchester 2018 (E2019) 560 course applications through Decision Manchester 379 initially successful 181 initially unsuccessful 23 reapplications successful 402 Applications to Manchester

Analysis of previous cohorts has shown Around 1/3 of MAP entrants to the University need the 2 A-level grade (or equivalent) reduction. Twice as many MAP students are accepted onto courses at Manchester compared to non-MAP applicants from the same colleges. Around 30% of students who complete MAP will progress onto a course at another university.

MAP undergraduate entrants have a continuation rate of nearly 95% 95% of students who successfully complete MAP will make an application to Manchester – with around 75% being made at least one offer Following expansion in 2013, around 500 students successfully complete MAP each year 29% of MAP entrants to the University do not need the 2 A-level grade (or equivalent) reduction Impact Since 2005, 1,974 MAP students have successfully gained a place at The University of Manchester And when they have completed University over 90% of MAP graduates are in a positive employment destination Many other examples of how we evaluate MAP – highlight some specific examples relating to student success Analysis for 2016 showed that 82.7% of MAP undergraduates were from households with incomes below £25,000 – compared to 25.1% for the University overall Analysis of a sample of 131 MAP entrants in 2015 showed that 29% of students did not need the 2 A-level grades The proportion of MAP graduates who are in a positive employment destination 6 months after graduation is higher than the University overall 2015/16 – 91.7% of MAP graduates were in a positive employment destination, compared to 82.2% for the University’s total undergraduate population For entrants between 2008-2014, the continuation rate of MAP students was 94.4% compared to 92.7% for the University’s total undergraduate student population Typically ¾ of our MAP undergraduates come from the lowest household incomes MAP undergraduate entrants have a continuation rate of nearly 95%

Studentship 3 year funded PhD studentship To further develop understanding of: the effect of MAP on raising attainment - does the work we do to raise expectations and support students in developing a range of skills to demonstrate their preparedness for university study have any impact on their attainment? the choice to engage/not engage with MAP -what are the challenges and opportunities in engaging schools and recruiting students? the nature of the impact of MAP, in the context of everyday life and experiences. Key theories: Habitus, cultural capital, fields (Bourdieu); transactionalism, pragmatism Methods: Interviews, case study group followed through the programme and beyond, quantitative analysis of attainment data

So.. To RCT, or not RCT? Context is really important– ‘one size does not fit all’ Ethical and logistical considerations An RCT may tell us that an intervention has worked – however it will not tell us how an intervention has worked. Ideally would also need qualitative work to happen at the same time RCTs are not the only way to evaluate impact - a rigorous mixed methods approach is better than a poor RCT! It is possible to show how the intermediate outcomes make a strong case for a longer-term impact – but may not be able to generate a causal conclusion Resources – cost of funding an RCT