1. To RCT, or not RCT: that is the question
Mandy Crow, Lucy Lernelius, Alex Webb
Widening Participation and Outreach Team,
The University of Manchester

2. “Craft a patchwork of evaluation methodologies”*
WP programmes aim to change the lives of children and young people by intervening in complex systems
Complexity aware evaluation “positions evaluation as an internal, team function integrated into action and ongoing interpretive processes” (Douthwaite et al, 2017)
Harrison and Waller (2017) recommend a ‘realist small steps approach’ whereby evidence is built incrementally, based upon an evolving theory of change framework
*Connelly, 2018

3. Manchester Access Programme (MAP)
MAP is a structured scheme for local post-16 students from less advantaged backgrounds

4. Manchester Access Programme (MAP)
Decision Manchester
Opportunity for students to get advice and an early decision on their application to The University of Manchester, before submitting their UCAS application
Progress and Options workshops – to discuss higher education options with a member of university staff Support with writing UCAS Personal Statement Submission of pre-UCAS form and feedback from admission decision-makers Interview skills workshops

5. Post programme questionnaire 2018 (E2019) As a result of completing MAP do you feel more confident about your university applications? 94%
Yes, I have gained so many invaluable skills from MAP as well as application support and guidance, especially with writing a personal statement
…it has given me so much confidence to progress to the next stage of my life. It was a privilege to be part of it and I feel the lower offer has given me a chance to achieve my goals.
Yes, it is because of MAP that I am more confident in applying for University

6. Pre-UCAS applications – Decision Manchester 2018 (E2019)
560
course applications through Decision Manchester
379
initially successful
181
initially unsuccessful 23
reapplications
successful
402
Applications to Manchester

7. Analysis of previous cohorts has shown
Around 1/3 of MAP entrants to the University need the 2 A-level grade (or equivalent) reduction.
Twice as many MAP students are accepted onto courses at Manchester compared to non-MAP applicants from the same colleges.
Around 30% of students who complete MAP will progress onto a course at another university.

8. MAP undergraduate entrants have a continuation rate of nearly 95%
95% of students who successfully complete MAP will make an application to Manchester – with around 75% being made at least one offer
Following expansion in 2013, around 500 students successfully complete MAP each year
29% of MAP entrants to the University do not need the 2 A-level grade (or equivalent) reduction
Impact
Since 2005, 1,974 MAP students have successfully gained a place at The University of Manchester
And when they have completed University over 90% of MAP graduates are in a positive employment destination
Many other examples of how we evaluate MAP – highlight some specific examples relating to student success
Analysis for 2016 showed that 82.7% of MAP undergraduates were from households with incomes below £25,000 – compared to 25.1% for the University overall
Analysis of a sample of 131 MAP entrants in 2015 showed that 29% of students did not need the 2 A-level grades
The proportion of MAP graduates who are in a positive employment destination 6 months after graduation is higher than the University overall
2015/16 – 91.7% of MAP graduates were in a positive employment destination, compared to 82.2% for the University’s total undergraduate population
For entrants between , the continuation rate of MAP students was 94.4% compared to 92.7% for the University’s total undergraduate student population
Typically ¾ of our MAP undergraduates come from the lowest household incomes
MAP undergraduate entrants have a continuation rate of nearly 95%

9. Studentship 3 year funded PhD studentship
To further develop understanding of:
the effect of MAP on raising attainment - does the work we do to raise expectations and support students in developing a range of skills to demonstrate their preparedness for university study have any impact on their attainment?
the choice to engage/not engage with MAP -what are the challenges and opportunities in engaging schools and recruiting students?
the nature of the impact of MAP, in the context of everyday life and experiences.
Key theories: Habitus, cultural capital, fields (Bourdieu); transactionalism, pragmatism
Methods: Interviews, case study group followed through the programme and beyond, quantitative analysis of attainment data

10. So.. To RCT, or not RCT?
Context is really important– ‘one size does not fit all’
Ethical and logistical considerations
An RCT may tell us that an intervention has worked – however it will not tell us how an intervention has worked. Ideally would also need qualitative work to happen at the same time
RCTs are not the only way to evaluate impact - a rigorous mixed methods approach is better than a poor RCT!
It is possible to show how the intermediate outcomes make a strong case for a longer-term impact – but may not be able to generate a causal conclusion
Resources – cost of funding an RCT