1. Three perspectives on outcome measurement
22nd November 2017
Three perspectives on outcome measurement
Findings from the CORC Evaluation
Dr Emily Stapley, Rosa Town & Dr Jessica Deighton

2. Background to the CORC Evaluation
CORC have asked the Evidence Based Practice Unit (EBPU) to conduct an evaluation to explore how their member services and associates experience:
Engaging in routine outcome monitoring (ROM) – including outcome measurement, data collation, data analysis, and interpretation of findings
Receiving support from the CORC team
This includes child mental health and wellbeing services (in both the statutory and voluntary sector) and commissioners
The aim is to identify the impact of and barriers/facilitators to engaging in ROM and working with CORC, from stakeholders’ perspectives

3. Structure of the CORC Evaluation
Three waves of activity:
What are the experiences of statutory child and adolescent mental health services (CAMHS) of implementing ROM and working with CORC?
What are the experiences of voluntary sector child mental health and wellbeing services of implementing ROM and working with CORC?
What are the experiences of commissioners in relation to ROM and working with CORC?

4. 7 interviewees (CORC implementers, service managers, clinicians)
CORC Evaluation
Wave 1 - Methodology
Sample:
3 CAMH services
7 interviewees (CORC implementers, service managers, clinicians)
Data collection:
Structured telephone interviews
Average length = minutes (range )
Data analysis:
Thematic analysis (Braun & Clarke, 2006)

5. Wave 1 findings - Perceived impact of ROM and CORC on the service
CORC Evaluation
Wave 1 findings - Perceived impact of ROM and CORC on the service
Enhancing clinical practice, e.g. facilitating discussions with clients and provision of a ‘temperature check’ on progress
“I really find it useful with some clients to kind of plot where they were at the start and kind of where they are now” (Clinician)
Service improvement
“The [experience of service] measure flags up things that we’ve actually [taken] on board and then changed” (CORC Implementer)
Boosting service confidence and morale
“You’re constantly thinking that you’re not up to the state where you should be, whereas when we’re getting these reports and this information from CORC, it’s really given the service a boost” (CORC Implementer)

6. Wave 1 findings - Perceived barriers to engaging with ROM
CORC Evaluation
Wave 1 findings - Perceived barriers to engaging with ROM
IT system issues, e.g. glitches in services’ IT systems and a mismatch between CORC and service database systems
“Our computers at work don’t access the internet very well. They’re very old browsers” (CORC Implementer)
Lack of time and staffing capacity
“We haven’t been able to keep up to pace with doing lots and lots of outcome measures. The difficulty we have is processing the data on a day-by-day basis, because it involves data entry, data extraction, graphing” (Clinician)
Resistance to ROM
“I know that we have struggled actually getting staff on board to use ROMs and to see the uses in them however that is getting a lot better” (CORC Implementer)

7. 10 small voluntary sector services
CORC Evaluation
Wave 2 - Methodology
Sample:
10 small voluntary sector services
12 interviewees (CORC implementers, service managers, clinicians)
Data collection:
Structured telephone interviews
Average length = minutes (range )
Data analysis:
Thematic analysis (Braun & Clarke, 2006)

8. Wave 2 findings - Perceived impact of ROM and CORC on the service
CORC Evaluation
Wave 2 findings - Perceived impact of ROM and CORC on the service
Areas of impact
Factors limiting impact
Provision of evidence of service impact
Limitations of data collected by service
A learning experience
Fit between service and outcome measures
An impetus to improve data collection
Obstacles to data collection
“What we wanted to do was show a sort of, a bigger picture of the stuff that we do so that’s kind of what we’ve been using [the goal based outcome measure] with CORC [for]” (Service Manager)
“Other services have, they’ll do six sessions and then they do the measurement at the beginning, they’ll do the measurement at the end. We’re much more messy and fluid than that. You know we have young people who come to us for a while and go away, [then] they come back” (Service Manager)

9. 8 clinical commissioning groups (CCG) 8 interviewees (commissioners)
CORC Evaluation
Wave 3 - Methodology
Sample:
8 clinical commissioning groups (CCG)
8 interviewees (commissioners)
Data collection:
Structured telephone interviews
Average length = minutes (range – )
Data analysis:
Thematic analysis (Braun & Clarke, 2006)

10. CORC Evaluation
Wave 3 findings – The use or influence of data from commissioners’ perspectives
To facilitate understanding of service provision and impact
“[We want to use the CORC report] to look at how well we’re doing, where the gaps are, what’s not so strong, where we need to get better in collecting outcomes reported. A lot of things really. We want to marry that with sort of activity data, so ethnicity as well, look at the demographics around it too” (Commissioner)
Making comparisons
“[The CORC report] gives me a bit of a sense of where [area] is versus other members of the CORC partnership” (Commissioner)
CORC data and reports are one part of the bigger picture around commissioning decisions
“[The CORC report is] a starting point for a deeper discussion so it just takes the form of triangulation of information about the quality of the service” (Commissioner)

11. CORC Evaluation
Wave 3 findings – Perceived challenges around ROM and working with data
Use of technical language
“I think initially [when working with CORC] there was some stumbling around language used so things like ‘funnels’ and ‘funnel plot’, ‘confidence levels’, ‘meaningful statistical data’” (Commissioner)
Time and capacity issues for commissioners and providers
“The pressures on commissioners these days, particularly in CAMHS actually, are huge” (Commissioner)
Services’ difficulties in collecting data
“We haven’t had the data systems set up here to get that information to CORC. So it’s been quite, the reports that have come back have lacked a lot of data, a lot of information” (Commissioner)

12. Methodological limitations:
CORC Evaluation
Conclusions
What are the key messages around outcome measurement from these three groups?
Methodological limitations:
Transferability of findings to other services and staff members
Potential sampling bias
What next?

13. CORC Evaluation
References
Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3,