Three perspectives on outcome measurement 22nd November 2017 Three perspectives on outcome measurement Findings from the CORC Evaluation Dr Emily Stapley, Rosa Town & Dr Jessica Deighton

Background to the CORC Evaluation CORC have asked the Evidence Based Practice Unit (EBPU) to conduct an evaluation to explore how their member services and associates experience: Engaging in routine outcome monitoring (ROM) – including outcome measurement, data collation, data analysis, and interpretation of findings Receiving support from the CORC team This includes child mental health and wellbeing services (in both the statutory and voluntary sector) and commissioners The aim is to identify the impact of and barriers/facilitators to engaging in ROM and working with CORC, from stakeholders’ perspectives

Structure of the CORC Evaluation Three waves of activity: What are the experiences of statutory child and adolescent mental health services (CAMHS) of implementing ROM and working with CORC? What are the experiences of voluntary sector child mental health and wellbeing services of implementing ROM and working with CORC? What are the experiences of commissioners in relation to ROM and working with CORC?

7 interviewees (CORC implementers, service managers, clinicians) CORC Evaluation Wave 1 - Methodology Sample: 3 CAMH services 7 interviewees (CORC implementers, service managers, clinicians) Data collection: Structured telephone interviews Average length = 26.03 minutes (range 18.03 - 31.06) Data analysis: Thematic analysis (Braun & Clarke, 2006)

Wave 1 findings - Perceived impact of ROM and CORC on the service CORC Evaluation Wave 1 findings - Perceived impact of ROM and CORC on the service Enhancing clinical practice, e.g. facilitating discussions with clients and provision of a ‘temperature check’ on progress “I really find it useful with some clients to kind of plot where they were at the start and kind of where they are now” (Clinician) Service improvement “The [experience of service] measure flags up things that we’ve actually [taken] on board and then changed” (CORC Implementer) Boosting service confidence and morale “You’re constantly thinking that you’re not up to the state where you should be, whereas when we’re getting these reports and this information from CORC, it’s really given the service a boost” (CORC Implementer)

Wave 1 findings - Perceived barriers to engaging with ROM CORC Evaluation Wave 1 findings - Perceived barriers to engaging with ROM IT system issues, e.g. glitches in services’ IT systems and a mismatch between CORC and service database systems “Our computers at work don’t access the internet very well. They’re very old browsers” (CORC Implementer) Lack of time and staffing capacity “We haven’t been able to keep up to pace with doing lots and lots of outcome measures. The difficulty we have is processing the data on a day-by-day basis, because it involves data entry, data extraction, graphing” (Clinician) Resistance to ROM “I know that we have struggled actually getting staff on board to use ROMs and to see the uses in them . . . however that is getting a lot better” (CORC Implementer)

10 small voluntary sector services CORC Evaluation Wave 2 - Methodology Sample: 10 small voluntary sector services 12 interviewees (CORC implementers, service managers, clinicians) Data collection: Structured telephone interviews Average length = 29.79 minutes (range 14.23 - 41.04) Data analysis: Thematic analysis (Braun & Clarke, 2006)

Wave 2 findings - Perceived impact of ROM and CORC on the service CORC Evaluation Wave 2 findings - Perceived impact of ROM and CORC on the service Areas of impact Factors limiting impact Provision of evidence of service impact Limitations of data collected by service A learning experience Fit between service and outcome measures An impetus to improve data collection Obstacles to data collection “What we wanted to do was show a sort of, a bigger picture of the stuff that we do . . . so that’s kind of what we’ve been using [the goal based outcome measure] with CORC [for]” (Service Manager) “Other services have, they’ll do six sessions . . . and then they do the measurement at the beginning, they’ll do the measurement at the end. We’re much more messy and fluid than that. You know we have young people who come to us for a while and go away, [then] they come back” (Service Manager)

8 clinical commissioning groups (CCG) 8 interviewees (commissioners) CORC Evaluation Wave 3 - Methodology Sample: 8 clinical commissioning groups (CCG) 8 interviewees (commissioners) Data collection: Structured telephone interviews Average length = 28.93 minutes (range 17.13 – 39.45) Data analysis: Thematic analysis (Braun & Clarke, 2006)

CORC Evaluation Wave 3 findings – The use or influence of data from commissioners’ perspectives To facilitate understanding of service provision and impact “[We want to use the CORC report] to look at how well we’re doing, where the gaps are, what’s not so strong, where we need to get better in collecting outcomes reported. A lot of things really. We want to marry that with sort of activity data, so ethnicity as well, look at the demographics around it too” (Commissioner) Making comparisons “[The CORC report] gives me a bit of a sense of where [area] is versus other members of the CORC partnership” (Commissioner) CORC data and reports are one part of the bigger picture around commissioning decisions “[The CORC report is] a starting point for a deeper discussion so it just takes the form of triangulation of information about the quality of the service” (Commissioner)

CORC Evaluation Wave 3 findings – Perceived challenges around ROM and working with data Use of technical language “I think initially [when working with CORC] there was some stumbling around language used . . . so things like ‘funnels’ and ‘funnel plot’, ‘confidence levels’, ‘meaningful statistical data’” (Commissioner) Time and capacity issues for commissioners and providers “The pressures on commissioners these days, particularly in CAMHS actually, are huge” (Commissioner) Services’ difficulties in collecting data “We haven’t had the data systems set up here to get that information to CORC. So it’s been quite, the reports that have come back have lacked a lot of data, a lot of information” (Commissioner)

Methodological limitations: CORC Evaluation Conclusions What are the key messages around outcome measurement from these three groups? Methodological limitations: Transferability of findings to other services and staff members Potential sampling bias What next?

CORC Evaluation References Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101.