1. CQC’s approach to assessing its Impact and Value for Money EPSO Effectiveness working group meeting 11 May 2016, Oslo
Emily Hutchison, Economics Manager
PPP, Strategy and Intelligence
151 Buckingham Palace Road, London, SW1W 9SW

2. Problem: How do we demonstrate the difference we make?
We make sure health and social care services provide people with safe, effective, compassionate, high-quality care and we encourage them to improve.

3. Why do we need to demonstrate the difference we make?
History & context
CQC’s resources
CQC’s expenditure:
: £149m
: £168m
: £195m
: £222m
: £262m
In 2014/15 this represented approximately 0.16% of public spending on health & social care
CQC regulates 50,774 locations:
28,578 adult social care locations
20,208 primary medical services locations
1,988 hospital sector locations
CQC employs 3,262 full time equivalent staff members
In 2011/12 was widely criticised as “not fit for purpose” and National Audit Office (NAO) identified CQC as not offering value for money.
CQC went through a number of changes following this:
Budget increase
More comprehensive inspection model
Remain subject to much external scrutiny:
General pressure on public spending
Financial challenges for care organisations
CQC fee increases
Government drive to reduce red tape on businesses
National Audit Office review – second part in summer 2016

4. Solution: Annual publication on CQC’s Impact & VFM
CQC’s Board committed to the continuous assessment of CQC’s Impact and Value for Money to ensure transparency and to demonstrate:
To providers
To the public and people who use services
To parliament and government, including Department of Health, National Audit Office and Public Accounts Committee
Accountability
Improve how CQC works, ensuring it becomes more economical, efficient and effective
Continuous Improvement

5. Impact and Value for Money report
CQC started a programme of work assessing its Impact and Value for Money (VfM)
Annual reporting, 1st report to be published in October 2016
Assessment draws on CQC’s existing surveys e.g. of providers and inspection teams; performance, costing, inspection and monitoring data; and provider case studies
Assessment underpinned by logic model structure & three E’s: effectiveness, economy and efficiency
£££
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
The financial resources CQC receives through grant in aid and fees.
How we convert £ into resources that can be used to deliver activity.
The activities undertaken which lead to creation of outputs.
The outputs which in total create impact and help CQC to achieve its objectives.
The impact we have which allow CQC to demonstrate its value for money.

6. £££, inputs & activities: CQC’s costs (economy & efficiency)
New Costing Model establishes the cost of delivering CQC Operating Model activities
Costing Model will be used to
benchmark progress over time
establish standard costs for core activities, e.g. total cost of inspections and unit cost per inspection
measure impact of steps taken to achieve economy savings, e.g. renegotiation of contracts, rationalisation of CQC estate, etc. and
support CQC in identifying areas where CQC can be more efficient
Operating Model Costs
Direct Cost (£)
Indirect Cost (£)
Overhead Costs (£)
Total costs (£)
Registration
To show direct costs to CQC of delivering each component of the Operating Model
To show indirect costs to CQC of delivering each component of the Operating Model
Overheads associated with delivering each component of the Operating Model
Total costs per component of the Operating Model
Inspection
Monitoring
Enforcement
Independent Voice

7. Analysis of Quality Indicators
Inputs, activities & outcomes to providers & people using services (effectiveness)
CQC analyses quality indicators to see if the impact CQC has on these can be identified
When CQC re-inspect services, we assess if these have improved and if this is because of CQC
Provider interviews to identify costs of being regulated by CQC and what providers have done differently because of CQC activities
Surveys of providers, people who use services and other stakeholders so they can explain the impact CQC has on them
Surveys
Case Studies
Analysis of Quality Indicators
Ratings

8. What have we learned so far?

9. What we have learned so far?
Source
What we have learned
Surveys
Providers do make changes to their services because of CQC, in a range of different ways because of various CQC activities. Frequently cited examples providers have made improvements in are recruitment checks, improve processes like medicines management and staff appraisals & supervision.
Case studies
Providers experience monetary costs, time costs and non-financial costs as from CQC regulation.
A significant non-financial cost is stress to those working in health and social care organisations and there is likely to be a financial cost implication to this
Some changes providers make are short-term and do not lead to sustained change, but other changes lead to sustained improvements
Ratings
The majority of GP practices improve their rating on re-inspection and a substantial minority of Adult Social Care Services do too
We need to do further analysis to look at rating changes at a more granular level (key questions) and to see if we can link this with survey data to help with attribution
Analysis of quality indicators
Difficult to identify changes in quality indicators around the time of CQC inspection
Where we have identified changes corresponding to inspections for some providers, this needs to be verified by further qualitative research

10. Questions to discuss
How do your organisations assess value for money and performance?
Do you seek to establish and document the difference you make?
Do you publish results?
Are we focusing on right areas / questions?
What other evidence should we draw on going forward?

11. Annex

12. Hypothesis to test during the Impact and VfM assessment
In its assessment of costs and benefits CQC is looking to cover a range of hypothesis across several topics:
Topic
Hypothesis to test
CQC
CQC’s existence deters providers from falling below minimum standards.
Providers change the way they work because of CQC
Financial constraints, new models of care, devolution, etc. may influence the quality of care. CQC influences the quality of care within this context.
CQC reports can inform choice where people who uses services access these reports.
Regulation
Regulation influences quality.
The extent to which individual providers perceive CQC regulation as a burden is influenced by the provider’s rating and how long the sector has been regulated.
Ratings
CQC’s ratings and reports inform public choice about the care people who use services receive.
Providers change how they work because of ratings.
Providers rated poorly improve following their rating, and enforcement leads to faster improvements.

13. Hypothesis continued Topic Hypothesis to test Registration
Paper based approaches to registration are more costly to online alternatives, there are efficiencies that can be made to the registration process.
Registration gives clarity to providers on what standards of care are expected of them and what they need to do to achieve these.
Registration in some form is a pre-requisite for subsequent regulation and it helps providers put in place systems and plans for delivering good quality care.
Monitoring
There is value in monitoring to providers over and above it being a mechanism for managing risk and informing where and when to inspect.
Monitoring enables inspect and rate functions.
Inspection
Inspection poses a deterrence and providers prepare beforehand.
Published inspection reports and ratings facilitate the sharing of best practice and competition between providers, thereby improving care.
Providers change how they work because of inspections.
Poor providers improve following an inspection

14. Approach Assess the costs and benefits of everything CQC does…
…across each of the regulated sectors:
Registering care providers
Monitoring services
Inspecting and rating services
Taking action to protect people who use services
Speak with our independent voice, publishing regional and national views of the major quality issues in health and social care
Residential adult social care
Community adult social care
Hospices
NHS Acute Hospitals
Private Acute Services
Community health care
Mental health services
Primary medical services that we rate
Primary medical services we don’t rate
Primary dental services
Services we inspect with others such as children’s services and prisons
For each of these we look at the activities we undertake, the cost of these and what happens as a result

15. Engagement on Impact and Value for Money approach
Scoping
Engaged economic consultancy to develop approach, consulted with multidisciplinary advisory group and providers to test approach (FY 2014/15)
Development of methodology, e.g. of surveys and case studies
Facilitated provider workshops to; gather views on approach taken; test survey questions (Aug 2015); scope case study approach (Oct 2015)
Sharing of progress to date
Engagement on analysis and progress to date (May and June 2016)
Road testing emerging findings
Engagement on emerging findings (June 2016) ahead of publication (Oct 2016)
Previous engagement
Other Engagement Surveys
Annual Provider Survey
Public awareness and sentiment tracking surveys (Annual)
Stakeholder Survey (Annual)
Inspection Report Survey (Ongoing)
Publications Survey (Ongoing)
Targeted Analysis
Stats team looking at c.30 acute trusts, rated inadequate or require improvement for safety. Looking to see if there are improvements before or after inspection using Intelligent Monitoring Indicators related to safety
Ratings Analysis
Intelligence team looking to identify if ratings improve in between first inspection and re-inspection following first rating
Case studies
Ipsos Mori are in the process of providing 28 case studies, illustrating providers’ experience of CQC regulation. Case studies are to:
Identify what providers have done differently as a result of CQC activity
Determine costs to providers of CQC activity
Understand benefits of any actions taken as a result of CQC activity
Ascertain admin costs of being regulated by CQC
Gain insight into overall impact of CQC existing
The challenge is to find an approach that brings together a range of sources of different types so ratings are based on the full body of evidence. Although some may be used only as means of corroboration
Current
Planned

16. Impact and Value for Money report
CQC started a programme of work assessing its Impact and Value for Money (VfM)
Annual reporting, 1st report to be published in October 2016
Assessment draws on CQC’s existing surveys e.g. of providers and inspection teams; performance, costing, inspection and monitoring data; and provider case studies
Assessment considers both costs and benefits:
Impact and Value for Money assessment
Costs
Benefits
CQC’s financial costs
Financial and non-financial costs to providers and stakeholders
Intermediate benefits; providers and people who use services doing something differently due to CQC, e.g. providers changing how services operate or people who use services making choices based on CQC information
Final benefits; improvements to quality of care or people accessing better quality of care

17. Demonstrating the Difference we make: An annual assessment
£££
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
The financial resources CQC receives through grant in aid and fees.
How we convert £ into resources that can be used to deliver activity.
The activities undertaken which lead to creation of outputs.
The outputs which in total create impact and help CQC to achieve its objectives.
The impact we have which allow CQC to demonstrate its value for money.
IMPACT APPROACH – UNDERSTANDING COSTS AND BENEFITS
Economy – Is CQC procuring goods and services and paying the market rate in a way that allows us to get the greatest number of inputs for the money we have available?
Efficiency – Is CQC using its resources in the best way? How might we deliver our activities better? How are we converting inputs to outputs and how can we maximise this process?
Effectiveness – What is the impact of our regulation and are we having the desired effect? Does the cost to CQC and others justify the benefits?