CQC’s approach to assessing its Impact and Value for Money EPSO Effectiveness working group meeting 11 May 2016, Oslo Emily Hutchison, Economics Manager PPP, Strategy and Intelligence 151 Buckingham Palace Road, London, SW1W 9SW Emily.Hutchison@cqc.org.uk

Problem: How do we demonstrate the difference we make? We make sure health and social care services provide people with safe, effective, compassionate, high-quality care and we encourage them to improve.

Why do we need to demonstrate the difference we make? History & context CQC’s resources CQC’s expenditure: 2011-12: £149m 2012-13: £168m 2013-14: £195m 2014-15: £222m 2015-16: £262m In 2014/15 this represented approximately 0.16% of public spending on health & social care CQC regulates 50,774 locations: 28,578 adult social care locations 20,208 primary medical services locations 1,988 hospital sector locations CQC employs 3,262 full time equivalent staff members In 2011/12 was widely criticised as “not fit for purpose” and National Audit Office (NAO) identified CQC as not offering value for money. CQC went through a number of changes following this: Budget increase More comprehensive inspection model Remain subject to much external scrutiny: General pressure on public spending Financial challenges for care organisations CQC fee increases Government drive to reduce red tape on businesses National Audit Office review – second part in summer 2016

Solution: Annual publication on CQC’s Impact & VFM CQC’s Board committed to the continuous assessment of CQC’s Impact and Value for Money to ensure transparency and to demonstrate: To providers To the public and people who use services To parliament and government, including Department of Health, National Audit Office and Public Accounts Committee Accountability Improve how CQC works, ensuring it becomes more economical, efficient and effective Continuous Improvement

Impact and Value for Money report CQC started a programme of work assessing its Impact and Value for Money (VfM) Annual reporting, 1st report to be published in October 2016 Assessment draws on CQC’s existing surveys e.g. of providers and inspection teams; performance, costing, inspection and monitoring data; and provider case studies Assessment underpinned by logic model structure & three E’s: effectiveness, economy and efficiency £££ INPUTS ACTIVITIES OUTPUTS OUTCOMES The financial resources CQC receives through grant in aid and fees. How we convert £ into resources that can be used to deliver activity. The activities undertaken which lead to creation of outputs. The outputs which in total create impact and help CQC to achieve its objectives. The impact we have which allow CQC to demonstrate its value for money.

£££, inputs & activities: CQC’s costs (economy & efficiency) New Costing Model establishes the cost of delivering CQC Operating Model activities Costing Model will be used to benchmark progress over time establish standard costs for core activities, e.g. total cost of inspections and unit cost per inspection measure impact of steps taken to achieve economy savings, e.g. renegotiation of contracts, rationalisation of CQC estate, etc. and support CQC in identifying areas where CQC can be more efficient Operating Model Costs Direct Cost (£) Indirect Cost (£) Overhead Costs (£) Total costs (£) Registration To show direct costs to CQC of delivering each component of the Operating Model To show indirect costs to CQC of delivering each component of the Operating Model Overheads associated with delivering each component of the Operating Model Total costs per component of the Operating Model Inspection Monitoring Enforcement Independent Voice

Analysis of Quality Indicators Inputs, activities & outcomes to providers & people using services (effectiveness) CQC analyses quality indicators to see if the impact CQC has on these can be identified When CQC re-inspect services, we assess if these have improved and if this is because of CQC Provider interviews to identify costs of being regulated by CQC and what providers have done differently because of CQC activities Surveys of providers, people who use services and other stakeholders so they can explain the impact CQC has on them Surveys Case Studies Analysis of Quality Indicators Ratings

What have we learned so far?

What we have learned so far? Source What we have learned Surveys Providers do make changes to their services because of CQC, in a range of different ways because of various CQC activities. Frequently cited examples providers have made improvements in are recruitment checks, improve processes like medicines management and staff appraisals & supervision. Case studies Providers experience monetary costs, time costs and non-financial costs as from CQC regulation. A significant non-financial cost is stress to those working in health and social care organisations and there is likely to be a financial cost implication to this Some changes providers make are short-term and do not lead to sustained change, but other changes lead to sustained improvements Ratings The majority of GP practices improve their rating on re-inspection and a substantial minority of Adult Social Care Services do too We need to do further analysis to look at rating changes at a more granular level (key questions) and to see if we can link this with survey data to help with attribution Analysis of quality indicators Difficult to identify changes in quality indicators around the time of CQC inspection Where we have identified changes corresponding to inspections for some providers, this needs to be verified by further qualitative research

Questions to discuss How do your organisations assess value for money and performance? Do you seek to establish and document the difference you make? Do you publish results? Are we focusing on right areas / questions? What other evidence should we draw on going forward?

Annex

Hypothesis to test during the Impact and VfM assessment In its assessment of costs and benefits CQC is looking to cover a range of hypothesis across several topics: Topic Hypothesis to test CQC CQC’s existence deters providers from falling below minimum standards. Providers change the way they work because of CQC Financial constraints, new models of care, devolution, etc. may influence the quality of care. CQC influences the quality of care within this context. CQC reports can inform choice where people who uses services access these reports. Regulation Regulation influences quality. The extent to which individual providers perceive CQC regulation as a burden is influenced by the provider’s rating and how long the sector has been regulated. Ratings CQC’s ratings and reports inform public choice about the care people who use services receive. Providers change how they work because of ratings. Providers rated poorly improve following their rating, and enforcement leads to faster improvements.

Hypothesis continued Topic Hypothesis to test Registration Paper based approaches to registration are more costly to online alternatives, there are efficiencies that can be made to the registration process. Registration gives clarity to providers on what standards of care are expected of them and what they need to do to achieve these. Registration in some form is a pre-requisite for subsequent regulation and it helps providers put in place systems and plans for delivering good quality care. Monitoring There is value in monitoring to providers over and above it being a mechanism for managing risk and informing where and when to inspect. Monitoring enables inspect and rate functions. Inspection Inspection poses a deterrence and providers prepare beforehand. Published inspection reports and ratings facilitate the sharing of best practice and competition between providers, thereby improving care. Providers change how they work because of inspections. Poor providers improve following an inspection

Approach Assess the costs and benefits of everything CQC does… …across each of the regulated sectors: Registering care providers Monitoring services Inspecting and rating services Taking action to protect people who use services Speak with our independent voice, publishing regional and national views of the major quality issues in health and social care Residential adult social care Community adult social care Hospices NHS Acute Hospitals Private Acute Services Community health care Mental health services Primary medical services that we rate Primary medical services we don’t rate Primary dental services Services we inspect with others such as children’s services and prisons For each of these we look at the activities we undertake, the cost of these and what happens as a result

Engagement on Impact and Value for Money approach Scoping Engaged economic consultancy to develop approach, consulted with multidisciplinary advisory group and providers to test approach (FY 2014/15) Development of methodology, e.g. of surveys and case studies Facilitated provider workshops to; gather views on approach taken; test survey questions (Aug 2015); scope case study approach (Oct 2015) Sharing of progress to date Engagement on analysis and progress to date (May and June 2016) Road testing emerging findings Engagement on emerging findings (June 2016) ahead of publication (Oct 2016) Previous engagement Other Engagement Surveys Annual Provider Survey Public awareness and sentiment tracking surveys (Annual) Stakeholder Survey (Annual) Inspection Report Survey (Ongoing) Publications Survey (Ongoing) Targeted Analysis Stats team looking at c.30 acute trusts, rated inadequate or require improvement for safety. Looking to see if there are improvements before or after inspection using Intelligent Monitoring Indicators related to safety Ratings Analysis Intelligence team looking to identify if ratings improve in between first inspection and re-inspection following first rating Case studies Ipsos Mori are in the process of providing 28 case studies, illustrating providers’ experience of CQC regulation. Case studies are to: Identify what providers have done differently as a result of CQC activity Determine costs to providers of CQC activity Understand benefits of any actions taken as a result of CQC activity Ascertain admin costs of being regulated by CQC Gain insight into overall impact of CQC existing The challenge is to find an approach that brings together a range of sources of different types so ratings are based on the full body of evidence. Although some may be used only as means of corroboration Current Planned

Impact and Value for Money report CQC started a programme of work assessing its Impact and Value for Money (VfM) Annual reporting, 1st report to be published in October 2016 Assessment draws on CQC’s existing surveys e.g. of providers and inspection teams; performance, costing, inspection and monitoring data; and provider case studies Assessment considers both costs and benefits: Impact and Value for Money assessment Costs Benefits CQC’s financial costs Financial and non-financial costs to providers and stakeholders Intermediate benefits; providers and people who use services doing something differently due to CQC, e.g. providers changing how services operate or people who use services making choices based on CQC information Final benefits; improvements to quality of care or people accessing better quality of care

Demonstrating the Difference we make: An annual assessment £££ INPUTS ACTIVITIES OUTPUTS OUTCOMES The financial resources CQC receives through grant in aid and fees. How we convert £ into resources that can be used to deliver activity. The activities undertaken which lead to creation of outputs. The outputs which in total create impact and help CQC to achieve its objectives. The impact we have which allow CQC to demonstrate its value for money. IMPACT APPROACH – UNDERSTANDING COSTS AND BENEFITS Economy – Is CQC procuring goods and services and paying the market rate in a way that allows us to get the greatest number of inputs for the money we have available? Efficiency – Is CQC using its resources in the best way? How might we deliver our activities better? How are we converting inputs to outputs and how can we maximise this process? Effectiveness – What is the impact of our regulation and are we having the desired effect? Does the cost to CQC and others justify the benefits?