4 th National Patient Safety Conference Friday 7 th November, 2014 Inspecting for improvement? Lessons from evaluating the Care Quality Commission's new approach to acute hospitals Professor Kieran Walshe

4 th National Patient Safety Conference Friday 7 th November, 2014 Or: Making improvement work: learning from research Professor Kieran Walshe

Hieronymus Bosch. The Conjuror. c.1500

What’s inside the “black box” of quality improvement What do we know? –Improvement projects work – sometimes… –How/why they work is often quite hard to see/tell/predict –Scaling up and transferring improvement is problematic –Is it method, or implementation, or systems, or people, or…? –Organisations vary in “improvement capability” Issues or problems with improvement research –Terminology, taxonomy and description –Theories and mechanisms –Context and implementation –Cumulation of knowledge or evidence

Diagram from James Albert Bonsack's patent application (U.S. patent 238,640, granted March 8, 1881)

Case study: Care Quality Commission and regulating health and social care Established in 2009 to regulate health and social care in England Mission “to protect and promote” the health, safety and welfare of service users and the “general purpose of encouraging the improvement of health and social care services” Much criticised from all sides for its performance and effectiveness – several major high profile failures Complete change of board and senior executives from 2012, and review of strategy and approach in 2013

What is known about effective regulation? What can be learned from other settings? How does evidence from elsewhere apply to health and social care? What important knowledge gaps exist, and how can they be filled?

Mapping the theories of regulation

CQC’s new approach to hospital inspection Much larger and more expert inspection teams More detailed and extensive set of inspection processes drawing on a wider range of data sources and fieldwork Focusing the inspections on eight defined core service areas Assessing and rating performance in five domains (safe, effective, caring, responsive and well-led) using a four-point rating scale (inadequate, requires improvement, good or outstanding) Much more detailed and comprehensive inspection report with a full narrative description of services in each of the core service areas alongside ratings.

Researching CQC’s new approach About 16 interviews with people in CQC and outside about the new acute hospital regulatory model Six observed hospital inspections in late 2013 – about 48 days of non-participant observation, review of documents, attending QA group meetings, quality summits etc – and 4 followup observations in 2014 About 65 1:1 telephone interviews with CQC inspection team members and NHS trust staff following inspections in 2013/14 Surveys of CQC inspection team members and trust staff following inspections in 2014

Evidence, KLOEs and information Data pack really impressive – not well understood, so not well used KLOEs – good starting point but mostly general, not specific to the organisation, quite high level Wider sources of data – up to 500+ documents from trust and others, some not even read, hard to sift/sort and synthesise Organisation’s own ability to assess/explain/use information on own performance only tangentially tested

Inspection teams Large teams maximise content expertise and “feet on the ground”, at a cost to selectivity, coherence and manageability Minimal selection/training/matching, some skills gaps evident, prior experience largely shapes approach Team roles and management sometimes unclear and variable – best teams have clearly defined complementary roles Team members’ competence, confidence and credibility crucial – learning from experience

Inspection processes Actual fieldwork process – highly inductive, experiential, intuitive, dependent on team skills, little structured or directed inquiry, variable across teams but scope for specialisation Some aspects like focus groups, corporate dimension, interviews, listening events need some rethinking to maximise value Corroboration sessions great idea but hard in large teams and become bilaterals under time pressure Relentless activity – no time to think/reflect/discuss

Evidence, judgements, ratings, reports Inspector observation/experience tend to predominate Core service areas – good but don’t touch some clinical services, and lots of corporate systems/issues Domains have some ambiguities and hard to know what fits where; well-led an organisational cover-all Rating levels not really defined and so interpreted variably within teams, across teams, across inspections Process for integrating and weighting disparate evidence to form rating (and back it up) implicit A lot left to CQC team leads to write up afterwards

Some ideas for CQC on improvement Maximise the quality of inspection team members through selection, training, deployment, experience and review/feedback Maximise validity, reliability, efficiency and utility of inspections through greater structure/process without constraining flexibility or losing scope for professional judgement Maximise validity/reliability of ratings through simplification, definition, training and testing Start to measure impact after the inspection to drive change and improvement

Some lessons learned from improvement research Build research/evaluation capacity in healthcare organisations Make evaluation an integral part of all improvement programmes/projects – from design onwards Test taxonomy/description (what does it do?); theory/mechanism (how does it work?) and context/implementation (who does it and where?) Celebrate both successful and unsuccessful projects/programmes for different reasons Do not overclaim effectiveness and impact