1. Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran Walshe and Denham Phipps University of Manchester, UK kieran.walshe@mbs.ac.uk

2. Programme theory in evaluation research Applying programme theory to understanding the Care Quality Commission’s regulatory model Conclusions and reflections

3. Programme theory “A plausible and sensible model of how a programme is supposed to work” which “provides the hypothesized links between program features and planned outcomes to be tested” (Bickman 1996) Derived from existing knowledge/research and from programme actors – policymakers, leaders etc Makes programme design choices explicit/visible and open to contestation/adjudication In “what works” – helps distinguish between theory and implementation, to understand mechanism and context, and to generalise

5. CQC study objectives and methods Develop a “logic model” to map programme theories underlying the regulatory model Use it to develop a framework for identifying and prioritising topics and questions for evaluation Document review – mainly CQC publications/reports Interviews with circa 20 CQC staff in London and Preston Review of the research literature and comparison of practice with four other regulators (two in health elsewhere, two in other settings in UK)

6. Regulatory model: components Registration Compliance –Standard setting –Information gathering and risk assessment –Inspection and reporting Enforcement Information provision

7. Registration Three main models of registration: –Threshold – tool for driving improvement, deterring poor providers from registering, leverage, assurance –Relationship – get to know provider, build knowledge base, establish relations, triage or risk assess –Administrative process – collect basic data, register and then pass on to others to regulate Consequences – foregone opportunities, deferred regulatory load

8. Compliance: standard setting Four main models of standard setting: –Mechanism to frame/set stakeholder expectations –Mechanism for improvement through self-enforced compliance –Mechanism for compliance through measurement and enforcement –Mechanism for differentiating performance of providers Consequences: generic standards, minimal/median /maximal level, content and focus, understanding and implementation

9. Compliance: information gathering and risk assessment Three main models of proportionate regulation: –Using information to determine when to use regulatory interventions –Using information to focus/direct the content of regulatory interventions –Letting providers know about likely regulatory intervention Consequences: timely, meaningful, predictive information and intelligence, capacity to use/interpret it

10. Compliance: inspection Four main models of inspection: –Driver for improvement in advance of inspection –Measure of compliance to support enforcement –Measure of performance to support improvement –Driver for other providers to improve Consequences: direct observation and proxy measures, reliability and validity of judgements, inspection workforce competence

11. Enforcement Four main models of enforcement: –Informal enforcement or prospect of enforcement –Incentive to drive compliance –Symbolic action to drive compliance –Driver for other organisations to achieve compliance Consequences: resource to enforce, scale of penalty, effects of frequency, spillover effects, capacity to change

12. Information provision Three main models of information provision: –Information to be used by other stakeholders in their decision making –Information to be used by providers in compliance and improvement –Information as a mechanism for public accountability Consequences: limited volume, utility and timeliness of data, usability by stakeholders

14. Summary of current regulatory model Registration as an administrative process Compliance –Standard setting to measure compliance and support enforcement –Information gathering and risk assessment to inform or direct regulatory intervention –Inspection and reporting to measure compliance Enforcement as an incentive for compliance Information provision to support decision making and accountability

15. Current regulatory model: key issues Consistent with a mission as a “safety-net” regulator focused mainly/only on compliance at the bottom of the performance distribution Some problems even so with the model –Regulatory standards - measurability and meaning –Attempts proportionate regulation without predictive information and capacity to use it –Inspection process – reach and workforce competence –Information provision – little useful information to share

16. Conclusions and reflections For CQC – a contribution towards their rethinking of their regulatory strategy/philosophy and model Programme theory evaluation – most useful (and most difficult) when theory(ies) in use unclear, ambiguous, complex/complicated Place for programme theory in regulatory design – not just post hoc evaluation Wider question of how best to integrate evaluation into regulatory innovation/policy development/practice