Independent evaluation: the experience of undertaking policy evaluations commissioned by the Department of Health in England Nicholas Mays Director, Policy Innovation Research Unit Department of Health Services Research & Policy Workshop on ‘New approaches to health policy evaluation: what role for independence?’, Congress of the Swiss Evaluation Society (SEVAL), 4 September 2015 Improving health worldwide

What does ‘independence’ mean in research and evaluation? Means different things to different participants (academics, think tanks, commercial agencies, not-for-profits) Can be defined in terms of structure, source of finance (e.g. endowment, government, charitable, commercial), ideology, ethos, ability to choose what to study and how to approach it (intellectual freedom) How independence is ‘framed’ can affect degree of influence I see it as always a relative concept, along a series of continua – always a (different) set of constraints (e.g. performance metrics in universities including to raise research funding) or incentives/disincentives & rewards

DH Policy Research Programme A national programme of independent research since 1970s to provide evidence that can be used by DH officials to advise their Ministers Based on simple ‘customer-contractor’ model, mostly uses university researchers Three main modes of (competitive) funding: – Policy Research Units, usually 5 years, potentially renewable – initiatives, consisting of groups of linked, commissioned projects – single commissioned projects

How DH typically commissions evaluations 1.Detailed invitation to tender inviting proposals from external research teams 2.One or two-stage proposal process 3.External peer reviews and comments from DH policy ‘customers’ 4.Response to reviewers’ reports 5.Assessment of proposals by commissioning panel of officials and academics, usually independently chaired 6.Recommendation to minister of successful team (and/or negotiation of refinements of original proposal) 7.Contract, including rights to publish after 28-day notice/comment period, with maximum 3-month delay – ‘Publication of scientifically robust research results is encouraged’ month process if two stages

Assessment criteria RELEVANCE of the proposed research to the research brief QUALITY of the research design QUALITY of the work plan and proposed management arrangements STRENGTH of the research team IMPACT of the proposed work VALUE for money (justification of the proposed costs) INVOLVEMENT of patients and the public

Assumptions underpinning the DH approach External, independent evaluation is superior to the alternative Objectivity is a realistic and important goal Conflicts of interest should be minimised by ensuring that evaluators are not involved in what they are studying – No action-research or ‘engaged’ form of evaluation Emphasis on (mainly positivist) ‘science’ Centrality of peer review at all stages Evaluation can remain relevant, timely and applicable while showing the above features – no tensions between ‘independence’ and applicability DH and Ministers are the dominant ‘customers’

Advantages Evaluation not subject to day to day interference from government – formal rules of engagement Evaluators can report what they find Evaluators can focus on research skills rather than being expected to be implementers, management consultants, etc. Disadvantages DH can comment in ways that may put researchers under pressure to self-censor – researchers have to determine their own balance between being ‘helpful’ and ‘unresponsive’ Requires expert intermediaries Works best with a stable ‘customer’ group Experience of the DH approach from the point of view of different participants

Advantages Value for public money in that findings are published Degree of ‘independence’ maintains the value of using external evaluators – possible to provide some challenge function Disadvantages Can appear slow & cumbersome to busy policy officials – Researchers can be seen to be ‘remote’ Still tensions when findings do not correspond to policy makers’ expectations Accusations of ‘policy-based evidence’ since researchers do not choose topics Experience of the DH approach from the point of view of different participants

Advantages Government can easily distance itself from unpalatable findings Disadvantages Can be used to maintain a façade of EBP – ‘independence’ no guarantee of greater use of findings – can undermine engagement with policy-relevant issues Ignores 60 years’ experience of more ‘engaged’ forms of evaluation Local implementers may not see value of ‘national’ evaluations – may not cooperate Experience of the DH approach from the point of view of different participants

Different stakeholder perspectives on pilots and evaluations Other Government departments HM Treasury Local stakeholders, e.g. politicians, frontline staff; providers; users/patients Academic community, e.g. peer reviewers, commentators