1. Erica Wimbush Head of Evaluation, NHS Health Scotland Swiss Jubilee Symposium on Public Health Evaluations – Lessons Learned and Future Directions 17 June 2011, University of Berne Getting the most out of evaluation Reflections on achievements and challenges in commissioning and managing evaluations - A Scottish perspective

2. Scotland in Europe What is distinctive? What do we offer?  A small country in Europe  Strong tradition of public health  Focus: Health inequalities, deprivation, rurality  Public health located within NHS but strong emphasis on ‘partnership working’  Data collection of a global standard

3. The national public health agency 60 staff, €6.7 million300 staff, €30.3 million UK Devolution from 1997 – divergent roles Health Education Board for Scotland (1991-2002) (2003- present) Role: Provides specialist advice to Government on improving Scotland’s health and reducing inequalities and supports policy and programme implementation at local level

4. Health Scotland: Evaluation Team  Our mandate: ‘To coordinate the evaluation of health improvement policy in Scotland’ Health Improvement: The Challenge, Scottish Govt 2003  Our role: Enables a better understanding of the reach and impacts of policies and actions through outcome-focused planning, monitoring and evaluation. 3 main areas of work:  Commissioning and managing evaluations  Supporting others to do outcome planning, evaluation and performance reporting  Evaluation Summer School & workshops Key principle: Utilisation-focused evaluations

5. Public health evaluation 75% of evaluations of health improvement interventions are commissioned by Health Scotland, the Scottish Government or local NHS boards Health Scotland evaluations:  Policy evaluations  Programme evaluations  Local project evaluations

6. Achievements & Challenges

7. Improving Evaluation Quality What are the Quality issues?  Research design and methods  Timeliness and relevance for decision-making  Responsiveness, flexibility, creativity What worked?  Development of theory-based evaluation practice  Establishing a vibrant evaluation community  Explicit quality standards and competencies (work in progress) Continuing challenges  ‘Institutionalisation’ of evaluation – part of the culture but no independent budget or requirement

8. Independence vs Utilisation What is the issue?  Maintaining independent position viz a viz policy/program can be in tension with commitment to stakeholder involvement What worked?  Intermediary ‘bridging’ role Continuing challenges  Recognising the un-evaluable!

9. Evaluation Capacity & Culture What is the Capacity-building issue?  Lack of confidence and competence (practitioners)  Intelligent demand (funders)  No recognized training or qualification (evaluators) What did we try?  Workshops, training, toolkits  Importing talent from abroad  Most effective – policy move to outcomes management & accountability across public services Challenges  Extension of team support role and skills  Sustaining the gains if/when political environment changes

10. Commissioning &/or Coordinating? What is the issue?  Reduced funding available  Commissioning focus – single study perspective  Quality issues – making more of in-house expertise What worked?  Evaluation strategy – shift evaluation focus to policy  Coordination of a portfolio of inter-linked studies Continuing challenges  In-house capacity - Further extension of role  Internal infrastructure to support data collection and analysis

11. Future – Sharing practice What can we bring?  Theory-based evaluation practice  Vibrant evaluation community  Bridging role What do we need?  Qualifications  Standards & Competencies