Applied Methodologies in PHI Session 5: Evaluation Kath Roberts (NOO/EMPHO)

Learning Objectives 1.To understand what is meant by evaluation 2.To increase awareness, through examples, of the use of evaluation in public health 3.To increase awareness of the process of evaluation

Evaluation What it is? Why do it Who is it for? How is it done?

What it is “Evaluation is concerned with assessing an activity against values and goals in such a way that results can contribute to future decision making and or policy” Tones & Tilford, 1994

What it is “Evaluation is attributing value to an intervention by gathering reliable and valid information about it in a systematic way, and by making comparisons, for the purpose of making more informed decisions or understanding causal mechanisms or general principles.” Ovretveit.J 1998

Stakeholders those involved in delivery and managing the service/intervention those served or affected commissioners/funders

Evaluation involves comparing e.g a group which gets the intervention v. one which doesn’t (controlled trial)... state of people, populations, organisations, services before and after an intervention... the achievements of intervention compared with the objectives at outset... audit is not the same as evaluation - focuses on processes against standards NOT on impacts

Reliable and valid information -Relate evaluation to aims & objectives (SMART!) -Type of evaluation: Formative, Process, Impact/Outcome -Evaluation design: Experimental, Quasi- experimental, Pre-experimental

Reliable and valid information -Data Collection Methods: direct and indirect measures, focus groups, interviews, questionnaires -Analyses and Reporting

What can you evaluate in public health? services - new or well-established one-off interventions policy changes screening programmes media campaigns surveillance systems outbreak investigations communication methods – newsletters, websites, etc IT systems and other tools a training course!... pretty much anything

Questions that evaluations can answer What impact is the intervention/service having on the community? Have service users/local community benefited? Are we meeting their needs? Are we reaching the right people? Who are we reaching? Who are we not reaching? Why? Are our services of a high standard? Do they provide good value for money? Which services are well regarded/not so well regarded? How well are we working with partners?

Why evaluate? If you can demonstrate results, you can win support and funding (or not) If you don’t measure results, you can’t tell success from failure If you can’t recognise failure, you can’t correct it If you can’t see success, you can’t learn from it

Case Study: HEALTH FOR LIFE

Towards an evaluation design – clarifying the customer’s needs 1.What is the purpose of the evaluation? 2.Who are you hoping to influence? 3.What outputs and outcomes will they value? 4.What activities are currently underway and what are you planning to do next? 5.What initial results do you expect from the project? (early wins) 6.What medium term outcomes do you expect from the project (i.e. in the next year/18 months)? 7.What long term outcomes do you expect from the project (i.e. after 5 years)?

Think about what information is needed for the evaluation Think about the different ways you can collect additional information that might be useful (?quantitative, ?qualitative) and try to avoid collecting data that will not be useful There is no right and wrong way to collect information Be creative

Think about people who you can collect information from Professionals Managers Frequent users Infrequent/sporadic users Partner organisations The local community – and its different component parts Politicians

Reporting the results Is there going to be a written report and / or other ways of reporting the findings of the evaluation? Will the findings need to be reported to different audiences using different formats? How can the clarity of the report be ensured? (it should use clear, plain language, and be honest about its findings!)

Some things to think about (1) Every intervention/service should be evaluated Try and build evaluation into the intervention/service as early as possible Be clear and transparent about why you are evaluating it Evaluation time and resources should be in proportion to the overall project Think about what future funders would want to know

Some things to think about (2) Involve people from the start – all stakeholders Keep it simple and (if possible) fun Make sure that everyone is using the same terminology Provide plenty of feedback... to stakeholders and participants Think qualitative as well as quantitative Share all outcomes - positive, negative, unexpected Share findings and learning as widely as possible

Useful additional reading National Obesity Observatory, Standard Evaluation Framework for weight management interventions. Nutbeam, D. & Nauman, A., Evaluation in a Nutshell: a practical guide to the evaluation of health promotion programs. Sydney: McGraw Hill Publishers Pawson, R. & Tilley, N., Realistic Evaluation. London: Sage Publications

Learning Objectives 1.To understand what is meant by evaluation 2.To increase awareness, through examples, of the use of evaluation in public health 3.To increase awareness of the process of evaluation