Evaluation Nicola Bowtell

Some myths about evaluation 2Presentation title - edit in Header and Footer always involves extensive questionnaires means employing expensive consultants is about counting everything should be tagged onto the end of a project always has to involve a control group

What is evaluation? “Evaluation is attributing value to an intervention by gathering reliable and valid information about it in a systematic way, and by making comparisons, for the purpose of making more informed decisions or understanding causal mechanisms or general principles.” Ovretveit.J 1998 “Evaluation is concerned with assessing an activity against values and goals in such a way that results can contribute to future decision making and or policy” Tones & Tilford, Presentation title - edit in Header and Footer

More definitions… Judgement based on careful assessment and critical appraisal (WHO 1981) Making a judgement about the value of something by looking critically (Ewles and Simnett 1995) Concerned with making judgements about the value of activities (Downie, Tannahill and Tannahill 1996) Identifying and ranking criteria (value and aims) and gathering information which makes it possible to assess the extent to which criteria are being met (Perberdy 1997) 4Presentation title - edit in Header and Footer

Monitoring, Evaluation, Research..? Monitoring:  the process of appraising and assessing work activities through performance monitoring.  WHAT? How are we doing so far? Have we done what we said we’d do? Evaluation:  a formal and systematic activity where assessment is linked to original intentions / outcomes and is fed back into the planning process.  SO WHAT? What difference have we made? Research: a formal and systematic activity where the intervention, method and context of activity are constructed by the researchers. 5Presentation title - edit in Header and Footer

Why bother with Evaluation? Evaluation and evidence of outcomes are essential for sustainability and learning. If we don’t evaluate…. 6Presentation title - edit in Header and Footer Projects won’t know whether they are making a difference or not Projects may become ineffective or irrelevant Projects may lose funding if they can’t show results To inform practice and contribute to the evidence base To optimise use of resources Projects won’t know how to improve their work To ensure ethical practice

What can we evaluate in public health..?  Services / projects  screening programmes  surveillance systems  policy changes  media campaigns  outbreak investigations  communications – e.g. newsletters, websites  IT systems  a training course ... pretty much anything! 7Presentation title - edit in Header and Footer

Questions evaluation can help answer  What impact is the service having on the community?  Have service users benefited? Are we meeting their needs?  Are we reaching the right people? Who are we not reaching?  Do services provide good value for money?  What do people think of services?  How well are we working with partners? 8Presentation title - edit in Header and Footer

Quantitative data Quantitative  How many? Where? In what proportion?  Breadth  ‘hard’ data Acquired from:  existing sources e.g. demographic data, Service records  using new data collection tools e.g. Surveys, Assessment tools 9Presentation title - edit in Header and Footer

Qualitative data  Views, experience & opinions  Why? How? What for?  Depth  ‘soft’ data Acquired by conversation, observation or written accounts from stakeholders e.g. :  Service users  Service staff or Volunteers  Other agencies  Funders 10Presentation title - edit in Header and Footer

To be useful, outcomes must…  Be in line with national, local and programme priorities  Make sense to service users  Be measurable  Lie within the scope of the service 11Presentation title - edit in Header and Footer

Example Outcomes 12Presentation title - edit in Header and Footer People and communities… have improved mental well-being e.g. Looked after children have improved social networks. are more physically active e.g. Older adults increase their weekly participation in moderate physical activity. eat more healthily e.g. Young parents are able to prepare healthy meals

Steps towards a robust evaluation design... Types of evaluation:  Formative - during a project’s development stage  Process - implementation and delivery  Impact/Outcome - changes for individuals / communities 13Presentation title - edit in Header and Footer

Process evaluation Development Intervention Outcomes 14Presentation title - edit in Header and Footer Aims to provide an explanation of how or why intended outcomes of the project were (or were not) brought about.

Benefits and costs of process evaluation Benefits Can show if you’ve done what you said you’d do What worked well What didn’t Improve scheme by using learning Demonstrate progress to stakeholders Can inform practice 15Presentation title - edit in Header and Footer

Outcome evaluation Focuses on the various impacts of a project over time 16Presentation title - edit in Header and Footer

Outcome evaluation: challenges Size of effect Gains may be modest because: The size of the target population The nature of outcome measures and endpoints The scope of the intervention The timescale Attributing outcome to intervention 17Presentation title - edit in Header and Footer

Study designs  Post-intervention  Pre-post (Before and after)  Quasi-experimental  Experimental (e.g. RCT) 18Presentation title - edit in Header and Footer

Quantitative methods Collect numerical data.. Surveys Existing quantitative data sources e.g. HES Mathematical modelling 19Presentation title - edit in Header and Footer

Qualitative methods Interviewing - structured/semi-structured - individual - group (focus group) Observation - participant observer - documentary and textual analysis - official documents, historical records, newspapers Case studies 20Presentation title - edit in Header and Footer