1. Workshop 6 - How do you measure Outcomes? www.yhtphn.co.uk/win

2. “ if you don’t measure it, people will know you are not serious about delivering it!!” (Duxbury 2005) www.yhtphn.co.uk/win How do you measure outcomes

3. Listen to others and value the diversity of opinions in the group Be constructive Value the differences; there are no rights or wrongs, no good or bad Be open and honest Keep to agreed time, especially start and finish Mobile phones! (extracted from, Working with Groups. General Improvement Skills. Modernisation Agency) www.yhtphn.co.uk/win Ground Rules

4. appreciation of evaluation methods skills to evaluate their practice or any changes to practices that have been implemented This is not a session on research techniques. www.yhtphn.co.uk/win Programme Itinerary

5. What is evaluation? Why evaluate? Including evaluation within Project Planning Involving stakeholders in evaluation Methods End report and dissemination Putting evaluation into practice Complex Interventions Evaluation Evaluation SOS www.yhtphn.co.uk/win Contents

6. The means of systematically determining the extent to which a planned intervention or programme achieves predetermined objectives (World Health Organisation) a way of: o assessing whether objectives have been achieved o assessing the outcomes of projects o learning from successes and mistakes www.yhtphn.co.uk/win Definition of evaluation

7. Have the aims been met? Was it cost effective? Funding Accessibility for target population Measuring impact/outcomes Process www.yhtphn.co.uk/win Why Evaluate

8. Consider the following: Evaluation should be an ongoing part of planning and development – not an afterthought Evaluation should be part of learning and sharing successes and difficulties Evaluation should involve all stakeholders and be relevant to them Who is going to conduct each stage of the evaluation What type of evaluation is needed ie internal/external Are there any ethical issues www.yhtphn.co.uk/win Including Evaluation within Project Planning - 1

9. What do you want to know? How are you going to measure this? Clarify what information already exists? What information do you need to collect? What do you need to be able to measure this? How will we learn from our experiences en route? Are there milestones we need to reach as part of the evaluation of the project? www.yhtphn.co.uk/win Including Evaluation within Project Planning - 2

11. What ethical issues should you be considering? o Data sensitivity o Confidentiality o Informed consent o Storage of information o Insure there is no coercion o Respect www.yhtphn.co.uk/win Ethical issues

12. A stakeholder is anyone who is affected by or who can influence the impact Primary stakeholders: people who are directly affected by the initiative (either benefiting from it or adversely affected by it) Secondary stakeholders: all others who have a stake or interest in the initiative www.yhtphn.co.uk/win Involving Stakeholders within Evaluation - 1

13. Stakeholders are: People who use the service/s People who live in the area Staff and volunteers Partner agencies Funding bodies Local and national policy or decision-makers Do you think all stakeholders have the same level of influence or importance ?…. www.yhtphn.co.uk/win Involving Stakeholders within Evaluation - 2

14. Example of a basic stakeholder analysis High importance High influence Low influence Service users on steering group Vols Local Services Staff Service users Local community Funders National Gov’ Vols Local Services Local community Funders National Gov’ Local Gov’ Low importance High influence Low importance Low influence

15. You have been working with a community organisation that have been delivering support and advice to mums with post natal depression. This involves a wide range of social, mental health and other advice. Your funding is coming to an end. You are seeking further funding, but need to conduct an evaluation of what is currently being delivered www.yhtphn.co.uk/win Scenario 1

16. You are involved in a multi disciplinary falls clinic for older people. This involves providing advice and support to people identified at risk of falling. Your manager has asked ‘if it is working’ www.yhtphn.co.uk/win Scenario 2

17. Who What power do / might they have Who is most important What might they want to know from an evaluation What level of involvement should they have? www.yhtphn.co.uk/win Exercise 1- Identify the potential stakeholders

18. www.yhtphn.co.uk/win Coffee

19. Methods e.g. What type of data? www.yhtphn.co.uk/win

20. Methods e.g. has the project improved access to services for its target group? Evaluation questionsPossible measures/methods Do more people access services? take-up rates treatment records self report through questionnaires survey of attendance levels Are people more aware of services? questionnaire/survey of target group interviews with target group Do people feel more able to use services? focus groups on views of service users individual interviews with service users & non service users questionnaires Do professionals feel more people are accessing their services? telephone interview with professional questionnaire to professionals www.yhtphn.co.uk/win

21. Methods e.g. has the project increased community involvement in services? Evaluation questionsPossible measures/methods Are more people having a say in the way services are run? interviews focus groups questionnaires number of people on advisory board/steering groups levels Do people feel more involved in the way services are run? focus groups interviews questionnaires pictures drama Do people have power and influence in this project? attendance rates influence on decisions membership of steering group/advisory board www.yhtphn.co.uk/win

22. For your scenario What might be the key evaluation questions? Thinking of your stakeholders – what types of data / information might you be interested in seeking out What methods can you use to do this Who might be responsible for doing this and at what stage How might you disseminate your findings www.yhtphn.co.uk/win Exercise 2

23. Who is the evaluation for? How effective was the process? Have the planned outcomes been achieved? Is it cost effective? Who is the audience for the end product? What role do service users have within the evaluation? Who is going to do the work? What is the timescale? What format is the end product e.g. a report. www.yhtphn.co.uk/win The evaluation planning checklist

24. “ A Health indicator is a characteristic of an individual, population, or environment which is subject to measurement (directly or indirectly) and can be used to describe one or more aspects of the health of an individual or population (quality, quantity and time)” (WHO 1998) Some evidence is easier to collect than others which can make it tempting to collect what is collectable regardless of how useful/meaningful it may be. www.yhtphn.co.uk/win Indicators

25. “A change in the health status of an individual, group or population which is attributable to a planned intervention or series of interventions, regardless of whether such an intervention was intended to change health status” (WHO 1998) Crucially important that you consider which outcomes  you may want  able to measure. A hierarchy of outcomes. www.yhtphn.co.uk/win Outcomes

26. Consider What outcomes do you need to measure? What are the most important and are they measurable? Attribution - Can you be certain that the outcome is as a result of your intervention? For complex/long term interventions it is unlikely to show impact quickly www.yhtphn.co.uk/win Outcome measures / indicators

27. For your scenario Pick the indicators and outcomes that you might choose to evaluate your intervention www.yhtphn.co.uk/win Exercise 3

28. Public Health’s interventions are often complex with multiple components and influences Boundary-less – many different activities undertaken by many stakeholders in the name of the intervention Complex social change depends on many environmental factors often beyond reach of intervention www.yhtphn.co.uk/win In summary

29. Choosing Health - strategy for public health. It is focused on improving lifestyles – with a view to reducing the incidence of chronic diseases and improving life expectancy. eg Smoke-free agenda Discuss www.yhtphn.co.uk/win Example of a complex intervention Is Choosing Health

30. www.yhtphn.co.uk/win Evaluation SOS