1. Introduction to Evaluation Techniques
Puja Myles

2. Module outline Session 1: Introduction to evaluation techniques
Session 2: Evaluation design and implementation
Session 3: Participatory approaches to evaluation
Session 4: Economic evaluation: the basics
Session 5: Economic Evaluation: types and costs
Session 6: Economic Evaluation: Outcomes and uses
Session 7: Prioritisation exercise

3. Module Learning Outcomes
At the end of this module you should be able to:
Recognise the basic principles of healthcare evaluation
Identify the key features of different types of economic evaluation
Distinguish between various evaluation types and select the appropriate type for a given scenario
Choose an appropriate outcome measure for a given evaluation question
Propose a suitable study design for a given evaluation problem
Design an evaluation drawing on the theory of change and logic model
Apply evaluation principles to prioritisation of healthcare

4. Module Assessment Write a protocol for an evaluation of either:
A new or existing service you work with OR
2. One of the examples provided
Your evaluation can look at the whole service or focus on particular aspects. It MUST include some economic evaluation, as well as looking at the effectiveness and quality of the service

5. What is Evaluation?
At its most basic, evaluation is concerned with assessing whether interventions are effective…

6. Defining Evaluation-1
“The purpose of evaluation is to measure the effects of a program against the goals it set out to accomplish as a means of contributing to the subsequent decision-making about the program and improving future program-making”
-Weiss (1972)
Feedback from students: Do you think this is good definition? Does this give you an insight into what evaluation is?

7. Defining Evaluation-2
“The aim of evaluation is to contribute towards solving practical problems, in terms of what works and why. It is about collecting information to inform action. Most of all it is about learning from experience.”
-Springett (2001)
2-minute reflective exercise for students: What aspirations for evaluation are revealed in these definitions?

8. Types of evaluation- 1
Ovretveit (1998) categorised evaluation depending on when the evaluation is carried out in the healthcare intervention planning cycle:
Formative/developmental evaluation
Process evaluation
Summative evaluation

9. Types of evaluation- 2
Green and South (2006) outline different models of evaluation based on the purpose of the evaluation:
Goal-oriented evaluation: effectiveness, efficiency and economy of an intervention
Decision-orientated evaluation: improve decision-making
Evaluation research: providing explanations for outcomes
Responsive evaluation: process of evaluation and perspectives of evaluation
Goal-free evaluation: Openness to achievements other than those prescribed by the intervention’s aims and objectives
Two more models proposed by Green and South (2006):
Alternative explanations evaluation focussing on alternatives to accepted descriptions about what is happening
Utilisation-orientated evaluation: Utility of findings to different stakeholders
2-minute reflective exercise for students: Why do you think evaluation research is considered a different category? Isn’t all evaluation ‘research’?

10. Evaluation, Research or Audit?
‘The use of evaluation within a research context is distinct from its use within other management activities such as quality assurance and monitoring. These are applicable only to a particular context, whereas research aims to be generalisable across different contexts’
-Smith et al (2005)
What is the relationship between evaluation and research?
Research usually seeks to prove a hypothesis and an important focus is generalisability of results to other contexts. Evaluation tends to be project or context specific leading to project improvements and results are not always applicable in other contexts. Some evaluations can be research and vice versa.

11. Audit Useful in quality assurance of a process
Compare practice against a set of standards
When you know that the process will predict outcomes provided it is in keeping with the accepted standards
Example: Audit of prescribing (you know the drug works provided prescribing guidelines are followed)
An audit should not be treated as an interesting academic exercise. Audits are powerful quality management tools- once gaps are identified by means of an audit, it is imperative to draw up an action plan and carry out a repeat audit at a specified time period to assess if improvement has occurred. This complete cyclical process is called the audit cycle or the audit loop.

12. And what about Monitoring?
‘Emphasis on recording what has happened in terms of programme delivery, whereas evaluation is concerned with assessing what has been achieved and how any changes have come about’
-Green and South (2006)
Reflection: Reminiscent of audits?
Monitoring as defined by Feuerstein (1986): ‘the systematic and continuous following, or keeping trace, of activities to ensure they are proceeding according to plan’.
Monitoring data can be collated at some point to answer questions like: how well are we doing, are we doing things right, what difference are we making?
Evaluation seeks to answer these questions. An evaluation should be used as an instrument of meaningful change-to improve processes or outcomes. Evaluation is also important for accountability: has the project worked? Has the money been well spent? Should this project continue?

13. How are evaluation questions commonly framed?

14. Efficacy, effectiveness and efficiency*
Efficacy- does an intervention work in ideal conditions?
Effectiveness-does an intervention work in real life?
Efficiency-ratio of useful output to total input; incorporates the notion of minimising waste. Often phrased as ‘value for money’ or ‘return on investment’
Context is important in health promotion- we need to move beyond ‘does this work’ and ‘what works’ to ‘what works for whom in what circumstances?’- Nick Tilley on realistic evaluation.

15. External vs. Internal evaluators?
What do you think?

16. How can you make an evaluation meaningful for practice?

17. Stakeholder involvement
Identify key stakeholders
Recognise potential conflict of interests (multi-agency funded projects)
Specify aim of evaluation (e.g. to assess overall improvement in a given health outcome; or quality of service; or value for money…)
It is important to define the various stakeholders and specify “whose impacts will be measured”. Evaluators should recognise that there may be a conflict of interests among the goals of different parties (Borus). It may be that some programme goals are shared by all stakeholders (for e.g. an overall improvement in a given health outcome), but other goals (for e.g. quality of service) may be viewed and indeed defined differently by different stakeholders.

18. Measuring health outcomes

19. Influence of Health Perspectives 1
Biomedical Model of Health
Health = Absence of disease (pathology)
‘Functionalist’ approach
Objective, numerical measurements based on ability to function
Preconceived assumptions about individual perceptions of health
The biomedical model views health in terms of the absence of disease and emphasises objective, numerical measurements based on physical and chemical data. The functionalist approach bases measurements on the ability to function, whether to perform personal, social or economic roles. Examples of functionalist measures include instruments such as the Sickness Impact Profile (SIP) and the Short-Form 36 (SF-36). While the SIP focuses on observable behaviour, the SF-36 also incorporates subjective health perceptions (van Straten et al, 1997). However, both these instruments are nomothetic, i.e., they make preconceived assumptions about how individuals perceive health related quality of life. Moreover, nomothetic measures make no allowances for variations in how different individuals perceive health (Bowling, 1997).

20. Influence of Health Perspectives 2
Social Model of Health
‘Disease’ and ‘Illness’ different concepts
WHO definition of Health
‘Hermeneutic’ approach to measure aspects of health unique to individuals
Open-ended questions and qualitative
The social model of health on the other hand, distinguishes between disease as underlying pathology and illness as the subjective perception of sickness or ill health. The WHO definition of health reflects the social model of health by emphasising that health is not merely the absence of disease, but a state of complete physical, mental and social wellbeing (WHO, 1946).
Hermeneutic approaches use structured and semi-structured approaches to allow for the measurement of those aspects of health-related quality of life that are unique to the individual. One of the ways this is achieved is by employing open-ended questions. These are termed idiographic measures. However, there is a third approach related to the hermeneutic approach, which is taken by phenomenologists. Phenomenologists tend to favour pure qualitative methods to evaluate health status (Bowling, 1997).
A study looking at the impact of oral health on life used both hermeneutic and functionalist approaches:
Employing the hermeneutic approach, open-ended questions were utilised. Respondents were asked simply:
• 'In what way does the condition of your teeth, gums, mouth or false teeth reduce your quality of life (negative effects)?' and
• 'In what way does the condition of your teeth, gums, mouth or false teeth add to your quality of life (positive effects)?'.
Using the functionalist approach, respondents were asked: first, to rate 'What effect, if any, does the condition of your teeth, gums, mouth or dentures have on your* (*each of the 16 key areas such as eating, appearance, speech, social relationships etc.)?' − 'bad − none − good'.

21. Qualitative versus Quantitative
- quantifiable
reliable
generalisable (? possibly)
Decontextualises processes
Qualitative
-provides context
-understanding of the ‘whys’
-generalisability is a major issue
Qualitative research involves analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact). Quantitative research involves analysis of numerical data.
Local authority councillors like stories (case studies as a political tool) but ‘numbers count’ when it comes to commissioning decisions, bid applications etc.

22. Health Outcomes- examples
Health Behaviours
Adherence to treatment
Specific Diseases
Depression
Emotional balance or well-being
Global health status using EQ-5D, SIP or SF-36
Self-reported health status

23. Health Outcomes? Self-confidence Self-efficacy Social involvement
Sense of achievement/direction
Stress relief
Participation in social networks

24. Introducing EQ-5D Standardised measure of health status
Generic measure of health
Can be used to generate quality of life measures
2-page self-completion questionnaire
Need to register study with EuroQol:
May have licensing fees depending on study
Do you think this is a good measure?
Class Exercise: Fill out EQ-5D questionnaire and score it
Disadvantages of generic health outcome measures: while these encapsulate the ultimate aim of public health i.e. health as a whole, they are not very useful for improving services or identifying areas of practice that need to be addressed. For e.g. if a psychiatry service was evaluated in terms of patient health outcomes, it wouldn’t tell the service what was good in their provision and what they needed to work on.

25. Introducing the SF-36
Shorter variants like the SF-20 and SF-12 are available
Well-validated
Can be used to generate quality of life measures
Is available free in the public domain from RAND
Class Exercise: Fill out SF-36 questionnaire and score it
Feedback:
How easy was this to complete and score?
How could you use such an instrument? (Before and after measures- mean difference or compare mean scores to population values)