Evaluation: what is it? Anita Morrison

What is evaluation? Evaluation… –…is the process of determining the merit, worth, or value of something, or the product of that process. (Scriven) – uses a range of research methods to systematically investigate the effectiveness of interventions, implementation and processes, and to determine their merit, worth or value in terms of improving the social and economic conditions of different stakeholders (Government Social Research)

Easier as a set of questions? ………. What is the need we are trying to address? What specifically needs to change? How will we know if change has taken place? What will we actually do? How will we make sure we're doing it as planned? How successful have we been and what have we learned? What now needs to change? LEAP

Clarity of aims and objectives for any intervention – crucial! Identify needs or problem you are addressing Clarify inputs (staff, equipment, volunteers etc) Decide on activities / outputs Nail the outcomes you want to achieve Tools to do this

Weaver’s triangle

A Basic Results Chain 6 7. End results 7. What is our impact on ‘ends’? 6. Practice and behavior change 6. Do we influence [behavioural] change? 5. Knowledge, attitude, skill and / or aspirations changes 5. What do people learn? Do we address their needs? 4. Reactions 4.Are clients satisfied? How do people learn about us? 3. Engagement / involvement 3. Who do we reach? Who uses / participates? 2. Activities and outputs 2. What do we offer? How do we deliver? 1. Inputs 1. How much does our program cost? ($, HR etc) Program (Results) Chain of Events (Theory of Action) Key Questions Source:Adapted from Claude Bennett Taken from Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, Thousand Oaks, California, 1997, p 235. Indirect Influence Direct Influence Control WHY? WHAT? WHO? HOW?

Quick exercise: Input, output or outcome? Public information campaign Decline in HIV incidence Peer educators Increase in condom use Quality of care improvements Trained staff for sexual health clinics

Approaches to evaluation Focus of evaluation Key evaluation questionsEvaluation approach DeliveryHave we implemented the intervention / service effectively? How can we improve? Process ObjectivesHave we achieved our outcomes? What has worked / not worked? Outcome Problems/needsHave we made an impact on the problems and needs that provided the rationale for intervention? What has worked / not worked? Impact

Process evaluation Purpose: – to assess programme design/ coverage/ delivery Key questions: –have problems/needs been diagnosed correctly? –has the intervention been implemented effectively? –are the intended beneficiaries being reached? –what lessons can we learn to improve intervention? Common sources of information: –Administrative records/management information –Observation –Surveys: beneficiaries, staff, partners –Case study research

Evaluation of Free Personal Care ( ) Aim: to assess the operation of free personal care across Scotland and identify possible improvements Methods: Interviews with national organisations Survey of 32 LAs Telephone survey of 1000 members of the public Postal survey of unpaid carers 6 case studies Stakeholder workshop

Outcome evaluation Purpose: –assess degree to which objectives / intended outcomes achieved Key questions: –have intended beneficiaries actually benefited? –have we achieved our intended objectives/outcomes? –have we produced any unintended effects (beneficial/adverse)? –to what extent are the observed changes due to the intervention (and would not have occurred without it)?

Central Heating Programme: Assessing Impacts on Health ( ) Aim: assess whether the Scottish Executive Central Heating Programme has had a positive impact on recipients health How was it done? COMPARATIVE: The evaluation compared the experiences of a group of 1,281 households which received heating under the Programme with those of a 'comparison' group of 1,084 households not included in the CHP LONGITUDINAL - collected data on respondents' experiences at multiple points in time. First Interview; postal questionnaire; Final interview

CONCLUSION: CHP was associated with reduction of problems with cold, damp and mould, with higher levels of heating in the home, with feeling warmer in winter, and with greater level of satisfaction with heating. It was not clearly or consistently associated with direct impacts on the cardio- respiratory health of recipients, nor with reductions in NHS service use. It is possible that these direct health impacts will become visible over a longer period of time.

Exercise

Resources Evaluation Support Scotland Learning, Evaluation and Planning Barnardos