East Lothian Learning Partnership Conference Demonstrating Impacts and Outcomes – Is it really all that important? Ann Kivlin & Colin Forbes, CLDS 5th December 2011

Session Aims Highlight some of the advantages of being able to demonstrate impacts and outcomes Explore: WHY we need to do it, WHEN we should do it, WHAT is required in order to do it effectively WHO has responsibility. Explore and discuss a range of tools and techniques

Highlight the advantages of being able to demonstrate impacts and outcomes How confident are you? Extremely confident More than confident Confident Less than confident Find it a challenge Not confident at all

Highlighting the advantages of being able to demonstrate impacts and outcomes Hands up how many of you feel confident enough to facilitate a session on this topic with your staff teams?

mins Settle in and intro 5 Why, When, What, Who 40 Building your Capacity 10 Tools and Techniques 10 Questions 10

WHY we need to demonstrate impacts and outcomes WHY we need to demonstrate impacts and outcomes? WHEN we should be demonstrating impacts and outcomes? WHAT is required in order to do it effectively and demonstrate impacts and outcomes? WHO has the responsibility for demonstrating impacts and outcomes?

Group Task: In 4 groups spend 5 minutes considering one of the questions and then feedback your groups 3 most important answers

WHY - A list we prepared earlier! Value for money Spending Public Pound Accountability Transparency Measure improvement Report progress to stakeholder Proof the work makes a difference

WHEN - A list we prepared earlier! When you feel like it! For every piece of work? Significant pieces of work Best bits! Worst bits? Regularly? When your line manager requests!

WHAT - A list we prepared earlier! Ability to prove what you are doing is really making a difference Use a range of effective tools Ability to measure before and after and monitor performance Agree what is good evidence Provide the evidence Confidence

WHO - A list we prepared earlier! It’s everybody’s responsibility! Lead person

Building your Capacity in demonstrating Impacts and Outcomes To increase staff understanding and confidence in monitoring and evaluating activities through constructing clear outcomes, indicators and methods which can demonstrate impact of their engagement. 27 SLIDES!!

Your Thoughts? What do you think when you hear the word ‘evaluation’? Trainer's slides Your Thoughts? What do you think when you hear the word ‘evaluation’?

What are outcomes and why are they important? Discussion

The definition! Outcomes are the changes or difference you want to make through your services or activities (outputs)

Why do outcomes matter? Focus: You need to know what difference you want to make before you can evaluate to see if you have. Expectations: The people who use your services will know what to expect from you Motivation: Staff satisfaction is increased when they know they make a difference. Accountability: Many funders want to fund or purchase outcomes. They want to know what difference their money makes and how it helps target groups have a better life.

The science bit OutCOMEs are the changes and differences that come out of your activities. OutPUTs are the activities or services you put on for your users. Inputs are the resources you put in to do the work: money, people, time, premises, equipment. input  output  outcome  impact

Outcomes are lazy Outcomes don’t happen by themselves They need activities, services, people The output makes the outcome happen outcome output 10.15

Relevance Outcomes must be relevant to the activity What you do (activity) must link to your outcome in a logical way

GOOD words LESS SPECIFIC words e.g. improve e.g. support DEFINITION Outcomes are the changes or difference you want to make through your services or activities (outputs) GOOD words LESS SPECIFIC words  e.g. improve e.g. support  

- about change or difference Trainer's slides Outcome words decrease improve maximise minimise better expand reduce enhance more less produce increase strengthen sustain - not exhaustive list - about change or difference

These words don’t show change!! Trainer's slides These words don’t show change!! Support Assist Engage Encourage Enable Help Offering Seeing Watch out for verbs (doing words) – they are about activity not change BUT you could include these words as long as you also describe the change e.g. – ‘INCREASE levels of community participation by OFFERING crèche facilities to local families’. 22

Expressing the outcome Trainer's slides Expressing the outcome is changing (e.g. the service user, beneficiary, organisation, community etc.) Who is changing (e.g. knowledge, skills, environment, feelings etc.) What How it will change (e.g. increase, improve, reduce, etc.)

Examples of outcomes Parents’ confidence is increased Trainer's slides Examples of outcomes WHO WHAT HOW Parents’ confidence is increased Young people have reduced, stabilised or eliminated their drug use Communities are better represented in decision making

Trainer's slides Examples of outcomes WHO WHAT HOW Older Asian women have better access to community health and social care services There is an increase in the understanding of the recycling procedure within the office staff. Young carers are able to make informed choices about the future 25

Trainer's slides The ‘so what’ test I deliver an Employability Course at Ross High School - SO WHAT? I work with a group of 16 S4’s - SO WHAT? The group has visited colleges and taken part in indoor climbing sessions - SO WHAT? The group have learned what courses are available at local colleges and how to access these courses  OUTCOME The group have gained confidence and improved communication & team working skills through indoor climbing sessions  OUTCOME All important … but not all outcomes

Activities and Services Trainer's slides Weaver’s Triangle AIM Outcomes = change Skills Knowledge Behaviour  Feelings … SO WHAT? Outputs Activities and Services the main things you do with your users / clients

Outcomes and impact Lochniven Young Carers Trainer's slides Outcomes and impact Lochniven Young Carers Impact (aim) – ‘Our young people are successful learners, confident individuals, effective contributors and responsible citizens’ Outcomes need to be meaningful and realistic steps on the way.

Example Lochniven Young Carers Project Trainer's slides Example Lochniven Young Carers Project Short term: Young carers know where to go for support & are confident to do so Medium term: Young carers are less isolated and better able to cope Longer term: Young carers participation and attainment at school is improved Impact – Our young people are successful learners, confident individuals, effective contributors and responsible citizens

The outcome test Outcomes Are about change or difference. Trainer's slides The outcome test Outcomes Are about change or difference. Don’t happen by themselves – they are not the activity – they are the result of the activity. Must link logically to the activity. Use words like improve, decrease. Answer the ‘so what?’ question. Should be realistic – achievable in a few years. Should be (largely) in your power to deliver. Should be simple – not too many (3-5)!

Collecting information Trainer's slides Collecting information Monitoring what you do (e.g. numbers, range of activities) OUTPUTS AND Evaluating whether your activities are delivering your OUTCOMES

Measuring outcomes Thinking about each outcome: Trainer's slides Measuring outcomes Thinking about each outcome: What would it look like if we achieved it? These are your outcome indicators What are the 2-3 most important (and most likely) indicators for each outcome? Involve users and stakeholders. Your indicator list = your evaluation plan

Getting it right Measure indicators more than once to show change. Trainer's slides Getting it right Measure indicators more than once to show change. What is your baseline (the starting point) for the service and for individuals (from referral info, assessment, evidence of need etc) Don’t collect information on all outcomes all the time. Do you need to ask everyone or can you sample? Follow up some users if you can to find out what happened next.

Getting it right 6. Use one system – but a range of methods. Trainer's slides Getting it right 6. Use one system – but a range of methods. 7. Build into planning – and if possible activities. 8. Test your methods. 9. Be honest and involve clients. 10.Record unexpected outcomes (e.g. 6 volunteers got a job)

Common sources of info on indicators

Methods for collecting info Trainer's slides Methods for collecting info Questionnaires Interviews Observation Group structured feedback Wall charts Visual progress tools Mapping Third party feedback (Video) diaries

Outcome Evaluation Plan Indicators Methods for collecting information When to collect info Who is responsible Older Asian women have better access to community health and social care services Older Asian women know what community health & social care services are available in the local area Older Asian women know how to access community health & social care services Older Asian women receive more community health & social care services One-to-one mapping with the women to find out what services they know about at the start & end of the 1 year project Group interviews (focus groups) with the women Individual interviews with the women Interview a range of health and social care providers to find out if uptake of services has increased

The evaluation pathway Getting Started - understanding what you want to achieve. Setting and agreeing your outcomes Collecting information - what are you going to measure and how? Getting systems in place Analysing and Reporting - making sense of what you’ve collected and reporting on progress and learning. Learning from your findings - What did you learn? How will you do things differently? Any unexpected results? Acknowledgement: Evaluation Support Scotland

Tools & Techniques Wide range of evaluative tools Trainer's slides Tools & Techniques Wide range of evaluative tools Electronic Project Files VOICE National Standards for Community Engagement

Presentation adapted by

What is VOiCE? VOiCE is online planning and recording tool designed to assist individuals and organisations to design and deliver effective community engagement.

Why use VOiCE? It’s simple to use Provides evidence on meeting the National Standards for Community Engagement You don’t need previous knowledge of the National Standards or experience of community engagement to use VOiCE VOiCE provides tools to facilitate a dialogue with people Online link to the National Standards for Community Engagement: www.scdc.org.uk/what/national-standards/10-national-standards/

How can VOiCE help you? Analyse the need for community engagement Plan your engagement Do it – implement your plan effectively Review the outcomes VOiCE enables you to monitor and record the process This can be carried out with a group or at a meeting. You can discuss with others and check that the information written down is what they have agreed on!

Features of VOiCE Multi user environment Secure password protected accounts Provides reports Strategic analysis

Want support to use VOiCE? VOiCE training is supported by East Lothian’s Community Engagement Strategy action plan which is available online at: www.eastlothiancommunityplanning.org.uk/engagement Key contact: Meriel Deans, Public & Community Involvement Co-ordinator This is a shared post between East Lothian Community Health Partnership and East Lothian Community Planning Partnership Email: meriel.deans@nhs.net Tel: 0131 536 8025 or 07738 075232

Further discussion 10 minutes

Highlight the advantages of being able to demonstrate impacts and outcomes How confident are you? Extremely confident More than confident Confident Less than confident Find it a challenge Not confident at all

Hands up how many of you now feel confident enough to facilitate a session on this topic with your staff teams? Free copy of these slides for every correct answer!