1. East Lothian Learning Partnership Conference Demonstrating Impacts and Outcomes – Is it really all that important? Ann Kivlin & Colin Forbes, CLDS 5th December 2011

2. Session Aims Highlight some of the advantages of being able to demonstrate impacts and outcomes Explore: WHY we need to do it, WHEN we should do it, WHAT is required in order to do it effectively WHO has responsibility. Explore and discuss a range of tools and techniques

3. Highlight the advantages of being able to demonstrate impacts and outcomes
How confident are you?
Extremely confident
More than confident
Confident
Less than confident
Find it a challenge
Not confident at all

4. Highlighting the advantages of being able to demonstrate impacts and outcomes
Hands up how many of you feel confident enough to facilitate a session on this topic with your staff teams?

5. mins
Settle in and intro 5
Why, When, What, Who 40
Building your Capacity 10
Tools and Techniques 10
Questions 10

6. WHY we need to demonstrate impacts and outcomes
WHY we need to demonstrate impacts and outcomes? WHEN we should be demonstrating impacts and outcomes? WHAT is required in order to do it effectively and demonstrate impacts and outcomes? WHO has the responsibility for demonstrating impacts and outcomes?

7. Group Task: In 4 groups spend 5 minutes considering one of the questions and then feedback your groups 3 most important answers

8. WHY - A list we prepared earlier!
Value for money
Spending Public Pound
Accountability
Transparency
Measure improvement
Report progress to stakeholder
Proof the work makes a difference

9. WHEN - A list we prepared earlier!
When you feel like it!
For every piece of work?
Significant pieces of work
Best bits!
Worst bits?
Regularly?
When your line manager requests!

10. WHAT - A list we prepared earlier!
Ability to prove what you are doing is really making a difference
Use a range of effective tools
Ability to measure before and after and monitor performance
Agree what is good evidence
Provide the evidence
Confidence

11. WHO - A list we prepared earlier!
It’s everybody’s responsibility!
Lead person

12. Building your Capacity in demonstrating Impacts and Outcomes To increase staff understanding and confidence in monitoring and evaluating activities through constructing clear outcomes, indicators and methods which can demonstrate impact of their engagement SLIDES!!

13. Your Thoughts? What do you think when you hear the word ‘evaluation’?
Trainer's slides
Your Thoughts?
What do you think when you hear the word ‘evaluation’?

14. What are outcomes and why are they important? Discussion

15. The definition!
Outcomes are the changes or difference you want to make through your services or activities (outputs)

16. Why do outcomes matter?
Focus: You need to know what difference you want to make before you can evaluate to see if you have.
Expectations: The people who use your services will know what to expect from you
Motivation: Staff satisfaction is increased when they know they make a difference.
Accountability: Many funders want to fund or purchase outcomes. They want to know what difference their money makes and how it helps target groups have a better life.

17. The science bit
OutCOMEs are the changes and differences that come out of your activities.
OutPUTs are the activities or services you put on for your users.
Inputs are the resources you put in to do the work: money, people, time, premises, equipment.
input  output  outcome  impact

18. Outcomes are lazy Outcomes don’t happen by themselves
They need activities, services, people
The output makes the outcome happen
outcome
output
10.15

19. Relevance Outcomes must be relevant to the activity
What you do (activity) must link to your outcome in a logical way

20. GOOD words LESS SPECIFIC words e.g. improve e.g. support
DEFINITION
Outcomes are the changes or difference you want to make through your services or activities (outputs)
GOOD words LESS SPECIFIC words
 e.g. improve e.g. support

21. - about change or difference
Trainer's slides
Outcome words
decrease improve
maximise minimise
better expand
reduce enhance
more less
produce increase
strengthen
sustain
- not exhaustive list
- about change or difference

22. These words don’t show change!!
Trainer's slides
These words don’t show change!!
Support Assist
Engage
Encourage
Enable
Help Offering
Seeing
Watch out for verbs (doing words) – they are about activity not change
BUT you could include these words as long as you also describe the change e.g. –
‘INCREASE levels of community participation by OFFERING crèche facilities to local families’. 22

23. Expressing the outcome
Trainer's slides
Expressing the outcome
is changing (e.g. the service user, beneficiary, organisation, community etc.)
Who
is changing (e.g. knowledge, skills, environment, feelings etc.)
What
How
it will change (e.g. increase, improve, reduce, etc.)

24. Examples of outcomes Parents’ confidence is increased
Trainer's slides
Examples of outcomes
WHO WHAT HOW
Parents’ confidence is increased
Young people have reduced, stabilised or eliminated their drug use
Communities are better represented in decision making

25. Trainer's slides
Examples of outcomes
WHO WHAT HOW
Older Asian women have better access to community health and social care services
There is an increase in the understanding of the recycling procedure within the office staff.
Young carers are able to make informed choices about the future 25

26. Trainer's slides
The ‘so what’ test
I deliver an Employability Course at Ross High School - SO WHAT?
I work with a group of 16 S4’s - SO WHAT?
The group has visited colleges and taken part in indoor climbing sessions - SO WHAT?
The group have learned what courses are available at local colleges and how to access these courses  OUTCOME
The group have gained confidence and improved communication & team working skills through indoor climbing sessions  OUTCOME
All important … but not all outcomes

27. Activities and Services
Trainer's slides
Weaver’s Triangle
AIM
Outcomes = change
Skills
Knowledge
Behaviour  Feelings
… SO WHAT?
Outputs
Activities and Services
the main things you do with your users / clients

28. Outcomes and impact Lochniven Young Carers
Trainer's slides
Outcomes and impact
Lochniven Young Carers
Impact (aim) – ‘Our young people are successful learners, confident individuals, effective contributors and responsible citizens’
Outcomes need to be meaningful
and realistic steps on the way.

29. Example Lochniven Young Carers Project
Trainer's slides
Example
Lochniven Young Carers Project
Short term: Young carers know where to go for support & are confident to do so
Medium term: Young carers are less isolated and better able to cope
Longer term: Young carers participation and attainment at school is improved
Impact – Our young people are successful learners, confident individuals, effective contributors and responsible citizens

30. The outcome test Outcomes Are about change or difference.
Trainer's slides
The outcome test
Outcomes
Are about change or difference.
Don’t happen by themselves – they are not the activity – they are the result of the activity.
Must link logically to the activity.
Use words like improve, decrease.
Answer the ‘so what?’ question.
Should be realistic – achievable in a few years.
Should be (largely) in your power to deliver.
Should be simple – not too many (3-5)!

31. Collecting information
Trainer's slides
Collecting information
Monitoring what you do
(e.g. numbers, range of activities)
OUTPUTS
AND
Evaluating whether your activities
are delivering your OUTCOMES

32. Measuring outcomes Thinking about each outcome:
Trainer's slides
Measuring outcomes
Thinking about each outcome:
What would it look like if we achieved it?
These are your outcome indicators
What are the 2-3 most important (and most likely) indicators for each outcome?
Involve users and stakeholders.
Your indicator list =
your evaluation plan

33. Getting it right Measure indicators more than once to show change.
Trainer's slides
Getting it right
Measure indicators more than once to show change.
What is your baseline (the starting point) for the service and for individuals (from referral info, assessment, evidence of need etc)
Don’t collect information on all outcomes all the time.
Do you need to ask everyone or can you sample?
Follow up some users if you can to find out what happened next.

34. Getting it right 6. Use one system – but a range of methods.
Trainer's slides
Getting it right
6. Use one system – but a range of methods.
7. Build into planning – and if possible activities.
8. Test your methods.
9. Be honest and involve clients.
10.Record unexpected outcomes
(e.g. 6 volunteers got a job)

35. Common sources of info on indicators

36. Methods for collecting info
Trainer's slides
Methods for collecting info
Questionnaires
Interviews
Observation
Group structured feedback
Wall charts
Visual progress tools
Mapping
Third party feedback
(Video) diaries

37. Outcome Evaluation Plan
Indicators
Methods for collecting information
When to collect
info
Who is responsible
Older Asian women have better access to community health and social care services
Older Asian women know what community health & social care services are available in the local area
Older Asian women know how to access community health & social care services
Older Asian women receive more community health & social care services
One-to-one mapping with the women to find out what services they know about at the start & end of the 1 year project
Group interviews (focus groups) with the women
Individual interviews with the women
Interview a range of health and social care providers to find out if uptake of services has increased

38. The evaluation pathway
Getting Started - understanding what you want to achieve. Setting and agreeing your outcomes
Collecting information - what are you going to measure and how? Getting systems in place
Analysing and Reporting - making sense of what you’ve collected and reporting on progress and learning.
Learning from your findings - What did you learn? How will you do things differently? Any unexpected results?
Acknowledgement:
Evaluation Support Scotland

39. Tools & Techniques Wide range of evaluative tools
Trainer's slides
Tools & Techniques
Wide range of evaluative tools
Electronic Project Files
VOICE
National Standards for Community Engagement

44. Presentation adapted by

45. What is VOiCE?
VOiCE is online planning and recording tool designed to assist individuals and organisations to design and deliver effective community engagement.

46. Why use VOiCE? It’s simple to use
Provides evidence on meeting the National Standards for Community Engagement
You don’t need previous knowledge of the National Standards or experience of community engagement to use VOiCE
VOiCE provides tools to facilitate a dialogue with people
Online link to the National Standards for Community Engagement:

47. How can VOiCE help you? Analyse the need for community engagement
Plan your engagement
Do it – implement your plan effectively
Review the outcomes
VOiCE enables you to monitor and record the process
This can be carried out with a group or at a meeting. You can discuss with others and check that the information written down is what they have agreed on!

49. Features of VOiCE Multi user environment
Secure password protected accounts
Provides reports
Strategic analysis

50. Want support to use VOiCE?
VOiCE training is supported by East Lothian’s Community Engagement Strategy action plan which is available online at:
Key contact: Meriel Deans, Public & Community Involvement
Co-ordinator
This is a shared post between East Lothian Community Health Partnership and East Lothian Community Planning Partnership
Tel: or

51. Further discussion
10 minutes

52. Highlight the advantages of being able to demonstrate impacts and outcomes
How confident are you?
Extremely confident
More than confident
Confident
Less than confident
Find it a challenge
Not confident at all

53. Hands up how many of you now feel confident enough to facilitate a session on this topic with your staff teams?
Free copy of these slides for every correct answer!