1. Introduction to Outcomes Planning & evaluation Contribution Analysis H
Introduction to Outcomes Planning & evaluation Contribution Analysis H.McIntosh & S.Wilson 20/10/15
Adapted in part from material prepared by John Connolly, UWS . W

2. agenda
10:00 Welcome & Introductions 10:10 Overview of contribution analysis 10:40 Quiz Participative demonstration: evaluating the impact of applying knowledge to practice—Clinical Enquiry & Response Service (CLEAR) 11:15 Coffee 11:30 Group work 12:30 Overview of performance reporting 13:00 Lunch 13:45 Performance Reporting 15:40 Next steps 16:00 Close

3. objectives By the end of the session you will be able to:
List the key features of contribution analysis
Appreciate how an outcomes planning approach facilitates evaluation for learning and continuous improvement as well as accountability
Explain the outcomes chain to a colleague
Outline a basic evaluation framework (outcomes chain and monitoring & evaluation plan) for a project relevant to your own area of work

4. Attribution and Contribution
Questions about cause and effect are critical to evaluating impact
Attribution analysis
Did the intervention cause the observed results?
Contribution analysis
Has the intervention contributed to the observed results?
Has the intervention made a difference?
Outcomes Planning & Evaluation (OPE) is the process of undertaking contribution analysis

5. What is OPE?
A pragmatic approach to plan and evaluate the impact of programmes that operate in complex settings
Focuses on outcomes – does our programme make a difference?
Participative – involves stakeholders
Uses theory of change to map out the expected pathways to the intended outcomes within a time-based framework
Uses evidence to identify effective pathways
Facilitates evaluation for learning and continuous improvement, not just accountability
Supports performance reporting focussed on outcomes; the difference the programme is making

6. Theory of change The theory of how and why an intervention works
Explains how a programme, as designed, will theoretically result in the intended outcomes
Shows the plausible links between the programme’s activities and subsequent outcomes
Identifies underlying assumptions
Takes account of the context in which interventions work–how external factors may affect results

7. Monitoring & Evaluation
OPE Process
Current Situation
Outcome Plan
Outcome Planning
Theory of Change
Monitoring & Evaluation
Gather evidence & develop performance story
Gather more evidence
Performance reporting & sharing learning
Performance Story

8. Tools for OPE Outcomes chain Outcome planning Theory of change
Monitoring & Evaluation Plan
Developed from the outcomes chain
Measuring success—identifying indicators
Outcomes Focussed Performance Reporting template

9. The difference you make
OUTCOMES CHAIN
SPHERES OF INFLUENCE
Direct influence
Direct control
Indirect influence
What do you do?
The difference you make
Expected impacts
Resources you need
What you do
What you produce
Who you reach
Changes in knowledge, skills, awareness
Changes in behaviour, practice
Changes in health outcomes
Inputs
Activities
Outputs
Reach
Short term outcomes
Medium term outcomes
Long term outcomes
Political, technological, socio-economic, environmental, other factors
Organisational resources, skills, systems
Existing practices and capacity
External influences 9

10. Outcome types
Short term—usually associated with changes in knowledge, skills, attitudes, awareness
Changed attitudes to alcohol and drinking
Librarian’s acquire skills in evidence synthesis to support Practice Based Small Group Learning (PBSGL)
Medium term—usually associated with changes in behaviour and practice at an individual or organisational level
Reduced alcohol-related violence and abuse
Librarian’s provide evidence search & synthesis service support to PBSGL
Long term—aspirational outcome, the overarching goal
Safer and happier families and communities
More widespread application of evidence-based practice

11. Assumptions, Risks and competing explanations
What assumptions do you make to get from one step in the outcomes chain to the next?
What are the risks that getting from one step to the next won’t happen as you imagine?
Any barriers you might anticipate?
How might you mitigate these risks?
What are the other influencing factors?
Other programmes, policies, services
Social, political factors
Enablers as well as barriers
What alternative explanations could there be for how the intervention works?

12. Partner contributions to shared outcomes
Inputs
Activities/ Outputs
Reach
Short-term Outcomes
Medium-term outcomes
Long-term Outcomes
National/Local Outcomes
Culture
Practice & Behaviours
Environments
Partner 1
Partner 2
Partner 3
Outcomes chain 12 12

13. Cross-sector partner contributions – Alcohol
Long-term Outcomes
Improved mental wellbeing Reduced inequalities in healthy life expectancy Reduced inequalities in alcohol-related deaths and hospital admissions
Behaviour Reduced alcohol consumption levels Less drunkenness; less drink-driving
Environments Physical: Reduced exposure to alcohol-related hazards Economic: Reduced availability/affordability of alcohol Social: Drunkenness less attractive; sensible drinking the norm
Medium-term outcomes
Vol orgs
Detox, Intensive support
Addiction services
Adults with alcohol problems
Increased sobriety & stability
Police
Enforcement of drink driving laws
Random breath testing
Drivers
Increased detection rate
Local authorities
Enforcement of planning controls & licensing laws
Enforcement actions
Licensed trade
Compliance with laws
SG, UK govts, EU
Industry regulation
Taxation,displays, promotions, advertising
Alcohol industry
Increased price Reduced incentives
Service uptake & engagement
Short-term Outcomes
Understanding risks, attitudes to drinking
Hazardous and harmful drinkers
Reach
General public - targeted
Preventive services
Sensible drinking messages
Activities & Outputs
Screening & Brief advice
Media campaigns
Inputs
Scottish Govt
NHS
Outcomes chain

14. Definitions recap
Outputs are things produced as a result of programme activities
Outcomes are the benefits or results of a programme—the difference the programme makes
the changes that occur in the short, medium and long term as a result of the programme’s activities and outputs
Outputs are not outcomes

15. OPE Process: Monitoring & Evaluation
Current Situation
Outcome Plan
Outcome Planning
Theory of Change
Monitoring & Evaluation
Gather evidence & develop performance story
Gather more evidence
Performance reporting & sharing learning
Performance Story

16. Monitoring & evaluation plan
Developed from the outcomes chain
Define indicators to monitor progress and measure success
Identify how and from where you will gather the evidence
Agree who has responsibility for data gathering/storage
Establish the analysis plan: how, when, who?

17. indicators Outcome indicators Reach Reaction
Outcomes are the benefits or results of a programme—changes that occur in the short, medium and long term
Indicator is the measure of the extent to which the outcome is being achieved
Reach
Measures of who was engaged/involved
Reaction
Measures of people’s reaction to the engagement
Resource use, activities undertaken, outputs produced
Measures of the extent to which the programme was delivered as intended

18. Defining indicators
What evidence do you need to be able to report meaningfully on the impact of your programme?
Involve key stakeholders
Consider what data are already collected!
Indicators need to be
Measurable—recorded and analysed quantitatively or qualitatively
Precise—understood in the same way by all stakeholders
Sensitive and Consistent
What is ideal versus what is feasible
Do I have to evidence everything?
Do I need balancing measures?
Is hard data better than soft data?
How many indicators do I need?
Can I use proxy indicators? 18

19. Example: K2A support for PBSGL
Outcomes
Development of PBSGL modules routinely involves support from knowledge services
Relationship between librarians and practitioners is strengthened
Indicators
Increase in proportion of modules produced that draw on knowledge services’ evidence search & synthesis service
Change in librarians’ perception of their working relationship with practitioners as a result of their involvement in module production
Qualitative evidence that practitioners value collaboration with knowledge services

20. Monitoring & Evaluation Plan
OUTCOMES CHAIN
PROGRESS MEASUREMENT
Indicators
Data source
Responsibility
Long-term outcomes
Medium-term outcomes
Short-term outcomes
Reach
Outputs
Activities
Inputs

21. Quiz & participative demonstration

22. Impact of a clinical enquiry service
For each step in the chain, what will success look like?
NHS staff in primary care or remote and rural areas
Evidence informed clinical decisions
Improved patient care
Useful and relevant service which saves practitioner time
Improved understanding of diagnosis, prognosis & therapies
Web based evidence summaries answering clinical enquiries
Develop service: processes, train staff, develop website
Team of information professionals and researchers

23. Impact of a clinical enquiry service
For each step in the chain, what will success look like?
Team of information professionals and researchers
Develop service: processes, train staff, develop website
Web based evidence summaries answering clinical enquiries
NHS staff in primary care or remote and rural areas
Quality service which saves practitioner time
Improved understanding
Informed clinical decisions
Improved care

24. Assumptions, Risks, Indicators….
Practitioners value evidence based ways of working
Staff indicate their practice was informed by the information provided.
Number of requests received.
Number of visits to website (reuse).
Number of repeat users.
Visits to website.
Staff state they would recommend service and saves time.
Staff indicate they received new information or resources.
External factors override ability to apply new knowledge
Inaccurate information reported in summaries
Number of requests by professional group or health board
Practitioners will apply new learning to patient care

25. Assumptions, Risks, Indicators….
Practitioners value evidence based ways of working
Practitioners will apply new learning to patient care
Inaccurate information reported in summaries
External factors override ability to apply new knowledge
Number of requests received.
Number of visits to website (reuse).
Number of requests by professional group or health board
Number of repeat users.
Visits to website.
Staff state they would recommend service and saves time.
Staff indicate they received new information or resources.
Staff indicate their practice was informed by the information provided.

26. OPE Process: performance reporting
Current Situation
Outcome Plan
Outcome Planning
Theory of Change
Monitoring & Evaluation
Gather evidence & develop performance story
Gather more evidence
Performance reporting & sharing learning
Performance Story

27. Outcomes focussed Performance reporting
Performance report focussed on what difference you made
Template to help set out your report
A headline about your work
What you were trying to do and why
Your activities/outputs
Your results
Other factors/challenges
Learning
Adaptable to brief and longer formats for different audiences
How often should we prepare a performance report?

28. summary
OPE is a pragmatic, participative and outcomes focussed approach to plan and evaluate the impact of programmes that operate in complex settings
Enables credible assessment of the contribution a programme makes to observed results
Facilitates evaluation for learning and continuous improvement as well as accountability

29. The difference you make
OUTCOMES CHAIN
What do you do?
The difference you make
Expected impacts
Resources you need
What you do
What you produce
Who you reach
Changes in knowledge, skills, awareness
Changes in behaviour, practice
Changes in health outcomes
Inputs
Activities
Outputs
Reach
Short term outcomes
Medium term outcomes
Long term outcomes
Handout only 29

30. SPHERES OF INFLUENCE External influences Direct influence
Direct control
Indirect influence
Inputs
Activities
Outputs
Reach
Short term outcomes
Medium term outcomes
Long term outcomes
Handout only
Political, technological, socio-economic, environmental, other factors
Organisational resources, skills, systems
Existing practices and capacity
External influences 30