1. Shared evaluation: NHS funders
@OllieCarrington
@ImperialCharity
Oliver Carrington
Impact & Evaluation Manager, Imperial Health Charity
London Funders, 21 May 2019

2. You often see these statistics….
71% of grantees think our support was excellent
3,167 direct beneficiaries of our grants
68% of strongly agree that their well-being improved
Nearly three quarters of young people progressed into employment
8 of 10 service users increased in confidence
This year twice as many improved literacy skills
…but what do they really mean?

4. An NHS Charity, place-based funder for five West London hospitals
Supporting patients, NHS staff & local communities
Grantmaking, arts, volunteering & fundraising
Shared measurement, for us is an additional layer of evaluation
Piloted from 2016 with NPC’s help, current round of data analysis in progress

5. What is shared measurement?

6. Shared measurement is when more than one organisation comes together to reach a common understanding of what to measure together
& how to do it
Benefits include:
Thinking as a sector about how it achieves social change (whatever their size)
Reducing duplication in developing bespoke tools
Increased transparency in methods used & results
Finding out what works
Building trusting relationships with peers

7. Five steps to shared measurement
Understand your impact
Develop shared outcomes
Use common tools
Use common analysis methods
Share & compare results

8. Used by charities…
And funders….
Also grantee perception surveys….

9. What does shared measurement mean for funders…
Grantee
Grantee
Grantee
as part of a group of 9 NHS funders

10. How are NHS funders doing it?

11. Who is involved?

12. The group varies widely But similar enough for this to work
Annual grant expenditure
Number of grants
Hospital size
Team size within the charities
Evaluation expertise & capacity varies
But similar enough for this to work
Similar goals
Similar activities
Similar grantees
Similar service users
Funding similar interventions

13. Steps one & two: a shared theory of change
Understand your impact
Develop shared outcomes
Use common tools
Use common analysis methods
Share & compare results
A theory of change is a diagram that shows a charity’s path from activities to outcomes to impact
Reveals a shared understanding of the sector & co-creates shared outcomes that forms the basis of the tools

14. Maddox shared theory of change

15. Steps one & two: common tools
Understand your impact
Develop shared outcomes
Use common tools
Use common analysis methods
Share & compare results
Common tools such as grant reporting templates & surveys
Enables us to collect data in the same way, to make data comparable (eg, same wording of questions)
Co-developed at workshops to overcome different requirements—vital that the tools are high quality

16. Common tools: grant reporting
Same categories for all grants:
Location (in hospital or community)
Primary beneficiaries (patients, visitors, staff, etc…)
Grant types (equipment, training, well-being, non-clinical)
Same grant statistics collected:
Grant value
Grant duration
Number of direct & indirect beneficiaries
Same template used to collect this data

17. Common tools: survey data
Surveys types:
Patient support activities
Buildings, equipment, etc…
Staff training & conferences
Staff welfare activities
What was measured & tracked:
Impact on provision of care
Well-being
Hospital experience
Awareness of the NHS funder
Quality
Net promotor scores

18. How we use the data?

19. Steps one & two: common analysis & using results
Understand your impact
Develop shared outcomes
Use common tools
Use common analysis methods
Compare & share results
Common methods to analyse & compare data so it can be used for learning.
The latest phase will analyse:
Details of £7m in grants (700 grants)
300 completed surveys for 23 projects

20. What does the sector look like & how do we fit?
% budget spent per grant category
The sector is research & innovation focused
Also true for us—but higher spend on buildings & hardship grants
*Example data

21. What is the average & how does this compare to us?
Q: How much has [the activity] improved your well-being?
Similar % of patients had well-being improved very much or a bit improved
But for us less of this was very much—room to learn!
*Example data

22. The five steps to shared measurement
Understand your impact
Develop shared outcomes
Use common tools
Use common analysis methods
Share & compare results

23. Challenges & future hopes?

24. Challenges 1
Engaging others: communicating benefits of evaluation 2
Compromises: before & after surveys were not possible 3
Coordination: keeping everyone engaged takes effort 4
Administration: time-consuming data collection 5
Continuous improvement: requires working together 6
Competition: not true for our sector, but it is a valid concern for others that compete with peers for funding

25. What comes next? 1 2 3 4 5 Analysis & learning stage
Sharing the results (published report) 3
Scaling the project further to more NHS funders (an online menu of tools) 4
Making the data anonymous to aid benchmarking 5
Using the data to share best practice & improvements across the NHS funder sector

26. Where to find out more? Oliver.Carrington@ImperialCharity.org.uk
3x Inspiring Impact reports
NPC report ‘NHS Charity Evaluation Guidance’
**New** NHS Charities shared measurement report out in the summer