Designing a measurement framework For Elfrida Rathbone Camden (ERC), a charity supporting young disabled people, carers and families in need. Rose Drummond and Will Tong (with quality assurance from Jane Parkin)

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

Introduction- About the charity Elfrida Rathbone Camden (ERC) supports young disabled people, families in need, parents of children with disabilities and young carers in Camden Small charity involved in diverse range of projects: Advocacy – for families of young carers Life skills courses - for young disabled people to bridge gap from adolescence to adulthood Support and advice – for families with a range of problems. Debt and welfare advice, family relationships, parenting, children’s emotional problems, education. Diverse range of funders for different projects Each project had its own monitoring and evaluation framework

Introduction- Project aim Exec wanted to measure the impact of the organisation as a whole Project aim: Develop measurement framework at organisational level This would bring together the diverse range of projects initially with: Different target populations with different needs Different funders Different monitoring and evaluation frameworks

Introduction - Project set-up The team: Alexis (Director of ERC) Will and Rose (pro-bono OR analysts) Jane (pro-bono quality assurance analyst) Set out scope and objectives in project initiation template Template provided by OR society

Reflection: 1. Get the right team together Director of charity: Meant we had buy-in from the top of the organization. Alexis also in touch with the work of the projects, and could tell us what was feasible across the organization. Experienced quality assurance analyst: Fantastic to tap into the experience of someone who has run many such projects before.

Reflection: 2. Agree realistic objectives and scope up front We split the project in half - excluded funding applications. Needs to be realistic given that the work is being fitted into volunteer’s spare time.

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

PHASE 1 – Background research 1. Understanding the charity Charity Director explained about the charity and what they do Access to charity’s current monitoring and evaluation documents, funding proposals and project records. Mapped out the indicators used on different projects – very different! Charity website and mission statement

Reflection: 3. To understand the charity, if you can, visit in person. For us this was not practical, so we used alternative sources.

PHASE 1 – Background research 2. Researching frameworks used in the charity sector Lots of online resources! Structure: Outcomes star (already used by ERC) ASCOT (same as outcomes star, but outcomes weighted) Content: Outcome indicator banks - the evaluation cookbook for getting feedback from children in the form of a game.

PHASE 1 – Background research From our research: We realised it was unclear what outcomes to measure- the projects were too diverse to simply combine existing outcome frameworks. We chose Charities Evaluation Service template to structure the framework – flexible step-by-step guide to defining outcomes, indicators, collection methods and frequency.

PHASE 1 – Background research Source: Charities Evaluation Service Monitoring and Evaluation Framework Template

PHASE 1 – Background research Source: Charities Evaluation Service

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

PHASE 2 – engaging with the charity We agreed that we needed to go back to the charity and work with them to agree which were the important outcomes to measure. Wrote plan and timetable.

Reflection: 4. Beware of scope creep! Full of ideas after research stage – danger of scope becoming too ambitious!

PHASE 2 – engaging with the charity Data collection – key outcomes Initially just going to ask executive committee in a meeting. Jane (QA) had 2 suggestions: Collect views of all stakeholders (also staff, volunteers, trustees, service users, funders) in a workshop. First ask questionnaire - to inform workshop planning and get people thinking.

PHASE 2 – engaging with the charity Questionnaire Four variations of questionnaire, one for each group of respondents ERC charity exec/trustees Funders Service users Staff/volunteers Service users one went on the Facebook page, and was short and simple. Director ed funders who he had personal relationship with Went to over 30 people – 4 responses (including 1 very detailed response)

Reflection: 5. Get charity input – they know your target audience Minimize burden on respondents Right approach for each target group (Facebook, , phone call) Director was sure funders would be too busy to answer questionnaire – he was right!

PHASE 2 – engaging with the charity Workshop In a facilitated 3 hour workshop agreed key outcomes among 12 people representing: ERC charity exec/trustees Service users Staff/volunteers A key success of the project – unique opportunity to hear from all these groups and get buy-in, agreement and joint ownership.

PHASE 2 – engaging with the charity "If ERC could only achieve one thing for its service users, what should it be?" Agreed main outcomes: Improved mental health Increased opportunity to access employment, volunteering, training Increased access to support and social networks Reduced risks Cultural shift - the community benefitting from the contribution of disabled people

Reflection: 6. Running a workshop. Get the right people involved – personal approach from charity Questionnaire – got people thinking about key themes in advance, helped plan structure of workshop, gave us something to fall back on if room was silent. Plan, plan, plan –templates, needs of participants, extra activities Ice breaker – to get people talking Work in pairs – one facilitates, the other writes notes. Assigning discussion groups – ideal if someone who knows everybody assigns people to suitable groups Group tasks should have clear objective and time limit – to keep discussion focused. Be flexible.

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

PHASE 3 – Finalising detail and write-up Analysts filling in the gaps in different colours so that you could see the source: Workshop Questionnaire ERC current measurements Rose/Will’s input- using tested questions where possible (such as ONS well- being questions) Meeting with Director and 2 staff members. To remove overlap from outcomes, and pick out most important details to measure. QA – incorporate comments from Jane (ie measuring numbers or % increase, measuring too many things with one indicator)

Overview Introduction to the charity and purpose of project PHASE 1 – background research PHASE 2 – engaging with the charity PHASE 3 – finalising the detail and write-up Reflections on project

Reflections - Working with young people Workshop –avoid technical language and explain the language being used (as you should with any non-technical audience) Final framework – if asking users to assess their own progress against an indicator, questions must be easy to understand and answer.

Reflections on working pro-bono Great way to learn, meet people and make a positive impact! Choose a project/topic that you have a real interest in Be realistic about the time you are able to give up, and be prepared for long gaps if people go on holiday etc. Consider whether distance will be a hindrance Having an experienced analyst for QA was fantastic

Any comments or questions? Contact details for Rose Drummond:

PHASE 3 – Finalising detail and write-up