Monitoring and evaluation of carers’ services and projects Dr Andrea Wigfield - Associate Professor of Social Policy Centre for International Research on Care, Labour and Equalities (CIRCLE), University of Leeds Michael Shann – Carer Ambassadors Programme Manager, Carers UK Supporting Carers in Primary Care National Learning Event, Manchester, 16 April 2013

Why evaluate? For learning and development For accountability To demonstrate value for money to supporters Motivation of staff and volunteers To save time and money

Who is measuring impact? 74% of charities say they are investing more in impact measurement than five years ago Change in funder requirements and priorities of trustees/senior management are main reasons for increasing impact measurement BUT, main benefits are in improving services and being able to better demonstrate results (Source: Making an Impact (Oct 2012), New Philanthropy Capital

What do we want to find out? Are we achieving our planned outputs and outcomes? If not, why not? What has worked well and not so well? Are we reaching the carers and other stakeholders we want to? What do carers think of our services? How can we improve them?

Monitoring & Evaluation Terms Monitoring – routine collection of data Evaluation – using data we collect to make judgements about our projects and services Inputs – resources we put into projects & services Outputs – products and services we actually deliver Outcomes – changes and benefits we achieve Impact – effect at a higher level, often the result of contributions from several organisations Quantitative – numbers of things that happen Qualitative – people’s views and perceptions Indicators – measurements we can use to determine whether outputs and outcomes are being achieved

The Logic Model InputsOutputsOutcomesImpact

The 5 Step Approach to Monitoring and Evaluation What are your aims? What are your objectives? What are your performance indicators? Monitoring – how well are you doing? Evaluation – what change have you made? Source: First Steps in Monitoring & Evaluation (2002), Charities Evaluation Services

Identifying outputs & outcomes Usually straightforward e.g. number of carers visiting centre, calling, attending training courses Identifying Outputs Usually more difficult e.g. increase in carers accessing benefits, improved understanding of carers’ rights Identifying Outcomes

Identifying indicators (which data to collect) Keep it in proportion – to size of project / service Keep it manageable - time needed to collect data Quantitative and qualitative Consider existing information gathering systems – do these need adapting Where are the gaps? What new data needs to be collected?

Group 1: What do you currently measure and how do you collect the data? a)What outputs do you measure and how? b)What outcomes do you measure and how? c)What difficulties have you experienced in measuring outputs and outcomes?

Group 2: Evaluation Challenge A GP Liaison Worker has provided a carer awareness training session to a group of Practice Managers in their area. The main aim of this session was to inform the group of local carers’ services that are available. What output and outcome data would you gather to evaluate whether the session was successful?

Group 3: Evaluation Challenge A GP Champion (GP recruited by RCGP) and Carer Ambassador (volunteer recruited by Carers UK) have been advising a Clinical Commissioning Group on how to set up a Carers Partnership Board in their area. What output and outcome data would you gather to evaluate whether their contribution was successful?

Group 4: Evaluation Challenge Carers Trust have set up a new Facebook group for GP Liaison Workers to encourage the sharing of information, ideas and good practice across the country. How would you evaluate the benefits of having this Facebook group?

Group 5: Evaluation Challenge A GP Liaison Worker has established links with 12 GP practices in their area. Some of the GP practices have made changes to the ways they identify and support carers and refer them to local carers’ services. How would you measure the value of the GP Liaison Worker’s intervention?

Internal or External Evaluation? Factors to consider: How important is it to have an independent assessment of your project/service? Has your funder requested an external evaluation? What proportion of your project/service budget can you allocate to evaluation? What is appropriate for the size/budget of your project/service? How much staff time will be involved?

Reporting to Donors When applying for funding, ask about reporting requirements Include donor requirements in your evaluation plan Donors will be looking for evidence, but you could demonstrate progress using qualitative data as well as numbers An interim report could focus more on outputs with some indication of progress towards outcomes Learning from the project may make you want to make changes to the original objectives – this may be fine as long as you deliver the intended outcomes

Evaluation Good Practice Write an evaluation plan when you are planning your project or service ‘Proportionate and appropriate’ – don’t measure everything Build impact measurement into job descriptions and appraisals Take action on your results Develop a commitment to continuous improvement Communicate your impact Don’t be afraid to evaluate

Contact Andrea or Michael