An evaluation framework… Strategic Planners Network 2 November 2018 Jacqui Henry

Overview What is evaluation? Why an evaluation framework? Framework development Introduce framework Evaluation example Summary 2

What is evaluation? The systematic determination of the quality, value or importance of something. Its purpose is to determine how well something is doing, its value, and if its worth doing more of.

Why an evaluation framework? Support greater consistency and efficiency in the way evaluation is utilised Strengths Data rich Strategic, analytical, technical skills Compliance driven Data collection for monitoring purposes

Development of an evaluation framework Start with those interested willing to engage in evaluation process - Strategy team Develop and test at a strategic level Make time to engage in process so that evaluation is meaningful to those involved

Method Overall Purpose 1 2 3 Setting up evaluation for success What is evaluation? What’s the evaluation purpose? Who will use the evaluation? 2 Develop key evaluation questions What do we want to know about quality, value and importance? What are the learnings going forward? 3 Agree on performance criteria and importance What’s valued? What matters? What are the needs we are trying to address?

Training programme evaluation Method Overall Purpose 1 Setting up evaluation for success Continuous improvement Whose involved - Social and Economic Science section and others 2 Develop key evaluation questions How well has the programme been delivered? How effective the programme is at realising key results? What are the learnings going forward? 3 Agree on performance criteria and importance Aspects of programme delivery and effectiveness of the programme

Method Overall Purpose 4 5 6 Develop a framework to assess performance What does quality and performance look like in this context? Good and poor? 5 Agree on credible evidence What data sources will be used? Do we have multiple sources for each criteria? 6 Design and implement data collection What methods of data collection are appropriate?

Training programme evaluation Method Overall Purpose 4 Develop a framework to assess performance Rating scale from excellent to poor in terms of quality of programme delivery, and overall effectiveness. 5 Agree on credible evidence Delivery – feedback forms, planning documents, attendance records of numbers Effectiveness – survey/interviews post training 6 Design and implement data collection As above plus planning

Method Overall Purpose 7 8 9 Analyse, compile, assemble and order the evidence What does each of the data sources show? How robust is the data? 8 Synthesis and sense making Who should be part of the evaluative reasoning process? What do we think about performance and why? So what? 9 Reporting, using and socialising results Now what? What’s the best way to frame the evaluation findings and conclusions? How do we share the results in a useful and meaningful way?

Training programme evaluation Method Overall Purpose 7 Analyse, compile, assemble and order the evidence What does each of the data sources show? What are the common themes? 8 Synthesis and sense making So what? SES come together and go through sensemaking / reasoning process? What do we think about performance on a scale of excellent to poor and why? 9 Reporting, using and socialising results Now what? Action to be taken from results. Share the results in a useful and meaningful way with staff and others.

Guidance document

In summary Think about evaluation planning early on Involve multiple sources of data Framework is an inclusive process Evaluation is about the quality, value and importance of something