Evaluation Emma King

Evaluation is not at heart about collecting evidence to justify oneself, nor about measuring the relative worth of courses and teachers. It is about coming to understand teaching in order to improve student learning. Ramsden, 1992 p 241

Learning Outcomes By the end of the session you should be able to: Identify the purpose for evaluation Select appropriate evaluation strategies for your own context Recognise what makes effective an survey or focus group Discuss how you might make use of evaluation data

Evaluation Process Value Judgement Measurement Monitoring Action Butcher, C., Davies, C. and Highton, M. (2006, p189)

Evaluation Process TEACH FEEDBACK & REFLECT THINK PLAN Kolb (1984)

Planning an Evaluation Identify the purpose of the evaluation Identify and give priority to the constraints under which the evaluation will take place Plan the possibilities for an evaluation within these constraints Agree an evaluation design

Considerations WHY do you want to evaluate? WHAT do you want to evaluate? WHEN is the best time to conduct the evaluation? WHO will you get evidence from, and who is it for? HOW will you do it and how will you act on the outcome?

Why? To improve practice To ensure student needs are being met To demonstrate the value of the service and secure future budget To determine if we are meeting agreed quality standards and targets

What? Objectives model Are objectives being met? Experimental Is the innovation working? Decision making Which of the two strategies is most suitable? Illuminative What’s going on? Morrison (1993)

When? Diagnostic Check the needs of a particular student cohort Formative Find out how the current cohort are getting on Summative Inform the next iteration of support

Who? Self Students – current students and alumni Colleagues – within the department and beyond Manager

Kirkpatrick’s Four Levels of Evaluation Level 1 Reaction how do the target audience respond? Level 2 Learning what attitudes, knowledge and skills are acquired? Level 3 Behaviour to what extent do participants apply what they have learnt? Level 4 Results the impact of the training on the organisation or environment?

Methods Considerations Focus Groups Questionnaires Semi-structured interviews Observation Practicality Objectivity Reliability Validity

Survey Questions How old are you? 15-20 20-30 30-40 40-50 50-60 How effective was that session? Was the support offered timely, appropriate and making use of appropriate resources? Would your student group respond well to... ?

Writing good survey questions Only ask questions directly related to what you’re evaluating Keep the language clear and unambiguous Vary the format of questions to encourage students to think about their responses Order your questions logically Avoid yes/no questions Remember you can use an even number of options to encourage students to make a decision

Interviews and Focus Groups Consider how you will record the data Design questions to elicit reflection and allow participants to express their own point of view Ask questions which will encourage group interaction and discussion Consider how you will manage the group discussion Have a strategy ready for if the conversation goes off at a tangent

Interpreting Feedback What does it tell us? What weight should we give to different questions? How do we respond? Who’s responsibility is it to respond?

Closing the Loop After an evaluation make sure that you feedback to all involved about how you’ve made use of the data and what’s happening as a result

Pathway approach - IMPACT levels (levels 0 - 4) Cause Effect Researcher Training & Dev. Level 1 Reaction Level 2 Learning Level 3 Behaviour Results Level 4 Outcomes Level 0 Foundations Input OUTPUTS Benefits OUTCOMES Lead to

IMPACT Level 4 - Potential outcomes - Final results of the training and development activity Complexity – number of factors economic growth personal transformation B D social & cultural capital qualification rates skilled workforce Better quality research A C researcher engagement in training spin-outs and start-ups by trained researchers Time after training & development activity

IMPACT Level 4 - Potential outcomes - Final results of the career development activity Complexity – number of factors graduate gets job Career management skills Career adaptability Economic growth Student takes action B D social & cultural capital student engagement in career development activity A C Time after career development activity