1. Evaluation
Emma King

2. Evaluation is not at heart about collecting evidence to justify oneself, nor about measuring the relative worth of courses and teachers. It is about coming to understand teaching in order to improve student learning.
Ramsden, 1992 p 241

3. Learning Outcomes By the end of the session you should be able to:
Identify the purpose for evaluation
Select appropriate evaluation strategies for your own context
Recognise what makes effective an survey or focus group
Discuss how you might make use of evaluation data

4. Evaluation Process Value Judgement Measurement Monitoring Action
Butcher, C., Davies, C. and Highton, M. (2006, p189)

5. Evaluation Process
TEACH
FEEDBACK & REFLECT
THINK
PLAN
Kolb (1984)

6. Planning an Evaluation
Identify the purpose of the evaluation
Identify and give priority to the constraints under which the evaluation will take place
Plan the possibilities for an evaluation within these constraints
Agree an evaluation design

7. Considerations WHY do you want to evaluate?
WHAT do you want to evaluate?
WHEN is the best time to conduct the evaluation?
WHO will you get evidence from, and who is it for?
HOW will you do it and how will you act on the outcome?

8. Why? To improve practice To ensure student needs are being met
To demonstrate the value of the service and secure future budget
To determine if we are meeting agreed quality standards and targets

9. What? Objectives model Are objectives being met?
Experimental Is the innovation working?
Decision making Which of the two strategies is most suitable?
Illuminative What’s going on?
Morrison (1993)

10. When? Diagnostic Check the needs of a particular student cohort
Formative
Find out how the current cohort are getting on
Summative
Inform the next iteration of support

11. Who? Self Students – current students and alumni
Colleagues – within the department and beyond
Manager

12. Kirkpatrick’s Four Levels of Evaluation
Level 1 Reaction how do the target audience respond? Level 2 Learning what attitudes, knowledge and skills are acquired? Level 3 Behaviour to what extent do participants apply what they have learnt? Level 4 Results the impact of the training on the organisation or environment?

13. Methods Considerations Focus Groups Questionnaires
Semi-structured interviews
Observation
Practicality
Objectivity
Reliability
Validity

14. Survey Questions How old are you? 15-20 20-30 30-40 40-50 50-60
How effective was that session?
Was the support offered timely, appropriate and making use of appropriate resources?
Would your student group respond well to... ?

15. Writing good survey questions
Only ask questions directly related to what you’re evaluating
Keep the language clear and unambiguous
Vary the format of questions to encourage students to think about their responses
Order your questions logically
Avoid yes/no questions
Remember you can use an even number of options to encourage students to make a decision

16. Interviews and Focus Groups
Consider how you will record the data
Design questions to elicit reflection and allow participants to express their own point of view
Ask questions which will encourage group interaction and discussion
Consider how you will manage the group discussion
Have a strategy ready for if the conversation goes off at a tangent

17. Interpreting Feedback
What does it tell us?
What weight should we give to different questions?
How do we respond?
Who’s responsibility is it to respond?

18. Closing the Loop
After an evaluation make sure that you feedback to all involved about how you’ve made use of the data and what’s happening as a result

19. Pathway approach - IMPACT levels (levels 0 - 4) Cause Effect
Researcher
Training & Dev.
Level 1
Reaction
Level 2
Learning
Level 3
Behaviour
Results
Level 4
Outcomes
Level 0
Foundations
Input
OUTPUTS
Benefits
OUTCOMES
Lead to

20. IMPACT Level 4 - Potential outcomes - Final results of the training and development activity
Complexity – number of factors
economic growth
personal transformation B D
social & cultural capital
qualification rates
skilled workforce
Better quality research A C
researcher engagement in training
spin-outs and start-ups by trained researchers
Time after training & development activity

21. IMPACT Level 4 - Potential outcomes - Final results of the career development activity
Complexity – number of factors
graduate gets job
Career management skills
Career adaptability
Economic growth
Student takes action B D
social & cultural capital
student engagement in career development activity A C
Time after career development activity