Assessment Careers: towards a vision of post-modularisation Dr Gwyneth Hughes Dr Martin Oliver Dr Holly Smith SRHE Conference 2012

Some current problems with feedback Feedback rarely scrutinised, when it is practice is inconsistent Lack of learner engagement with and understanding of feedback (Lizzio & Wilson, 2008) Inefficiency in terms of effort and impact Transmitted feedback creates dependency on teacher (Carless et al. 2011)

Praise is not helpful to students unless accompanied by detail on why the praise is deserved. Encourages dependency on others (Hattie & Timperley, 2007). Praise-critique does not encourage dialogue and self- reliance (Nicol & Macfarlane-Dick, 2005) Lack of information about progress which could be motivating (Hughes, 2011) Critique and advice are for the current assignment. Encourages focus on short-term grade improvement rather than longer-term learning. 3

Assessment Careers Project: a response to modularisation Year 1: Baseline reporting and feedback analysis Year 2 (2012/13): Five pilot programmes Year 3 Institutional implementation Assessment Careers: Institute of Education website: and video clip:

Tool to identify feedback categories P1 Giving praise P2 Recognising progress or ipsative feedback Critique C1 Correction of errors C2 Factual criticisms C3 Criticism of approach Giving advice A1 Specific to content current assignment A2 General skills in current assignment A 3 For future assignments Q Clarifications and questions O Other unclassified statements Adapted from Orsmond & Merry, 2011 including Hughes, Adapted from Orsmond, & Merry (2011) using Hughes, (2011)

Analysing feedback categories The score is the number of times a classification appears in the feedback The default unit for analysis was the sentence Where a sentence contains clauses that make distinct points, it was split into separate clauses, each of which was classified separately. Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified.

Data Analysed formative and summative assessment feedback for modules on 5 postgraduate programmes at the IOE (total 228 pieces) Recorded the total number of comments in each category and the average per script Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 5 programmes.

Baseline feedback profile Total comments for 5 programmes Summative assessment (N= 165) Category of feedbackAverage per scriptRank P1 Praise4.41 P2 Ipsative (progress)0 (negligible)5 C1-C3 Critique2.72 A1-A3 Advice for current or future assignments 1.9 (mostly for current assignment) 3 Q Questions and clarification requests

Baseline feedback profile Total comments for 5 programmes (N= 165) Category of feedbackAverage per scriptRank P1 Praise4.41 P2 Ipsative (progress)0 (negligible)5 C1-C3 Critique2.72 A1-A3 Advice for current or future assignments 1.9 (mostly for current assignment) 3 Q Questions and clarification requests

What do students do with feedback? Formative n=50Summative n=35 didn't respond to the question43 no feedback received1 nothing45 read310 revised assessment323 systematically took forward713 10

Overview of interventions Pilot 1 MA Education, Health Promotion and International Development Pilot 2 MRes in Educational and Social Research Pilot 3 MA Clinical Education Pilot 4 MA/MSc Psychology of Education Pilot 5 PGCE Primary Approx. 400 students and 30 staff in total.

Assignment cover sheets Thinking about the feedback on your draft of this essay, please indicate what the key points were and what action you took to respond to this feedback to help you prepare for and write this essay 12 I would like tutor feedback on:

Learner reflections on assessment careers An early intervention, to inform pilots Postgraduate module, 18 students Peer assessment element: critical review of a reading Week 1: feedback from tutor on reviews Week 2: feedback from peers on reviews, tutor provides feedback on feedback Week 3-7: feedback from peers Post-experience reflective discussion about the process 13

I think you presented this quite clearly. For me online feedback has its place, however, its is less personal and sometimes this makes it easier to critique someone. I do believe that it could suffer from 'general feedback' where teachers, lecturers offer general feedback having assessed a few pieces. Personally I believe it could be quite beneficial and effective. Peer assessment, computer assisted assessment together with adaptive release grades will definitely become part of our practice, it is only a matter of time. What disadvantages, if any did you find in any of these? If these are not the future, what alternatives do you suggest? 14 Examples of peer feedback

Overall you have written a really comprehensive and detailed review of the document. Your strength lies in your fantastic ability to contextualise the document with your political and pedigogical references. […] What do you understand by the term disruptive in this context? […] You seem to suggest ( or I am interpreting wrongly) that disruptive in this context is bad? […] Finally, you quote and cite almost in every sentence- but what is your personal view of 'the rise of K-12 blended learning'in primary and secondary education. Where is your personal voice? It scares me - or at least having a purely distance online learning component for such students worries me. What about you? 15 Examples of peer feedback

Experience varied: some not positive I share others opinions who indicate that it would be beneficial to have more frequent feedback from our instructor on our postings rather than from our classmates […] it is much more difficult to give usefull feedback on a critical review that you have not even read. 16

Others were more positive Training us to give feedback to each other was a great idea. Granted, it didn't always work - it showed how hard it is to get the right balance between constructive criticism vs motivating encouragement. It's not really helpful if someone just focuses on being encouraging, but sometimes I really worried if I'd been demotivating by giving too many suggestions for improvement. Sometimes I wanted to give feedback to someone, but just couldn't work out what to say! But giving feedback to others made me develop a list of things to look for, which I immediately realised I wasn't doing in my own summaries! 17

Conclusion Need to move beyond modularised assessment Pre-pilot showed that some students (not all) can be supported in developing their capacity to give and act on feedback Pilots set up to facilitate longitudinal feedback in diverse ways The Assessment Careers project will evaluate the success of pilot interventions in enabling students and staff to develop feedback beyond the module level

References Carless, D. Slater, D.Yang, M. and Lam, J Developing sustainable feedback practices. Studies in Higher Education 36, no.4: Gibbs, G. & Simpson, C. (2004) Conditions Under Which Assessment Supports Students Learning. Learning and Teaching in Higher Education, 1: 3-31 Hattie, J. & Timperley, H The Power of Feedback. Review of Educational Research 77 no. 1: Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 –

Lizzio, A. & Wilson, K Feedback on assessment: students perceptions of quality and effectiveness. Assessment & Evaluation in Higher Education 33 no.3: Nicol, D. & Macfarlane,-Dick, D Formative assessment and self- regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31 no. 2: Orsmond, P. & Merry, S Feedback alignment: effective and ineffective links between tutors and students understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2):