1. Enhancing Assessment and Feedback Principles and Practice David Nicol Professor of Higher Education Centre for Academic Practice and Learning Enhancement (CAPLE] Director of REAP and PEER projects (www.reap.ac.uk)www.reap.ac.uk University of Strathclyde, Scotland ATN Australia: 20 th October 2011

2. National Student Survey (UK) Assessment & feedback (2008) NoSurvey Statement EnglandScotlandNorthern Ireland 5.The criteria used in marking have been clear in advance 69 6.Assessment arrangements and marking have been fair 74 73 7.Feedback on my work has been prompt 565153 8.I received detailed comments on my work 6152 9.Feedback on my work has helped clarify things I did not understand 565150 22Overall, I am satisfied with the quality of the course 828683

3. Plan Background Re-engineering Assessment Practices (REAP) project [£1m] Concepts and example of practice An institutional viewpoint PEER project

4. Background Departments and faculties: supporting educational improvement projects. REAP project: big implementation across 3 universities Policy/strategy: led development of policy in assessment and feedback [based on REAP] Students: ‘Feedback as dialogue’ campaign PEER project – JISC funded [£50k] HE Sector: Project facilitator for QAA Scotland on A&F Research/publications: assessment, learning, change See www.reap.ac.ukwww.reap.ac.uk

5. REAP : Re-engineering Assessment Practices Scottish Funding Council for Universities (£1m project) 3 Universities - Strathclyde, Glasgow & Glasgow Caledonian Large 1 st year classes (160-600 students) A range of disciplines (19 modules ~6000 students) Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offline Learning quality and teaching efficiencies Assessment for learner self-regulation www.reap.ac.uk

6. Background (1) Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students’ learning, Learning and Teaching in Higher Education, 1, 3-31. See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtlhttp://www.open.ac.uk/science/fdtl

7. Gibbs and Simpson (2004) Assessment tasks [Conditions 1-4] 1. Capture sufficient study time (in and out of class) 2. Are spread out evenly across timeline of study 3. Lead to productive activity (deep vs surface) 4. Communicate clear and high expectations i.e concern here is with ‘time on task’ how much work students do - their active engagement in study

8. Background (2) Literature Review Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218 Background Student Enhanced Learning through Effective Feedback [SENLEF] project funded by HE Academy REAP project: www.reap.ac.ukwww.reap.ac.uk

9. Rethinking assessment and feedback 1. Consider self and peers as much as the teacher as sources of assessment and feedback Tap into different qualities than teacher can provide Saves time Provides considerable learning benefits (lifelong learning) 2. Focus on every step of the cycle: Understanding the task criteria Applying what was learned in action 3. Not just written feedback: Also verbal, computer, vicarious, formal and informal

10. Seven principles of good feedback Good feedback: 1. Clarifies what good performance is (goals, criteria, standards). 2. Facilitates the development of reflection and self- assessment in learning 3. Delivers high quality information to students: that enables them to self-correct 4. Encourages student-teacher and peer dialogue around learning 5. Encourages positive motivational beliefs & self esteem 6. Provides opportunities to act on feedback 7. Provides information to teachers that can be used to help shape their teaching (making learning visible) Source: Nicol and Macfarlane-Dick (2006)

11. EMPOWERMENT/ SELF-REGULATION ENGAGEMENT Principle 1: Clarify what good performance is (the context of dialogue) Students create criteria Students add own criteria Students identify criteria from samples of work Exemplars of different performance levels provided Students rephrase criteria in own words Provide document with criteria

12. Two meta principles 1. Meta-PRINCIPLE 1: time and effort on task (structured engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions 2. Meta-PRINCIPLE 2: developing learner self- regulation (empowerment/self-regulation) i.e steers to encourage ownership of learning – the seven principles discussed above. Key task for teacher is to balance 1 and 2

13. Example: Psychology

14. 560 first year students 6 topic areas (e.g. personality, classical conditioning), 48 lectures, 4 tutorials, 12 practicals Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

15. Problems identified No practice in writing skills but required in the exam More detail provided in lectures than mentioned in exams (not enough independent reading) No feedback except on Multiple Choice Questions (percent correct) Didn’t want to increase staff workload

16. Psychology Redesign Discussion board in Learning Management System Students in 85 discussion groups of 7-8, same groups throughout year Also open discussion board for class Friday lectures cancelled – discover material themselves Series of online tasks

17. Structure of group tasks 6 cycles of 3 weeks (one cycle x major course topic) First week: ‘light’ written task (e.g. define terms) = 7 short answers (all answer) Second week = guided reading Week three: ‘deep’ written task: students collaborate in writing a 700-800 word essay on the same topic. Within each week: The Monday lecture – introducing material Immediately after lecture, task posted online – for delivery the following Monday Model answers (selected from students) posted for previous week’s task

18. The teaching role Participation in the discussions was compulsory but not marked (in subsequent years there was 2% mark for participation) Course leader provided general feedback to the whole class – often motivational He encouraged students to give each other feedback The group discussions were not moderated but monitored for participation

19. An example of ‘deep’ task The Task – 800 word essay: Assess the strengths and weaknesses of Freud’s and Eysenck’s theories of personality. Are the theories incompatible? readings suggested questions provided which all students should try

20. Relation to the Gibbs & Simpson’s four assessment conditions 1. Tasks require significant study out of class (condition 1) 2. Tasks are distributed across topics and weeks (condition 2) 3. They move students progressively to deeper levels of understanding (condition 3) 4. There are explicit goals and progressive increase in challenge (condition 4)

21. Relation to 7 feedback principles 1. Standard format and model answers provide progressive clarification of expectations (principle 1) 2. Students encouraged to self-assess against model answer (principle 2) 3. Course leader provides motivational and meta-level feedback and selects model answers (principle 3) 4. Online peer discussion aimed at reaching consensus is core feature of design about response (principle 4) 5. Focus on learning not just marks, sense of control/challenge enhanced motivation (principle5) 6. Repeated cycle of topics and tasks provide opportunities to act on feedback (principle 6) 7. VLE captures all interactions allowing course leader to monitor progress and adapt teaching (principle 7)

22. Benefits Students worked exceptionally hard Written responses of exceedingly high standard Students took responsibility for learning High levels of motivation: atmosphere in class improved Online interactions showed powerful ‘scaffolding’ and community building Feedback with 560 students through peer and self- feedback (model answers) Easy for tutors to monitor participation Improved mean exam performance (up from 51-59%, p<0.01) weaker students benefit most

23. Online postings/interaction 24,362 messages posted by groups for essay tasks Average number of postings per student 44.3 1067 postings to general open discussion forum Students set up online study groups for other subjects Structured tasks online triggered important socio- cognitive processes

24. Has it worked?

27. Questions and Discussion

28. Why use principles 1. Provides a framework for operationalizing the big idea – the development of learner self-regulation 2. Helps translate the research into accessible guidelines for teaching practice 3. Ensures change is educationally driven 4. Enables connections to be made across innovations in different disciplines 5. Provides a common language to talk about innovation and for dissemination 6. Can add to the evaluation – process measures 7. Helps identify where technology can leverage benefits

29. Guidelines for Implementation 1. A single principle or many? 2. Tight-loose – maintain fidelity to the principles (tight) but encourage disciplines to develop their own techniques of implementation (loose) 3. Balance teacher feedback with peer and self- generated feedback 4. The more actively engaged students are, the better the course design

30. Developments since REAP 1. Principles of Assessment and Feedback approved by University Senate and embedded in policy (2008) 2. Use of principles to inform curriculum renewal and Quality Assurance processes 3. ‘Feedback as Dialogue’ campaign to gain commitment of students 4. PEER Project (Peer Evaluation in Education Review)

34. Peer Review in Education Evaluation [PEER] The aims of the PEER project are to: Review evidence base for peer review Develop educational designs for peer review (and self-review) Identify software support for peer review Pilot implementations of peer review with large student numbers Produce guidelines for higher education – why do it, how to do it, pitfalls and solutions and software possibilities. see http://www.reap.ac.uk/peer.aspxhttp://www.reap.ac.uk/peer.aspx

35. Peer feedback: augmenting teacher feedback Increasing quantity and variety of feedback No extra workload on teacher with software (e.g. PeerMark, Aropa) More timely – e.g. collaborative projects Simulates professional life – reconciling different feedback perspectives

36. The argument Not enough attention has been focused on the potential of peer feedback not just as a way of increasing the quantity and quality of the feedback students receive, but as a way of giving students practice in constructing feedback.

37. The focus  Scenarios where students make evaluative judgements about the work of peers and provide a feedback commentary, usually written  Not talking about scenarios involving..….informal feedback in collaborative tasks.....students evaluating each other’s contribution to group working.....students grading/marking each other’s work, although some rating might be part of peer design

38. Benefits of feedback construction (1) Constructivist rather than ‘telling’ paradigm 1. High-level cognitive activity: students cannot easily be passive 2. Students actively exercise assessment criteria from many perspectives 3. Writing commentaries develops deep disciplinary expertise 4. See many approaches and learn that quality can be produced in different ways 5. Shifts responsibility to student – puts them in role of assessor exercising critical judgement

39. Benefits of feedback construction (2) 6. Learn to assess own work – as exactly the same skills are involved Develops capacity to make evaluative judgements - a fundamental requirement for life beyond university. Also, this capacity underpins all graduate attribute development (Nicol, 2010) Nicol, D (2011) Developing students’ ability to construct feedback, Published by QAA for Higher Education, UK

40. Example 1:Peer feedback Laboratory work [ Gibbs 2011 www.testa.ac.uk/resources ]  Reason – poor quality of lab reports in science  Strategy – students organised in groups and produce poster to represent their lab report  Peer process – hang poster in class and all students individually walk round analyse posters and write feedback on them (e.g post-its): questions, suggestions, inaccuracies etc  Result – significant improvements in lab work and reporting, positive competition in class, students did not want to look bad.

41. Example 2: Engineering Design Peer Project case study  DM 100 Design 1: first-year class  Dr Avril Thomson, Course Leader, Design Manufacturing and Engineering Management (DMEM), University of Strathclyde avril.thomson@strath.ac.uk  Caroline Breslin, Learning Technology Adviser, University of Srathclyde caroline.breslin@strath.ac.uk

42. Example 2: Design 1  82 first-year students  Design a product – ‘theme eating and resting in the city’  Research in groups (in city, in library etc.)  Individually produce a Product Design Specification (PDS) – detailed requirements for and constraints on design (rationale, performance,standards, manufacturing etc)  Given a PDS exemplar from another domain to show what’s required (stainless steel hot water cylinder)  Online learning environment: Moodle and PeerMark part of Turnitin suite

43. Product Design Specification Typical Headings  Rationale  Performance  Environment  Packaging  Maintenance  Weight and size  Disposal  Shipping  Processes  Time Scale  Patents  Product costs  Aesthetics  Ergonomics  Materials  Testing  Quality/reliability  Competition  Marketing  Life in service

44. DM 100: Design 1 Peer review task  Individually, each student peer-reviewed and provided feedback on the draft PDS of two other students  Criteria: completeness, convincingness of rationale, specificity of values (performance) and one main suggestion for improvements with reasons  Students used experience, giving and receiving feedback, to update PDS which comprises part of a Folio  Students’ self-review Folio and meet/discuss with tutor  Peer review not assessed directly but 10% marks for professionalism which included participation in peer review.

45. Peer review rubric: DM 100  Do you feel the PDS is complete in the range of headings covered? If no, can you suggest any headings that would contribute towards the completeness of the PDS and explain why they are important?  Is the PDS specific enough? Does it specify appropriate target values or ranges of values? Please suggest aspects that would benefit from further detail and explain.  To what extent do you think the rationale is convincing for the PDS? Can you make any suggestions as to how it might be more convincing? Please explain.  Can you identify one main improvement that could be made to the PDS? Provide reason(s) for your answer.

46. Evaluation 1.Online survey completed by 64 students 2.Course work marks compared to previous years 3.Focus group interviews 4.Peer review comments recorded online

47. Results 1 Which aspects of the peer review did you learn from?  Giving feedback10.9%  Receiving feedback26.6%  Giving and receiving feedback54.7%  Neither giving or receiving7.8%

48. Results 2 Did you modify your initial submission as a result of the peer review activity? Yes, as a result of the peer review given 23.4% Yes, as a result of the peer review received 25.0% Yes as a result of the peer review given AND received 28.1% No21.9% N/A1.6%

49. Results: student comments If yes, please give specific examples of modifications (n=41)  I added a couple of paragraphs and improved existing paragraphs, this added two full A4 pages to my work  I included specific materials as changed the formatting of the document so it looked more professional  I provided more specific numeric values and expanded my rationale after seeing someone else’s PDS and after receiving feedback

50. Results: RECEIVING feedback Please give examples of what you learned from RECEIVING peer reviews from other students (n=54)  Parts that I had previously missed were brought to eye such as market competition (noticing)  Receiving peer reviews gave me insight into what others thought of my work and gave me a direction to improve (reader response)  Where the PDS was confusing to understand (reader response)  I found out how good mine was (motivational)  The person who peer reviewed my PDS gave me positive feedback which helped me a lot (motivational)  Not much, they weren’t very good

51. Results: PROVIDING feedback Please give examples of what you learned from PROVIDING peer reviews of other’s work (n=47)  When giving advice to people on theirs, it gave me greater perception when reviewing my own work by listening to my own advice for example [transfer]  I had a chance to see other peoples work and aspects of their work that I felt were lacking in my work, this helped me to improve my work [transfer]  I was given a greater understanding of the level of the work the course may be demanding [standards]  Allowed me to see from an assessors perspective  Thinking from a critical point of view [critical judgement]  How to look at work critically that isn’t your own

52. Results: How you carried out peer review Could you make any comments about how you carried out the peer review? How did you evaluate the quality of the work to provide a response to the peer review questions? (n=37)  I compared it to mine and the ideal PDS and said how I would improve it  Partly by comparing my work to theirs  I tried to think about what I wrote and whether this PDS was better or worse  Looked through the headings first and noted any that should have been included then read more in detail about each individually and compared it with my own PDS

53. Focus groups How did you go about reviewing? ‘I read it through and compared it with what I had done to see if they had put something I had not done and then I added it in if they hadn’t. The four questions were useful as they provided a framework for the review. If we hadn’t had the questions it would have been difficult. I did the reviews separately and then answered one then the other. The first was a better standard than the other – so I used the ideas from the better one to comment on the weaker one. I also read the guidelines in class when I did the peer review. There were ideas from the good one that I hadn’t even thought of in mine’

54. Focus groups What do you think is best for learning –giving or receiving feedback? ‘For me it would probably be to give feedback because I think seeing what other people have done is more helpful than getting other people’s comments on what you have already done. By looking at other people’s work you can see for yourself what you have forgotten or not even thought about. When people give feedback on yours they generally just talk about what is there. They don’t say, well I did this on mine and you could put that in yours.’

55. Results  Yes76.6%  No3.1%  Maybe 18.8%  Don’t know1.6% Would you choose to participate in a peer review exercise in the future?

56. Some tentative conclusions  Giving and receiving feedback are qualitatively different  Feedback receipt helps bring deficiencies in own work to students’ attention, can be motivational and gives a experience of different audiences (readers)  Feedback production appears to be better for developing judgement – critical thinking, using criteria and standards, being more objective about own work  Fortunately receiving and giving usually occur in the same domain of assignment production – hence double duty.  More research required