1. September 2007 Practice in e-Assessment Conference Increasing learner success with technology supported assessment: findings from the REAP project Catherine Owen, Project Manager www.reap.ac.uk

2. September 2007 Practice in e-Assessment Conference The REAP Project 3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University) Focus is on large 1st year classes Pedagogy: assessment for learner self-regulation Range of technologies: online tests, simulations, discussion boards, e-voting, e-portfolios, peer/feedback software, admin systems, VLEs, offline-online Goals: Learning quality and teaching efficiencies Outputs: case studies of redesign: advice to support strategic change in institutions (transformation). Dissemination across HE/FE sector

3. September 2007 Practice in e-Assessment Conference Plan Review literature on formative assessment and feedback Suggest some simple ways of improving assessment and feedback practices in relation to the research Outline some interesting case studies of first year course redesign drawn from REAP project and elsewhere. Analyse these case studies in relation to assessment principles from literature Questions and discussion

4. September 2007 Practice in e-Assessment Conference Defining assessment and feedback Assessment – the narrow meaning is exam for external accreditation Feedback – the narrow sense is what the tutor writes about/on a finished piece of student work But in this talk also interested in: Self-assessment and reflection - a form of internally generated feedback Peer dialogue as feedback and peer assessment

5. September 2007 Practice in e-Assessment Conference Definitions (2) Who is involved in formative assessment and feedback Tutor Peers External (e.g. placement supervisor) Computer generated Self

6. September 2007 Practice in e-Assessment Conference Why take formative assessment and feedback seriously? Assessment is a key driver of student learning Assessment is a major cost in HE (staff time) Widely reported that students don’t read the feedback Dropout/retention – linked to academic experience First year experience – students need regular and structured feedback opportunities. National student survey – students are dissatisfied with feedback. QAA reports – main area of criticism in England

7. September 2007 Practice in e-Assessment Conference Assessment paradigm Transmission [teacher-centred] Teaching/learning paradigm Transmission Research on Assessment in HE Constructivist [student-centred]

8. September 2007 Practice in e-Assessment Conference Key messages Formative assessment and feedback by others can only have an impact on learning when it influences a student’s own self-regulatory processes (adapted from Boud, 1995) Students are already self-assessing and generating feedback so we should build on this capacity (Nicol and Macfarlane-Dick, 2004)

9. September 2007 Practice in e-Assessment Conference Scaffolding self regulation: 7 principles of good feedback (assessment design) 1.Clarify what good performance is (goals, criteria, standards). 2.Facilitate reflection and self-assessment in learning 3.Deliver high quality feedback to students: feedback that enables students to monitor and self-correct 4.Encourage peer and tutor dialogue around learning 5.Encourage positive motivational beliefs & self esteem through assessment 6.Provide opportunities to close the feedback loop 7.Use feedback information to shape teaching Source: Nicol and Macfarlane-Dick (2006),

10. September 2007 Practice in e-Assessment Conference 1. Help clarify what good performance is Documents with printed criteria/ published online Tutor explains/discusses criteria with students Frequently asked questions database (FAQs) Exemplars of performance with feedback (e.g. essays - - made available online) Students discuss good and bad examples of assignments and identify criteria Students rephrase criteria in own words Students generate their own criteria for a task Students construct own MCQs and post online

11. September 2007 Practice in e-Assessment Conference 2. Facilitates self assessment and reflection in learning Online quizzes (e.g. MCQs/short answers) with feedback (self monitoring) Self-assessment at submission (strengths/weaknesses) Open book use of online MCQs Peer processes where students judge work of others against standards and give each other feedback Simulations – dynamic feedback shows effects of actions PDP involving ongoing reflection on learning (e- portfolios) Confidence ratings with MCQs increases reflection

12. September 2007 Practice in e-Assessment Conference 3. Deliver high quality feedback information: that enables students to self-correct Link feedback to pre-defined criteria (comment databanks) Provide feedback soon after submission (clarify expectations) Point to resources where answers can be found (hyper- links) Have students request feedback Reader-response theory and feedback

13. September 2007 Practice in e-Assessment Conference 4. Encourage teacher and student dialogue around learning Break out discussions in tutorial to discuss written feedback and plan strategies for improvement Students give descriptive feedback to peers before submission Electronic voting technology – three levels of feedback (computer, peer and teacher) Unique weekly assessed tutorial sheets Online discussion as peer feedback (psychology example)

14. September 2007 Practice in e-Assessment Conference 5. Encourage positive motivational beliefs (self-efficacy) and self-esteem Online objective tests (repeatable, private, comparisons with own learning goals) Using authentic ‘real life’ tasks that require the kinds of thinking and skills required in employment Structured milestones with early low stakes tasks and feedback Focus on learning goals not just performance goals (e.g. rewarding effort and improvement) Two-stage assignments (drafts-feedback-marks) Supporting formation of learning communities around tasks (discussion boards)

15. September 2007 Practice in e-Assessment Conference 6. Provide opportunities to close the feedback loop Increase opportunities for resubmission Two-stage assignments (drafts-feedback-marks) Action points as feedback strategy Personal development planning – students keep records of work at different stages Initiate online peer review processes as available to academics when writing journal articles Not releasing the grade until students have commented on the feedback provided.

16. September 2007 Practice in e-Assessment Conference 7. Provides information that can be used to shape the teaching Student requested feedback (questions worth asking) Electronic voting systems Just-in time-teaching using MCQs Monitoring online discussions Monitoring project work within shared workspaces Blackboard/WebCT have some built in reporting functionality about student activity

17. September 2007 Practice in e-Assessment Conference Discussion Would any of this apply to your situation? How? How would you improve on the principles described? Why don’t these ideas apply? What would you do instead? Any questions?

18. September 2007 Practice in e-Assessment Conference Two super principles Super-principle 1: developing learner self-regulation (empowerment) i.e steers to encourage ownership of learning – the seven principles discussed above. Super-principle 2: time on task and effort (engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions Case examples from REAP – applying these principles/conditions

19. September 2007 Practice in e-Assessment Conference Case 1: Using a VLE to support collaborative group tasks (University of Strathclyde Psychology Department)

20. September 2007 Practice in e-Assessment Conference Case 1: Psychology Context: 560 first year students Mixture of psychology majors (130) and those taking psychology only for one year (430) 6 topic areas, 48 lectures, 4 tutorials, 12 practicals Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%) 85 VLE groups of 7-8 students Aims: To develop essay writing skills and provide effective feedback To reduce lectures and burden on staff

21. September 2007 Practice in e-Assessment Conference Structure of assigned group tasks 6 cycles of 3 weeks (one per major course topic) Each cycle: Week 1: "light" written task, ≈ 7 short answers, between group members. Week 2: Reading task only Week 3: "Heavy" written task, typically 7 short answers followed by combining them into a single coherent essay. Within each week is: The Monday lecture, introducing material. Immediately afterwards, that week's task posted online, for delivery before next Monday. Model answers (a few selected from the best student answers submitted) are posted for the previous week's task.

22. September 2007 Practice in e-Assessment Conference Big success, apparently! Students set targets beyond what was required: contradicts commonly held beliefs about assessment and external motivation (marks) Produced work at level ‘not seen before’ surpassing third year Spontaneous discussions about learning and learner responsibility Some students burdened by workload but easily detected Some groups participants moved at own request (3 groups) 13,429 messages posted by groups (postings from 40-400 per group) Quality of interactions ‘outstanding’ across the board Atmosphere in lecture class improved and online community developed

23. September 2007 Practice in e-Assessment Conference Case 1 and the principles Model answers and repeated task format provide progressive clarification of expectations (clear criteria, A1) Students encouraged to self-assess against model answer (A2) Online peer discussion aimed at reaching agreed response (A4) Staged complexity and focus on learning rather marks (A5) Repeated cycle of topics and tasks (A6)* VLE captures all group interchanges, allowing course leader to monitor progress and adapt (A7) Tasks require significant study out of class (capture enough work B1)* They are distributed across topics and weeks (are spread out evenly B2)* They move students progressively to deeper levels of understanding (B3) There are explicit goals and progressive increase in challenge (communicates clear and high expectations B4)

24. September 2007 Practice in e-Assessment Conference Case 2: Using a VLE to support collaborative group tasks (Glasgow University Biology Department)

25. September 2007 Practice in e-Assessment Conference Biology Case 700 students in first year class Module core for biologists but also optional for numerous other subjects Group task a key element of the class Encouraging communication and group working skills, independent research and presentation skills

26. September 2007 Practice in e-Assessment Conference Biology Case Students work in groups to create a poster investigating a species (chosen by the group) and its environmental impact Each group must argue for the extinction of a species in a head to head debate with another group in class An overall group mark is given by the tutor (for poster and debate), then peers moderate the marks in their groups to reflect individual contributions Participants also asked to reflect on their own contribution to the group and write a paragraph explaining how they added value to the task

27. September 2007 Practice in e-Assessment Conference Biology Case Improve the students’ learning experience (with positive impact on retention and progression) Support students to participate in the group discussions regardless of the timing or other commitments (family, travel, work etc.) Avoid problems arising from student absence and ‘no shows’, since all students in a group will have access to material ‘posted’ in the forum Reduce tutor mediation time by providing a permanent record of the group interactions for both students and staff

28. September 2007 Practice in e-Assessment Conference Biology Case 80 web forums for 80 groups of 8 members created on Moodle Group tasks released by tutors as a series of progressive mini-tasks with short submission deadlines (e.g. rationale for species choice) Tutor feedback provided to whole class via open message-board Peer-marking process refined to include pre-task group definition of criteria and a post-task reflection on and revision of criteria (reflection on criteria)

29. September 2007 Practice in e-Assessment Conference Benefits of the redesign Benefits of the redesign Using the VLE forums was popular with students Positive impact on student confidence and the development of face-to-face as well as online interactions Peer feedback valued as an addition to tutor feedback Students re-assured that contributions monitored by staff to support fair peer marking process Tutor mediation in group disputes significantly reduced Anecdotal evidence of improved class cohesion and greater social interaction around learning Quantitative results pending…

30. September 2007 Practice in e-Assessment Conference Relation to seven feedback principles Group discussion around criteria for marking helps to define goals and standards (principle 1, clear goals) Peer marking helps students to develop objectivity in judgement. Students self-assess their own contribution to the group and give a rationale for their contribution. (principle 2, self-assessment) GTAs monitor contributions of group members but teacher provides generic feedback to the open discussion board (principle 3, teacher feedback) Peer dialogue and feedback happens in groups and in producing a group submission to forum. Online interactions support the development of learning communities online and face-to-face (principle 4, peer dialogue) Opportunity to communicate with peers online as well as F2F liberating for many shy or anxious students (principle 5) Discussion boards provides record of all student outputs which teachers can use to make adjustments in how they teach (principle 7, shape teaching)

31. September 2007 Practice in e-Assessment Conference Relation to Gibbs and Simpson’s four conditions Structured, progressive tasks encourage time and effort on tasks (condition 1, captures sufficient study time and effort) Requires students to work in and out of class and make regular contributions (condition 2, distribute student effort) Debating with other students encourages deep rather than surface learning (condition 3, productive learning) Marking criteria are negotiated and the tasks are challenging (condition 4, clear and high expectations) Not forgetting efficiency gains in course delivery...

32. September 2007 Practice in e-Assessment Conference Case 3: Podcasts, electronic voting and SATs (University of Strathclyde Hospitality and Tourism Department)

33. September 2007 Practice in e-Assessment Conference Case 3: Tourism and Management 230 students, level course, two semesters Traditionally been delivered as two lectures per week supported by four one-hour tutorials per semester Attendance at lectures poor and declines a lot throughout the year Changes: lectures halved and replaced by: Interactive lectures using EVS Regular online SATs replacing class tests Video-podcasts of course materials Group presentations supported by online micro-tasks

34. September 2007 Practice in e-Assessment Conference Case 3: Tourism and Management Attendance at lectures improved Number of non-qualifying students reduced (from 40 to 30) Average coursework mark rose from 47.8% to 60%

35. September 2007 Practice in e-Assessment Conference Case 3 and the principles Online SATs, EVS allow self-assess against given answer, and the class’ response (A2) Online SATs & EVS usually followed by teacher’s model answer (A3) Group presentation encourages peer dialogue around task(A4) Success at tests (rather than uncertainty) promotes self-esteem (A5) Repeated cycle of tasks gives opportunities to close feedback loop(A6) EVS also lets teacher assess state of the class as a whole (A7) Podcasts, texts to be used before class (capture enough work B1) They are distributed across topics and weeks (are spread out evenly B2)

36. September 2007 Practice in e-Assessment Conference Involve learners in decision- making about assessment policy and strategy Give choice in assessment - content and processes Facilitate reflection and self- assessment in learning Encourage positive motivational beliefs & self- esteem Deliver quality feedback that helps learners self-correct Encourage time & effort on purposeful learning tasks Help clarify what good performance is EMPOWERMENT (Adapt institution to student) SOCIAL EXPERIENCE Encourage interaction and dialogue around learning (peer and teacher-student) ENGAGEMENT (Fit student to Institution) Support the development of learning communities Figure 1: A framework and ten principles for formative assessment and feedback in the first year and beyond Adapt teaching to student needs ACADEMIC EXPERIENCE © d.j.nicol 2007

37. September 2007 Practice in e-Assessment Conference Relevant papers Nicol, D (in press), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology (to be published July 2007). Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education. Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge. Nicol, D (2006), Increasing success in first year courses: assessment redesign, self-regulation and learning technologies, Paper prepared for ASCILITE conference, Sydney, Australia, Dec 3-6. Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199- 218. Nicol, D & Boyle, J. (2003), Peer Instruction versus Class-wide discussion in large classes. Studies in Higher Education, 28(4), 457-473 Boyle, J.T. and Nicol, D. J. (2003). Using classroom communication systems to support interaction and discussion in large class settings, Association for Learning Technology Journal, 11(3), 43-57