David Millard Karen Fill Hugh Davis Lester Gilbert Gary Wills Learning Societies Lab, University of Southampton, UK

Introduction What is PeerPigeon? –JISC funded 6 month project –Based at Southampton –12 person months Build a Set of Services to support the process of Peer Review Objectives –Develop a Peer Assessment SUM for the e-Framework –Develop Service Descriptions for Services to Distribute Resources in a Peer Assessment Scenario (hence the Pigeon bit!) –Develop simple web client to demonstrate those services –Investigate REST Services within the e-Framework

Peer Assessment Peer Assessment has many advantages: –Giving a sense of ownership of the assessment process, improving motivation –Encouraging students to take responsibility for their own learning –Treating assessment as part of learning, so that mistakes are opportunities rather than failures –Practicing the transferable skills needed for life-long learning, especially evaluation skills –Using external evaluation to provide a model for internal self- assessment of own learning –Encouraging deep rather than surface learning. Bostock S., “Student peer assessment”, Higher Education Academy Article, 16 Mar 2001

Case Studies Simple - The simplest form of peer review is where authors and reviewers are paired together Round Robin - Where participants are grouped, and each participant reviews the work of each other participant in their group. Group Activity - Where a group of authors work together to produce an artefact, and then that artefact is reviewed by a third party. Group Review - Where a group of authors work together to produce an artefact, and then individually review the efforts of their group. Committee Review - Where a group of reviewers act together and look at several different artefacts in order to produce one review. In the research community we are familiar with this as the conference committee stage of peer review Multiplicity - Where multiple authors create multiple artefacts which are then independently reviewed by multiple reviewers. For example, where students give a presentation and answer questions and are assessed by their classmates on both

Common Review Cycle All these cases can be thought of as being built of common review cycles 1.The cycle can be started in any one of its three states. For example, to begin an activity the student may be asked to Generate an artefact, to Submit an existing artefact, or the tutor may provide it, in which case the first task is to Distribute it. 2.The cycles can be interleaved, and occurring in parallel as well as in sequence. 3.Each stage within the process may involve 1...n participants (authors/tutors/reviewers), producing 1...m resources (artefacts/reviews/marks).

Multiplicity: n students, m tutors each student delivers a presentation and answers questions (i.e. two artefacts) students and tutors review/mark the presentations only tutors review/mark the answers

CycleCreators Authors/ Reviewers Resources Artefacts/ Reviews Receivers Reviewers /Authors 111n+1 2nm2 31m Multiplicity: n students, m tutors each student delivers a presentation and answers questions (i.e. two artefacts) students and tutors review/mark the presentations only tutors review/mark the answers

Use Case for PeerPigeon What is PeerPigeon?

PeerPigeon in FREMA

Future Plans Develop a definition of the Assessment Plan –Should contain: A Peer Review Pattern (an ordered description of the cycles of peer review and the roles of the participants in each cycle). A number of actual Participants (possible arranged into Groups) that populate the roles in the plan. A Schedule of upcoming dates and times, that ties the pattern to a real timescale. –Possible standards IMS LD QTI Factor a set of services from the Use Case –Author a Peer Assessment SUM in FREMA Build and test