David Millard Karen Fill Hugh Davis Lester Gilbert Gary Wills Learning Societies Lab, University of Southampton, UK Towards a Canonical View of Peer Assessment

PeerPigeon PeerPigeon is an out-of-the-box system that runs any type of peer review process by supporting a generalised case of Peer Review. Overview: Why Peer Assessment / Review? How do we provide for the generalised case? Introduction to PeerPigeon

Peer Assessment Peer Assessment has many advantages: –Giving a sense of ownership of the assessment process, improving motivation –Encouraging students to take responsibility for their own learning –Treating assessment as part of learning, so that mistakes are opportunities rather than failures –Practicing the transferable skills needed for life-long learning, especially evaluation skills –Using external evaluation to provide a model for internal self- assessment of own learning –Encouraging deep rather than surface learning. Bostock S., “Student peer assessment”, Higher Education Academy Article, 16 Mar 2001

Case Studies Simple - The simplest form of peer review is where authors and reviewers are paired together Round Robin - Where participants are grouped, and each participant reviews the work of every other other participant in their group. Group Activity - Where a group of authors work together to produce an artefact, and then that artefact is reviewed by a third party. Group Review - Where a group of authors work together to produce an artefact, and then individually review the efforts of their group. authors reviewers participants authors reviewer group workers

Case Studies Committee Review - Where a group of reviewers act together and look at several different artefacts in order to produce one review. In the research community we are familiar with this as the conference committee stage of peer review Multiplicity - Where multiple authors create multiple artefacts which are then independently reviewed by multiple reviewers. For example, where students give a paper and presentation and are assessed by their classmates on both paper reviewers committee authors reviewers

Common Review Cycle All these cases can be thought of as being built of common review cycles 1.The cycle can be started in any one of its three states. For example, to begin an activity the student may be asked to Generate an artefact, to Submit an existing artefact, or the tutor may provide it, in which case the first task is to Distribute it. 2.The cycles can be interleaved, and occurring in parallel as well as in sequence. 3.Each stage within the process may involve 1...n participants (authors/tutors/reviewers), producing 1...m resources (artefacts/reviews/marks).

Example of Multiplicity: n students, m tutors each student delivers a presentation and answers questions (i.e. two artefacts) students and tutors review/mark the presentations only tutors review/mark the answers

CycleCreators Authors/ Reviewers Resources Artefacts/ Reviews Receivers Reviewers /Authors 111n+1 2nm2 31m Example of Multiplicity: n students, m tutors each student delivers a presentation and answers questions (i.e. two artefacts) students and tutors review/mark the presentations only tutors review/mark the answers

Use Case for PeerPigeon What is PeerPigeon? Create Adjust Run The complexity of the system is encapsulated in a Peer Review Plan with three elements: A Peer Review Pattern (an ordered description of the cycles of peer review and the roles of the participants in each cycle). A number of actual Participants (possibly arranged into Groups) that populate the roles in the plan. A Schedule of upcoming dates and times, that ties the pattern to a real timescale.

Peer Review Patterns We need to formalise the cycles (make them machine readable) –Generation occurs outside of the system –Submission is simple We expect X to submit Y –Distribution is complex Where does Y go next! We represent Distribution using transforms: –a participant takes an existing artefact and produces a new artefact based on it –E.g. a reviewer takes a paper and transforms it into a review.

Peer Review Patterns Major challenge with specifying the transforms –Irreducible complexity –An infinite number of ways to choose the allocation algorithm We have used a Domain Specific Language (DSL) –a programming language for a specific set of tasks. –a clean syntax that can describe the specific steps/information –an agile alternative to XML (no need for schema/dtds, a data file, or a parser) Ruby is an ideal language for DSLs –Scripting language –Dynamic programming features –Lots of support / active community

PeerPigeon DSL Example Another simple review pattern: –in a group each person produces a paper and reviews the paper of someone else A simple algorithm: –pass each paper along to the next person In our Ruby DSL: cycle :review do |c| c.description 'Write Review' c.deadline '2 days from now','4 days from now' c.transform :paper_1, 2, :review_1 c.transform :paper_2, 3, :review_2 c.transform :paper_3, 1, :review_3 end c.distribution :reviewers, lambda { |group,person| c.transform group.wrap('paper', person, -1), person, group.wrap('review', person, +1) }

PeerPigeon Prototype

Conclusions and Future Work We have shown that Peer Review can be generalised –The building blocks are Peer Review Cycles –Generate -> Submit -> Distribute The Distribution can be encoded as a sequence of transforms –Paper 1 is transformed by Person 2 into Review 1 The Distribution algorithm is irreducibly complex –But can be handled by a Domain Specific Language (DSL) We are currently prototyping the PeerPigeon system –Implements the generalised use case –Written in Ruby on Rails (Ruby DSL) An alpha version is available from our website –Release version (late beta) is planned for end of August –