Web-Based Student Peer Review: A Research Summary Edward F. Gehringer Department of Computer Science North Carolina State University The Expertiza project has been funded by the National Science Foundation Please visit our Web site: http://tinyurl.com/expertiza-site

Credits … Arlene Russell, Calibrated Peer Review Chris Schunn, SWoRD Steve Joordens & Dwayne Paré, Peer Scholar Eric Ford & Dmytro Babik, Mobius SLIP Luca de Alfaro, CrowdGrader Helen Hu and David McNaughton, uJudge Gehringer, Classroom peer review efg@ncsu.edu

Outline What’s good about peer review? Rubrics Rating vs. ranking Formative vs. summative Quality control Who reviews whom? Online apps for peer review Examples of ethical analyses Gehringer, Classroom peer review efg@ncsu.edu`

Advantages of peer review? Gehringer, Classroom peer review efg@ncsu.edu

Some advantages of peer review Feedback is more extensive quicker scalable Can’t blame the reader! Forces students to think metacognitively Gehringer, Classroom peer review efg@ncsu.edu

Rubrics Why use a rubric? Students can help create the rubric Tell students what to look for “Fairness” in assessment Students can help create the rubric How detailed? Longer rubric: Draw attention to more criteria But also more fatigue Probably less textual feedback Short rubric: More textual feedback Students can miss things Gehringer, Classroom peer review efg@ncsu.edu

Rubric advice

Rating vs. ranking Should students rate others’ work on a Likert scale, or rank students against each other? Rating Easier to rate than rank using a rubric Can give 2 students the same rating Ranking May not be compatible with F2F review More robust when reviewers are not experts Can use a slider to show nearness Gehringer, Classroom peer review efg@ncsu.edu

Mobius SLIP’s approach to ranking Gehringer, Classroom peer review efg@ncsu.edu

Formative vs. summative peer review Formative—text feedback Summative—Likert scale Should peer review be used summatively? Gehringer, Classroom peer review efg@ncsu.edu

Quality control You can’t take review quality for granted. Approaches Metareviewing Calibration Reputation system Gehringer, Classroom peer review efg@ncsu.edu

Metareviewing “Review the reviewer” “Rate the rater” Who performs metareviews? 3 choices Author? Instructor? 3rd party? Can we automate the process? Gehringer, Classroom peer review efg@ncsu.edu

Calibration Basic idea: Training course for reviewers How they do  how much credence they get Before students review peers, they get 3 works to review 1 exemplary Their agreement with instructor  Reviewer Competency Index Others have known defects Gehringer, Classroom peer review efg@ncsu.edu

Gehringer, Classroom peer review efg@ncsu.edu Reputation algorithm s1 s2 s3 s4 s5 r1 0.6 — r2 0.3 0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1 0.6 0.4 — r2 0.3 0.2 r3 r4 0.5 r5 s1 gets the same scores from reviewers in both situations. Should it get the same grade? Gehringer, Classroom peer review efg@ncsu.edu Gehringer, Classroom peer review efg@ncsu.edu

Reputation algorithm, cont. s1 s2 s3 s4 s5 r1 0.6 — r2 0.3 0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1 0.6 0.4 — r2 0.3 0.2 r3 r4 0.5 r5 r2 and r3 agree with their co-reviewers … Gehringer, Classroom peer review efg@ncsu.edu

Reputation algorithm, cont. s1 s2 s3 s4 s5 r1 0.6 — r2 0.3 0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1 0.6 0.4 — r2 0.3 0.2 r3 r4 0.5 r5 r2 and r3 agree with their co-reviewers … while r1 gives higher scores. So, s1’s grade may be inflated. Gehringer, Classroom peer review efg@ncsu.edu

Reputation algorithm, cont. s1 s2 s3 s4 s5 r1 0.6 — r2 0.3 0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1 0.6 0.4 — r2 0.3 0.2 r3 r4 0.5 r5 In this situation, r1 agrees with his co-reviewers, while r2 and r3 give lower scores. So in this case, s1 was reviewed by “harder” graders, and thus deserves a higher grade. Gehringer, Classroom peer review efg@ncsu.edu

Reputation systems—how reliable? Two studies on Coursera MOOC [2013] Piech et al.: ≥ 26% of grades ± 5% from “ground truth.” Kulkarni et al.: 40% of grades off by 1 letter grade! But … difficulty affects accuracy, brief calibration, no metareviewing this was, after all, a MOOC Gehringer, Classroom peer review efg@ncsu.edu

Reputation systems: Crowdgrader Compute consensus grade As in Olympics, discard highest & lowest ¼ of grades; average the rest Compute an accuracy grade, based on how close the student’s grade is to the consensus grade Compute a helpfulness grade, based on ratings from reviewees (on a scale from –2 to +2) Weight these 3 factors as desired by instructor

Who reviews whom? Simplest: Each student reviews k other students Reviewing in groups—case study, etc. Individuals review teams Dynamic assignment, to make sure all get reviewed Dynamic assignment may require you to complete one review before it assigns you the next. Or it may be done like Expertiza … Gehringer, Classroom peer review efg@ncsu.edu

The PR app landscape Most widely used: CPR Sharable assignments for many disciplines But, you probably want to adapt. Gehringer, Classroom peer review efg@ncsu.edu

SWoRD Perhaps the most-researched system … from Pitt’s Learning Resource Development Ctr. Gehringer, Classroom peer review efg@ncsu.edu

Peer Scholar Came from U. of Toronto Now sold by Pearson in Canada Free (for now) in the US Supports (& recommends) revision and resubmission Gehringer, Classroom peer review efg@ncsu.edu

Mobius SLIP Origin in case-study courses Based on ranking “Double loop” Gehringer, Classroom peer review efg@ncsu.edu

Expertiza “Reusable learning objects through peer review” Supports signing up for topics/parts of a project Students (or instructor) form teams Individuals review teams Teammates review each other Gehringer, Classroom peer review efg@ncsu.edu

Signup sheet Gehringer, Classroom peer review efg@ncsu.edu

Viewing results Gehringer, Classroom peer review efg@ncsu.edu

Summary Reasons for doing peer review Rubrics are important Rating vs. ranking Formative vs. summative Quality control Who reviews whom? Online apps for peer review Examples of ethical analyses Gehringer, Classroom peer review efg@ncsu.edu`