1. Web-Based Student Peer Review: A Research Summary
Edward F. Gehringer
Department of Computer Science
North Carolina State University
The Expertiza project has been funded by the National Science Foundation Please visit our Web site:

2. Credits … Arlene Russell, Calibrated Peer Review Chris Schunn, SWoRD
Steve Joordens & Dwayne Paré, Peer Scholar
Eric Ford & Dmytro Babik, Mobius SLIP
Luca de Alfaro, CrowdGrader
Helen Hu and David McNaughton, uJudge
Gehringer, Classroom peer review

3. Outline What’s good about peer review? Rubrics Rating vs. ranking
Formative vs. summative
Quality control
Who reviews whom?
Online apps for peer review
Examples of ethical analyses
Gehringer, Classroom peer review

4. Advantages of peer review?
Gehringer, Classroom peer review

5. Some advantages of peer review
Feedback is
more extensive
quicker
scalable
Can’t blame the reader!
Forces students to think metacognitively
Gehringer, Classroom peer review

6. Rubrics Why use a rubric? Students can help create the rubric
Tell students what to look for
“Fairness” in assessment
Students can help create the rubric
How detailed?
Longer rubric:
Draw attention to more criteria
But also more fatigue
Probably less textual feedback
Short rubric:
More textual feedback
Students can miss things
Gehringer, Classroom peer review

7. Rubric advice

8. Rating vs. ranking
Should students rate others’ work on a Likert scale,
or rank students against each other?
Rating
Easier to rate than rank using a rubric
Can give 2 students the same rating
Ranking
May not be compatible with F2F review
More robust when reviewers are not experts
Can use a slider to show nearness
Gehringer, Classroom peer review

9. Mobius SLIP’s approach to ranking
Gehringer, Classroom peer review

10. Formative vs. summative peer review
Formative—text feedback
Summative—Likert scale
Should peer review be used summatively?
Gehringer, Classroom peer review

11. Quality control You can’t take review quality for granted. Approaches
Metareviewing
Calibration
Reputation system
Gehringer, Classroom peer review

12. Metareviewing “Review the reviewer” “Rate the rater”
Who performs metareviews? 3 choices
Author?
Instructor?
3rd party?
Can we automate the process?
Gehringer, Classroom peer review

13. Calibration Basic idea: Training course for reviewers
How they do  how much credence they get
Before students review peers, they get 3 works to review
1 exemplary
Their agreement with instructor  Reviewer Competency Index
Others have known defects
Gehringer, Classroom peer review

14. Gehringer, Classroom peer review efg@ncsu.edu
Reputation algorithm s1 s2 s3 s4 s5 r1
0.6 — r2
0.3
0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1
0.6
0.4 — r2
0.3
0.2 r3 r4
0.5 r5
s1 gets the same scores from reviewers in both situations. Should it get the same grade?
Gehringer, Classroom peer review
Gehringer, Classroom peer review

15. Reputation algorithm, cont.s1 s2 s3 s4 s5 r1
0.6 — r2
0.3
0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1
0.6
0.4 — r2
0.3
0.2 r3 r4
0.5 r5
r2 and r3 agree with their co-reviewers …
Gehringer, Classroom peer review

16. Reputation algorithm, cont.s1 s2 s3 s4 s5 r1
0.6 — r2
0.3
0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1
0.6
0.4 — r2
0.3
0.2 r3 r4
0.5 r5
r2 and r3 agree with their co-reviewers … while
r1 gives higher scores.
So, s1’s grade may be inflated.
Gehringer, Classroom peer review

17. Reputation algorithm, cont.s1 s2 s3 s4 s5 r1
0.6 — r2
0.3
0.4 r3 r4 r5 s1 s2 s3 s4 s5 r1
0.6
0.4 — r2
0.3
0.2 r3 r4
0.5 r5
In this situation, r1 agrees with his co-reviewers, while r2 and r3 give lower scores.
So in this case, s1 was reviewed by “harder” graders, and thus deserves a higher grade.
Gehringer, Classroom peer review

18. Reputation systems—how reliable?
Two studies on Coursera MOOC [2013]
Piech et al.: ≥ 26% of grades ± 5% from “ground truth.”
Kulkarni et al.: 40% of grades off by 1 letter grade!
But …
difficulty affects accuracy,
brief calibration, no metareviewing
this was, after all, a MOOC
Gehringer, Classroom peer review

19. Reputation systems: Crowdgrader
Compute consensus grade
As in Olympics, discard highest & lowest ¼ of grades; average the rest
Compute an accuracy grade,
based on how close the student’s grade is to the consensus grade
Compute a helpfulness grade,
based on ratings from reviewees (on a scale from –2 to +2)
Weight these 3 factors as desired by instructor

20. Who reviews whom? Simplest: Each student reviews k other students
Reviewing in groups—case study, etc.
Individuals review teams
Dynamic assignment, to make sure all get reviewed
Dynamic assignment may require you to complete one review before it assigns you the next. Or it may be done like Expertiza …
Gehringer, Classroom peer review

21. The PR app landscape Most widely used: CPR
Sharable assignments for many disciplines
But, you probably want to adapt.
Gehringer, Classroom peer review

22. SWoRD
Perhaps the most-researched system … from Pitt’s Learning Resource Development Ctr.
Gehringer, Classroom peer review

24. Peer Scholar Came from U. of Toronto Now sold by Pearson in Canada
Free (for now) in the US
Supports (& recommends) revision and resubmission
Gehringer, Classroom peer review

25. Mobius SLIP Origin in case-study courses Based on ranking
“Double loop”
Gehringer, Classroom peer review

26. Expertiza “Reusable learning objects through peer review”
Supports signing up for topics/parts of a project
Students (or instructor) form teams
Individuals review teams
Teammates review each other
Gehringer, Classroom peer review

27. Signup sheet
Gehringer, Classroom peer review

28. Viewing results
Gehringer, Classroom peer review

29. Summary Reasons for doing peer review Rubrics are important
Rating vs. ranking
Formative vs. summative
Quality control
Who reviews whom?
Online apps for peer review
Examples of ethical analyses
Gehringer, Classroom peer review