1. Leveraging Peer Review to Support and Enhance Learning at scale
Gabriel Perez; Peer Feedback
David Joyner; College of Computing

2. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

3. Personal Background About David:
Senior Research Associate in Georgia Tech’s College of Computing
Associate Director of Student Experience for the OMSCS Program
Director, LucyLabs
Creator and Instructor, CS6460: Educational Technology and CS6750: Human-Computer Interaction
Creator and Instructor, CS1301: Introduction to Computing
Course Co-Instructor: CSE6242: Data & Visual Analytics and CS7637: Knowledge-Based AI

4. Personal Background
About Gabriel:

5. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

6. Dimensions of Peer Review
Synchronous or Asynchronous? Formative or Summative? For Grades or For Feedback? Deliberately Assigned or Random? Teams or Individuals? Shuffled between Assignments or Consistent? Rubric or Free-Form? Papers, Code, Posters, Designs? Anonymous or non-anonymous?

7. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

8. Online Master of Science in Computer Science
Program Background
OMSCS
Online Master of Science in Computer Science

9. Program Background
OMSCS
Spring 2018:
6500 students, 8900 "seats"

10. 30 courses Program Background
CS6035: Information Security
CS6200: Introduction to Operating Systems
CS6210: Advanced Operating Systems
CSE6220: High-Performance Computing
CSE6242: Data & Visual Analytics
CS6250: Computer Networking
CSE6250: Big Data Analytics in Healthcare
CS6262: Network Security
CS6290: High-Performance Computer Architecture
CS6291: Embedded Software Optimization
CS6300: Software Development Process
CS6310: Software Architecture and Design
CS6340: Software Analysis and Testing
CS6400: Database Systems Concepts and Design
CS6440: Health Informatics
CS6460: Educational Technology
CS6475: Computational Photography
CS6476: Computer Vision
CS6505: Computability & Complexity
CS6601: Artificial Intelligence
CS6750: Human-Computer Interaction
CS7637: Knowledge-Based AI
CS7641: Machine Learning
CS7642: Reinforcement Learning
CS7646: Machine Learning for Trading
CS8803: Graduate Algorithms
CS8803: AI for Robotics
CS8803: Cyber-Physical System Security
CS8803: Compilers: Theory and Practice
CS8803: Cyber-Physical Design and Analysis
■ Course Developer ■ Developer & Instructor ■ Instructor

11. Program Background

12. More Online Master's Degrees
Program Background
Four Specializations
Computational Perception & Robotics Machine Learning
Computing Systems Interactive Intelligence
More Upcoming Courses
Computational Journalism Foundations of Cyber-Physical Design
Deep Learning Behavioral Imaging Cyber Attacks Cyber Defense
More Online Master's Degrees
Master of Science in Analytics

13. An online campus for a traditional degree.
Program Background
Traditional Elements
Application
Students apply and are accepted to Georgia Tech to participate in the program.
Enrollment
Students must register, pay tuition, and keep a high GPA to stay in the program.
Delivery
Assignments, projects, tests, and discussions graded and led by teams of TAs.
Graduation
30 credit hours and one complete specialization required to graduate.
An online campus for a traditional degree.

14. An online campus for a traditional degree.
Program Background
Unique Elements
Online
Not just delivered online, but built to take advantage of the internet's opportunities.
Affordable
$510 per class + $200 per semester = $6100 to $7100 for the entire degree.
Inclusive
Capacity scales with available TA resources; any qualified student can be admitted.
Flexible
Built for a working professional: no required synchronous or collocated tasks.
An online campus for a traditional degree.

15. OMSCS Program Background Residential MOOCs Accredited Rigorous
Reputable
Affordable
Scalable
Accessible
OMSCS

16. Program Background Accessible to: Full-Time Employees Parents
Non-Traditional Backgrounds
Lifelong Learners
International Students
Underprivileged Students
Flexible
Online
Affordable
Inclusive

17. Program Background

18. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

19. Four Benefits of Peer Review in OMSCS
Implicit Feedback
Learning-by-Teaching
Additional
Feedback
Grading as
Meta-Reviewing

20. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

21. In a traditional classroom, students are evaluated by one or more expert teachers.
Each assignment goes to one of these teachers, who returns an expert grade to the student.
If a class gets more students, it probably needs more teachers.
But what happens if you add a lot of new students? Hundreds? Thousands? Can you bring on more graders? What if you don’t want your students to have to pay very much?
That’s the problem we face with MOOCs: lots and lots of students, and not enough resources to hire teachers to grade them all manually.
TODO: Add many more students, many more teachers

22. MOOCs have tried a couple different approaches to resolve this problem.
One is autograding. With autograding, a computer takes care of the grading process.
Benefits:
Gives immediate feedback.
Requires no investment of more resources for more assignments.
Supports iteration and retries.
Drawbacks:
Closes the assessment space

23. A second solution is peer grading.
In peer grading, each student evaluates a number of their classmates’ assignments and assigns them a grade.
Each student’s actual grade is then the average of the grades they received from their peers.
The benefits of this:
Peer evaluation is known to be a valuable learning experience.
Scales with class size: more students means more graders.
Drawbacks:
Reliability: studies show peer evaluation can be reliable, but only in certain narrow circumstances
Feedback
Student perception: novices are evaluating me?
Student participation: retaliation, generosity, meanness
Some new initiatives in this area attempt to resolve these reliability issues through machine learning, seeding the peer review exercise with some assignments of known quality to measure peer graders’ reliability.
These efforts aren’t there yet, but they show significant promise for the future.
However, they only address one of the two issues: they may be able to take care of actual grades, but they do not resolve the challenge of getting expert feedback to students at scale.

24. Traditional Grading Automated Grading Peer Grading
Limits enrollment based on number of available graders
Traditional
Grading
Automated
Grading
Limits the types of assessments that can be used
So, we’ve covered three options, each with their significant drawbacks.
Traditional grading scales linearly: you need to hire a flat number of additional graders for every new batch of students. This accounts for some of the high costs of traditional education.
Automated grading: generally autograding incurs very low ongoing costs, but it significantly limits the kinds of assignments you can do.
Peer grading: scales nicely, but it limits the presence of expert feedback, and introduces questions about the reliability of the grading.
Here we see quite plainly the conflict that arises between traditional education and MOOCs. Traditional education requires students to pay heavy costs, so more teachers can be hired. As a result, it doesn’t scale very well. MOOCs scale better, but at the expense of the quality: either the assessments are less authentic or the feedback and grading aren’t expert-based.
What happens, then, if you want to do rigorous education at scale?
Limits the presence of expert feedback and grading
Peer
Grading

25. Graders as meta-reviewers is goes as you might expect.
First, students perform a round of peer review.
The results of that peer review are then provided to the graders.
The graders, looking at both the peer reviews and the original papers, assign grades to students.

26. Non-Staggered Peer Review Deadlines 2x as Many Peer Reviews Each Week
One Week
One Week
Staggered Assignment
Deadlines
Non-Staggered Peer Review Deadlines
2x as Many Peer Reviews Each Week
2x as Many Peer Reviews Available to Graders

27. One Week
After that, there’s one final phase that can occur at any time: students have the option to give meta-feedback to their graders (as well as their peer reviewers). This meta-feedback is used by the graders to improve their own feedback in the future, but importantly, it’s also used by us in this analysis to evaluate the effect of access to peer review on grading.

28. Hypothesized Effects:
☐ Increased efficiency
☐ Improved feedback
☐ New biases
Graders may resolve conflicts among peer reviews.
Graders may focus attention on students who have not yet received quality feedback.
Graders may elaborate on peer reviews.
Graders receive some quick initial baseline for assessment.
Graders may reference peer reviews in their own reviews.
Graders may move on when quality feedback has been provided.
Improved ease of grading may bias graders positively or negatively.
Graders may be biased toward grades assigned by peers.

29. Hypothesized Effects:
☐ Increased efficiency
☐ Improved feedback
☐ New biases
No systematic bias from access to peer reviews.
Small biasing (2.5%) effect from access to peer reviews including numeric grades.
No biasing effect from access to peer reviews excluding numeric grades.
No overall effect on grader efficiency, positive or negative.
Similar workflow across graders: read peer reviews, read paper, assign grade, write feedback.
Statistically significant (t = 2.75, p < 0.01) increase in students’ rating of feedback quality.
Expert feedback given when the expert had access to peer reviews was rated as 11% better than expert feedback given without access to peer reviews.

30. ☐ Moderate correlation ☐ No retaliation ☐ Positive experiences
Additional Results:
☐ Moderate correlation
☐ No retaliation
☐ Positive experiences
7 of 8 graders reported they would prefer to have access to peer reviews while grading.
R2 = between average peer-assigned grades and expert grades on assignments (n = 907).
R2 = between average peer-assigned grades and expert grades on project reflections (n = 844).
R2 = between expert grades and meta-feedback ratings (n = 190).

31. “Graders as meta-reviewers” improved the quality of feedback provided by graders to students.

32. Provide peer reviews to graders.
Hide or avoid peer-assigned scores.
Stagger deadlines to increase the number of peer reviews per assignment.

33. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

34. Open Issues of Peer Review
How do you teach students to do peer review? A peer review activity at the start of the semester. How do you normalize student grades? We don't, to emphasize feedback instead of grading. How do you ensure reviews are professional and respectful? We don't! Students should learn to deal with such feedback. How do you account for potential retaliation or negativity? Random pairing; the risk is low because reviews don't affect grades. How do you ensure peer reviews give accurate feedback? Focus on using peer review in subjective, open-ended areas. How do you evaluate the quality of students' peer reviews? Meta-review and deliberate grading, with a future for NLP

35. Talk Outline
Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]