Leveraging Peer Review to Support and Enhance Learning at scale Gabriel Perez; Peer Feedback David Joyner; College of Computing

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

Personal Background About David: Senior Research Associate in Georgia Tech’s College of Computing Associate Director of Student Experience for the OMSCS Program Director, LucyLabs Creator and Instructor, CS6460: Educational Technology and CS6750: Human-Computer Interaction Creator and Instructor, CS1301: Introduction to Computing Course Co-Instructor: CSE6242: Data & Visual Analytics and CS7637: Knowledge-Based AI

Personal Background About Gabriel:

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

Dimensions of Peer Review Synchronous or Asynchronous? Formative or Summative? For Grades or For Feedback? Deliberately Assigned or Random? Teams or Individuals? Shuffled between Assignments or Consistent? Rubric or Free-Form? Papers, Code, Posters, Designs? Anonymous or non-anonymous?

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

Online Master of Science in Computer Science Program Background OMSCS Online Master of Science in Computer Science

Program Background OMSCS Spring 2018: 6500 students, 8900 "seats"

30 courses Program Background CS6035: Information Security CS6200: Introduction to Operating Systems CS6210: Advanced Operating Systems CSE6220: High-Performance Computing CSE6242: Data & Visual Analytics CS6250: Computer Networking CSE6250: Big Data Analytics in Healthcare CS6262: Network Security CS6290: High-Performance Computer Architecture CS6291: Embedded Software Optimization CS6300: Software Development Process CS6310: Software Architecture and Design CS6340: Software Analysis and Testing CS6400: Database Systems Concepts and Design CS6440: Health Informatics CS6460: Educational Technology CS6475: Computational Photography CS6476: Computer Vision CS6505: Computability & Complexity CS6601: Artificial Intelligence CS6750: Human-Computer Interaction CS7637: Knowledge-Based AI CS7641: Machine Learning CS7642: Reinforcement Learning CS7646: Machine Learning for Trading CS8803: Graduate Algorithms CS8803: AI for Robotics CS8803: Cyber-Physical System Security CS8803: Compilers: Theory and Practice CS8803: Cyber-Physical Design and Analysis ■ Course Developer ■ Developer & Instructor ■ Instructor

Program Background

More Online Master's Degrees Program Background Four Specializations Computational Perception & Robotics Machine Learning Computing Systems Interactive Intelligence More Upcoming Courses Computational Journalism Foundations of Cyber-Physical Design Deep Learning Behavioral Imaging Cyber Attacks Cyber Defense More Online Master's Degrees Master of Science in Analytics

An online campus for a traditional degree. Program Background Traditional Elements Application Students apply and are accepted to Georgia Tech to participate in the program. Enrollment Students must register, pay tuition, and keep a high GPA to stay in the program. Delivery Assignments, projects, tests, and discussions graded and led by teams of TAs. Graduation 30 credit hours and one complete specialization required to graduate. An online campus for a traditional degree.

An online campus for a traditional degree. Program Background Unique Elements Online Not just delivered online, but built to take advantage of the internet's opportunities. Affordable $510 per class + $200 per semester = $6100 to $7100 for the entire degree. Inclusive Capacity scales with available TA resources; any qualified student can be admitted. Flexible Built for a working professional: no required synchronous or collocated tasks. An online campus for a traditional degree.

OMSCS Program Background Residential MOOCs Accredited Rigorous Reputable Affordable Scalable Accessible OMSCS

Program Background Accessible to: Full-Time Employees Parents Non-Traditional Backgrounds Lifelong Learners International Students Underprivileged Students Flexible Online Affordable Inclusive

Program Background

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

Four Benefits of Peer Review in OMSCS Implicit Feedback Learning-by-Teaching Additional Feedback Grading as Meta-Reviewing

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

In a traditional classroom, students are evaluated by one or more expert teachers. Each assignment goes to one of these teachers, who returns an expert grade to the student. If a class gets more students, it probably needs more teachers. But what happens if you add a lot of new students? Hundreds? Thousands? Can you bring on more graders? What if you don’t want your students to have to pay very much? That’s the problem we face with MOOCs: lots and lots of students, and not enough resources to hire teachers to grade them all manually. TODO: Add many more students, many more teachers

MOOCs have tried a couple different approaches to resolve this problem. One is autograding. With autograding, a computer takes care of the grading process. Benefits: Gives immediate feedback. Requires no investment of more resources for more assignments. Supports iteration and retries. Drawbacks: Closes the assessment space

A second solution is peer grading. In peer grading, each student evaluates a number of their classmates’ assignments and assigns them a grade. Each student’s actual grade is then the average of the grades they received from their peers. The benefits of this: Peer evaluation is known to be a valuable learning experience. Scales with class size: more students means more graders. Drawbacks: Reliability: studies show peer evaluation can be reliable, but only in certain narrow circumstances Feedback Student perception: novices are evaluating me? Student participation: retaliation, generosity, meanness Some new initiatives in this area attempt to resolve these reliability issues through machine learning, seeding the peer review exercise with some assignments of known quality to measure peer graders’ reliability. These efforts aren’t there yet, but they show significant promise for the future. However, they only address one of the two issues: they may be able to take care of actual grades, but they do not resolve the challenge of getting expert feedback to students at scale.

Traditional Grading Automated Grading Peer Grading Limits enrollment based on number of available graders Traditional Grading Automated Grading Limits the types of assessments that can be used So, we’ve covered three options, each with their significant drawbacks. Traditional grading scales linearly: you need to hire a flat number of additional graders for every new batch of students. This accounts for some of the high costs of traditional education. Automated grading: generally autograding incurs very low ongoing costs, but it significantly limits the kinds of assignments you can do. Peer grading: scales nicely, but it limits the presence of expert feedback, and introduces questions about the reliability of the grading. Here we see quite plainly the conflict that arises between traditional education and MOOCs. Traditional education requires students to pay heavy costs, so more teachers can be hired. As a result, it doesn’t scale very well. MOOCs scale better, but at the expense of the quality: either the assessments are less authentic or the feedback and grading aren’t expert-based. What happens, then, if you want to do rigorous education at scale? Limits the presence of expert feedback and grading Peer Grading

Graders as meta-reviewers is goes as you might expect. First, students perform a round of peer review. The results of that peer review are then provided to the graders. The graders, looking at both the peer reviews and the original papers, assign grades to students.

Non-Staggered Peer Review Deadlines 2x as Many Peer Reviews Each Week One Week One Week Staggered Assignment Deadlines Non-Staggered Peer Review Deadlines 2x as Many Peer Reviews Each Week 2x as Many Peer Reviews Available to Graders

One Week After that, there’s one final phase that can occur at any time: students have the option to give meta-feedback to their graders (as well as their peer reviewers). This meta-feedback is used by the graders to improve their own feedback in the future, but importantly, it’s also used by us in this analysis to evaluate the effect of access to peer review on grading.

Hypothesized Effects: ☐ Increased efficiency ☐ Improved feedback ☐ New biases Graders may resolve conflicts among peer reviews. Graders may focus attention on students who have not yet received quality feedback. Graders may elaborate on peer reviews. Graders receive some quick initial baseline for assessment. Graders may reference peer reviews in their own reviews. Graders may move on when quality feedback has been provided. Improved ease of grading may bias graders positively or negatively. Graders may be biased toward grades assigned by peers.

Hypothesized Effects: ☐ Increased efficiency ☐ Improved feedback ☐ New biases No systematic bias from access to peer reviews. Small biasing (2.5%) effect from access to peer reviews including numeric grades. No biasing effect from access to peer reviews excluding numeric grades. No overall effect on grader efficiency, positive or negative. Similar workflow across graders: read peer reviews, read paper, assign grade, write feedback. Statistically significant (t = 2.75, p < 0.01) increase in students’ rating of feedback quality. Expert feedback given when the expert had access to peer reviews was rated as 11% better than expert feedback given without access to peer reviews.

☐ Moderate correlation ☐ No retaliation ☐ Positive experiences Additional Results: ☐ Moderate correlation ☐ No retaliation ☐ Positive experiences 7 of 8 graders reported they would prefer to have access to peer reviews while grading. R2 = 0.2836 between average peer-assigned grades and expert grades on assignments (n = 907). R2 = 0.4608 between average peer-assigned grades and expert grades on project reflections (n = 844). R2 = 0.0187 between expert grades and meta-feedback ratings (n = 190).

“Graders as meta-reviewers” improved the quality of feedback provided by graders to students.

Provide peer reviews to graders. Hide or avoid peer-assigned scores. Stagger deadlines to increase the number of peer reviews per assignment.

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]

Open Issues of Peer Review How do you teach students to do peer review? A peer review activity at the start of the semester. How do you normalize student grades? We don't, to emphasize feedback instead of grading. How do you ensure reviews are professional and respectful? We don't! Students should learn to deal with such feedback. How do you account for potential retaliation or negativity? Random pairing; the risk is low because reviews don't affect grades. How do you ensure peer reviews give accurate feedback? Focus on using peer review in subjective, open-ended areas. How do you evaluate the quality of students' peer reviews? Meta-review and deliberate grading, with a future for NLP

Talk Outline Personal Backgrounds Dimensions of Peer Review OMSCS Program Background Four Benefits of Peer Review in OMSCS Grading as Meta-Reviewing Open Issues in Peer Review [Gabriel pastes his outline sections here]