Peer review in an economics writing course: the Pen is still mightier than the SWoRD Jennifer Imazeki Department of Economics
SWoRD Scaffolded Writing and Reviewing in the Disciplines –Students submit papers –Students review papers of 4-6 peers Comments and 7-point numeric rating –Students ‘back evaluate’ reviews they receive 5-point numeric rating
What makes SWoRD different? Student review and backevaluation scores grades for both writing and reviewing –Reviewing grade: “Accuracy” + “Helpfulness” Accuracy: Consistency compared to other reviewers Helpfulness: Back evaluation ratings –Writing grade: adjusted average of numeric ratings from reviewers –Scoring algorithm to remove bias
Econ 449W: Economic Literacy “The goal of this course is to make you better economists by making you better writers… This course will focus on how to apply and use [economic] knowledge, by incorporating economic material and the economic way of thinking into writing for a more general audience.”
Econ 449W: Structure 2010: 6 peer-reviewed papers, exchanged in class 1 reviewer for each paper, plus instructor feedback No feedback for reviewers, participation points only
Potential benefits of SWoRD (vs. other peer review tools) Students read multiple papers, receive feedback from multiple reviewers Back evaluations provide incentive to take reviews seriously, give students feedback for improving reviews (without requiring instructor intervention) Rubric scores converted to grades with adjustment for bias
Econ 449W: 2010 vs : 6 peer-reviewed papers, exchanged in class 1 reviewer for each paper, plus instructor feedback No feedback for reviewers, participation points only 2011: 4 peer reviewed papers, all online submission 4-5 reviewers for each paper, no/less instructor feedback on draft 1 Writers back-evaluate reviewers, reviewing score based 50% on those back evaluations
Assignment timeline First Draft 5 days Reviews (5) 5 days Final draft 5 days Final reviews (5) Back reviews by Thurs Back reviews by Wednesday 9pm Sunday9pm Friday9pm Wednesday 9pm Monday
Issues Lack of flexibility Lack of transparency Scoring algorithm questionable
Peer review in an economics writing course: the Pen is still mightier than the SWoRD Jennifer Imazeki Department of Economics