1. Scaling Internet Research Publication Processes to Internet Scale Jon Crowcroft University of Cambridge S. Keshav University of Waterloo Nick McKeown Stanford University

2. Outline  Problems with current process  Author and reviewer incentives  Two mechanisms Punch-line: While any system is imperfect and can be gamed … … it is time to experiment with more radical approaches.

3. Problems  Rapid increase in number of papers –More researchers in field (a good thing!) –Recirculation of minor variants –Publish-or-perish and merit schemes –Not commensurate increase in number of reviewers  Skimpy reviews  Declining paper quality –Just getting feedback from reviewers?  Favoritism –Preference to the in-crowd, friends and “famous” researchers  Overly negative reviews –A systems tradition… The increase in number of papers seems to amplify the problem

4. The “Game” A cynical but pragmatic view of wants  Authors’ wants: –Get published or get feedback. –Avoid reviewing.  Reviewers/PC members’ wants: –Minimize work (scores, but no justification) –Reject competing papers –Accept papers from friends/famous people –Reject embarrassing papers –Prestige by association  Conference organizers, PC Chairs, All of us –Highest slate of papers –Fresh ideas

5. Incentives and Goals A1Authors should not submit poor papers A2Authors should become reviewers R1Reviewers should file good reviews R2Reviewers should not favor friends R3Reviewers should not favor famous people R4Reviewers should not trash competition C1Conference should have high quality papers and fresh ideas.

6. Mechanisms  Any mechanism can be gamed, so these are necessarily vague at this point.  Just trying to get discussion going.  Needs more study and discussion.

7. Author incentives A1: Authors should not submit poor papers Proposal: Conference publish author’s acceptance rate for conference (even if zero) Conjecture: Peer pressure will help. More radical: Publish titles/authors of rejected papers. Conjecture: Will reduce recirculation.

8. Author Incentives (2) A2: Authors should become reviewers. Proposal: Best reviewer award and public acknowledgment.

9. Author Incentives (3) More radical: Virtual economy. Tokens paid to reviewers; spent to submit papers. Basic idea:  One token per review, three tokens to submit (assuming 3 reviews per paper)  Accepted papers are credited 1-3 tokens – no cost for good papers Conjecture: Solves A1 and A2 Clearly many details needed to get implementation right….

10. Reviewer Incentives R1: Reviewers should submit good reviews R2: Reviewers should not favor their friends R4: Reviewers should not trash competing papers Proposal: Authors rate (anonymous) reviews. Reviews circulated to PC. Conjecture: Peer pressure among PC. More radical: Use pseudonyms and publish all reviews Even more radical: Openly publish reviews with name of reviewer. Or a blend of real names and pseudonyms.

11. Reviewer Incentives (2) R3: Reviewers should not favor “famous” researchers. We have no idea how to solve this! Public reviews can make this worse.

12. A more radical approach  Too much of process is hidden  We don’t exploit the web or reputation schemes  Can we make the entire process open and transparent? Basic Idea: All papers and reviews are public and signed (some could use pseudonyms).

13. Consequences  A1: Readers can draw their own conclusion and tell the world. Peer pressure.  A2: Infrequent reviewers would be exposed.  R1- R4: Reviewer names would be alongside reviews.  Correlation of reviews help us decide what papers to read.  Some good papers in less well-known conferences are raised to the top by acclaim.  Continued discussion and feedback after paper has been published.

14. We believe…  Academic community is crying out for a better system.  There are underlying incentives we need to acknowledge. Our list is probably incomplete.  We don’t exploit the web enough.  While any system is imperfect and can be gamed …  … it is time to experiment with more radical approaches.