1. Refereeing “And diff’ring judgements serve but to declare, That truth lies somewhere, if we knew but where.” – William Cowper, Hope

2. Refereeing: What’s in it for you? “Refereeing is … a good way to learn to write better papers; evaluating the work of others gives one insight into one’s own.” (Smith) “Refereeing is … a good way to learn to write better papers; evaluating the work of others gives one insight into one’s own.” (Smith) Guidelines for reviewing are, by implication, guidelines for writing. Guidelines for reviewing are, by implication, guidelines for writing.

3. What is a Publishable Paper? Sufficient contribution Sufficient contribution New and interesting results New and interesting results Insightful synthesis of existing results Insightful synthesis of existing results Useful survey or tutorial Useful survey or tutorial Small results OK—might spark new research Small results OK—might spark new research But not But not Papers that have major flaws or significant minor flaws Papers that have major flaws or significant minor flaws Papers that are mostly repetitions of other papers Papers that are mostly repetitions of other papers Papers that have good ideas badly expressed (encourage a rewrite) Papers that have good ideas badly expressed (encourage a rewrite)

4. The Referee Report Summarizes the point of the paper Summarizes the point of the paper Evaluates significance and quality Evaluates significance and quality Gives a recommendation Gives a recommendation Justifies the recommendation Justifies the recommendation

5. Make Reports Constructive Acceptance Acceptance Persuade editors that the paper should be accepted. Persuade editors that the paper should be accepted. Superficial comments provoke suspicion that the paper has not been read carefully. Superficial comments provoke suspicion that the paper has not been read carefully. Describe changes required to fix residual faults. Describe changes required to fix residual faults. Rejection Rejection Clearly explain why—detail faults. Clearly explain why—detail faults. Explain what the authors might do next. Explain what the authors might do next. Do further work Do further work Submit to a different venue Submit to a different venue Correct major presentation flaws to make the results clear enough to be judged Correct major presentation flaws to make the results clear enough to be judged Consider additional questions, whose answers would make the work worthwhile Consider additional questions, whose answers would make the work worthwhile

6. Issues in Evaluating a Paper (a checklist) What is the purpose? What is the purpose? (What’s the problem?) Is it appropriate for the venue? Is it appropriate for the venue? Does the paper recognize previous work? Does the paper recognize previous work? Is the goal significant? Is the goal significant? (Why’s the problem a problem?)

7. Issues in Evaluating a Paper (a checklist …) Does the paper achieve its goal? Does the paper achieve its goal? (What’s the solution?) Is the research method valid and are correct conclusions drawn? Is the research method valid and are correct conclusions drawn? (Why’s the solution a solution?) Is the presentation satisfactory? Is the presentation satisfactory?

8. Other Issues Timely response Timely response Confidentiality Confidentiality Single-blind reviews Single-blind reviews Misuse of information to which the reviewer is privy Misuse of information to which the reviewer is privy Delicate problem of asking the author to cite the referee’s own work (no easy solution) Delicate problem of asking the author to cite the referee’s own work (no easy solution) Authors’ reputation Authors’ reputation Shouldn’t matter Shouldn’t matter Double-blind reviews Double-blind reviews Conflict of interest Conflict of interest

9. Other Issues … Reviewers often referee work that is unfamiliar or outside the referee’s domain of expertise. (implications, especially for conferences?) Reviewers often referee work that is unfamiliar or outside the referee’s domain of expertise. (implications, especially for conferences?) Shouldn’t recommend acceptance unless sure of the quality (implications, especially for conferences?) Shouldn’t recommend acceptance unless sure of the quality (implications, especially for conferences?)

10. Nitpicking on the Writing? Help authors improve their work. Help authors improve their work. Correct typos and suggest other improvements. Correct typos and suggest other improvements. For excessive numbers of errors, give examples and recommend proofreading. For excessive numbers of errors, give examples and recommend proofreading. Give non-native writers help, but also a break. (example) Give non-native writers help, but also a break. (example)example Frequent errors indicate that the author has not been careful—calling even the results into question. Frequent errors indicate that the author has not been careful—calling even the results into question. Must (reluctantly) accept some poorly written work, but results are of no value if the paper is so poor that it cannot be read. Must (reluctantly) accept some poorly written work, but results are of no value if the paper is so poor that it cannot be read.

11. Receiving Reviews of Your Paper Make use of reviews: One way to get peer feedback on your work One way to get peer feedback on your work When the reviewer didn’t understand—ask why and fix the probable cause When the reviewer didn’t understand—ask why and fix the probable cause When the reviewer makes a mistake—ask why and make it better the next time When the reviewer makes a mistake—ask why and make it better the next time

12. Your Reputation is at Stake “The [reviewer] forms an opinion of the author.” “The [reviewer] forms an opinion of the author.” “If the quality … [reflects] badly on the author, [the paper] should not even be submitted for publication.” “If the quality … [reflects] badly on the author, [the paper] should not even be submitted for publication.”