1 How to review a paper by Fabio Crestani

2 Disclaimer 4 There is no fixed mechanism for refereeing 4 There are simple rules that help transforming a review in a constructive document 4 In time you will develop your own style of refereeing 4 This talk mostly reflects my style

3 Purpose 4 A review serves several roles 4 The precise combination varies with the type of review –Technical/experimental approach and analysis –Computation –Ignorance of related research –Presentation style –Patents or legal issues 4 In our case: quality control!

4 Method 4 Who is interested in quality? –Program chair –Journal/book editor –You! 4 How do you assure quality? –Papers are reviewed by expert in the field –Peer review –You!

5 The process Journal Area “generalist” (choose appropriate reviewers) Conference Area specialist (peer) Submitted to Referees Editor in chiefProgram chair Editors / Editorial Board members Recommendations Referees Program committee (meta reviewers)

6 The role of the referee 4 Advisory –Authors bears responsibility for correctness of result presentation –Editor bears responsibility for acceptance/rejection

7 Why doing it? 4 Several reasons –Enhance reputation (with editor/prog. committee) –Expedites processing of your own papers –Get on editorial board or program committee –Good practice Increase your own critical appraisal ability Your papers become better –Sometimes it gets preferential treatment for your papers 4 … but refereeing means more work!

8 Consideration 4 Most reviews have strict deadlines 4 By agreeing to review you take the responsibility of doing a thorough job 4 If you cannot commit to this, notify the editor asap 4 Editors understand you may not have the time, but are unforgiving if you commit and do a poor job 4 Good editors keep a list …

9 Types of reviews 4 Anonymous –Standard for journals and conference –Can be “blind” or “double blind” 4 Friendly –When you ask a colleague to give comments on a draft –Has limitations … 4 Internal –Used mostly in commercial labs for patents and legal issues

10 How to do a review 4 Plan to read the paper 3 times 1.To get a feel for it 2.Read the paper in depth 3.Mark in up 4 Fill out the review right after the 3 rd reading, while things are still fresh in memory

11 Review structure a. The actual refereeing form b. General comments on the paper c. Specific comments on the paper d. Confidential note to editor 4 General idea: be professional and non- hostile: write the review in a style that you would like to receive for your paper

12 The refereeing form 4 Forms might look quite different but basically ask the same things (see examples) 4 Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate 4 Make sure you read and understand it well

13 General comments 4 Usually starts with 1-3 sentences summarising the paper to show that you understood it 4 Discuss author’s assumptions, motivations, technical approach, analysis, results, conclusions, references. 4 Be constructive, suggest improvements

14 Specific comments 4 Comments on style, figure, grammar, spelling mistakes, etc. 4 You can mark up directly on the paper or type in list (or bullet points) form, with reference to the page, section, etc. 4 It is up to you to decide the level of detail of your specific comments 4 You are not asked to rewrite the paper!

15 Confidential note 4 Comments to the editor that you do not want the author to see 4 Not necessary and do not feel obliged 4 Remember the review should mostly help the author, so do not “hide” comments

16 Outcome 4 Usually: a.Accept the paper as it is b.Paper requires minor changes c.Paper requires major changes (with or without a new refereeing process) d.Reject publication of the paper 4 You can only suggest, the choice is not yours –Decision is based on at least 3 reviews

17 What to consider (1) 4 Correctness –Of argument/method/algorithm/proof 4 Significance –Rule out the obvious/trivial solutions –Valid problem –Significance to area/journal 4 Innovation –Original, novel –Not trivial extension or combination of old work

18 What to consider (2) 4 Interesting –Well motivated –Relevant (when and where) 4 Timeliness –Of current interest to community –Take into account: publication delay, pre-exposure (WWW) 4 Succint –Message should be: clear, compelling, to the point

19 What to consider (3) 4 Accessible –Is it appropriate to the audience –Readable, good grammar, good structure –People do not have the time to read badly written papers

20 Ethics of refereeing (1) 4 Objectivity –Judge paper on its own merits –Remove prejudice –If you are not able to review it, return it 4 Fairness –Author may have different point of view / methodology / arguments –Judge from their school of thought not yours 4 Speed –Be fast, but do not rush. Author deserves a fair hearing

21 Ethics of refereeing (2) 4 Professional treatment –Act in the best interest of the author and conference/journal –Specific rather than vague criticism 4 Confidentiality –Cannot circulate paper –Cannot use without permission 4 Conflict of interest –Discuss with editor

22 Ethics of refereeing (3) 4 Honesty –About your expertise and confidence in appraisal 4 Courtesy –Constructive criticism –Non-inflammatory language –Suggest improvements

23 Need practice? 4 Euan Minto Prize –Make sure you read the notes that accompany the review form 4 Give your draft paper to a colleague for comments … … and be ready to provide some on his/her paper!