The peer review process and the task of a referee Olli Silvén University of Oulu Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Contents Introduction Peer review process Journals Conferences Research programmes The tasks of a referee Reviewing a research paper Preparing the referee report & recommendations Evaluating a research proposal Acting as an editor or program chairperson How to become a referee? Final words Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Introduction A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field Number of scientific papers and articles (2000): > 600 000 (ISI) About 50% in the fields of science and technology The number of papers and articles submitted for publication is much larger refereeing process selects the ones to be published Examples of acceptance rates after refereeing: IEEE journals: ~10-20% (large variance) IEEE conferences: ~10-50% Workshops: ~30%-90% Refereeing is also used in selecting research projects to be funded Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Introduction What is a sufficient contribution? new result, theoretical or experimental new insight novel synthesis of ideas useful survey useful tutorial What is not a sufficient contribution new, novel, useful badly written erroneous data MPI = Minimum Publishable Increment depends on the forum Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Peer review process Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution novelty, significance, correctness, readability Refereeing is public service to the scientific community professional obligation, carried out on volunteer basis requires high expertice helps in improving one’s own expertice ensures the integrity of science Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Peer review process of a journal submission publish editor author accept reject revise selection of associate editor reviews recommendations associate editors referees selection of referees checking of revised papers Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Peer review process of a conference submission program committee program chair author accept/ reject/ accept with revisions selection of the referees checking of revisions accept/reject/minor revision recommendations referees extra referees Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Peer review process of a workshop submit program committee program chair author accept/ reject refereeing checking of revisions extra referees Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Peer review process of a research programme submission steering committee proposer accept with partial funding/ reject referees Notice: not representative of TEKES or EU research programmes Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

The tasks of a referee The reviewer grades a paper based on its novelty, significance, correctness, and readability In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly Both positive and negative findings are summarized in a referee report confidential part only for the editor/program committee: information that could reveal the identity of the reviewer or in minor conflicts of interest non-confidential part for the author/program committee Learn from the other reviews, if they are sent to you after the process Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

The right attitude: I can learn something! Humbleness and an open mind needed; 100% self-confidence can be harmful Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided an elegantly written paper may have zero actual contribution a paper with broken English may contain a major new idea The papers recommended for acceptance should have novelty and be correct If the reviewer can’t check a fact or is unsure, this should be stated in the review report But don’t waste your time on analysing in detail a paper that is never publishable a single crucial error is enough Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper The paper to be reviewed is typically accompanied with a review form fill the five point scale questions last it is most important to write an itemized review report Relevance [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Originality Background knowledge of the subject and references Technical content Presentation Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper: analysis The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990) What is the purpose of the paper Is the problem clearly stated and have the key issues been pointed out? Is it clear what has been accomplished? Is the paper appropriate for the intended forum? If it is not, what could be a better choice? Is the goal significant = has the work been worth doing? Are the results just trivial variations or extensions of previous results? Are there any new ideas, or novelties in research methodology? Citation analysis using electronic libraries are a big help! Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper: analysis (cont’d) Is the method of approach clear and valid? Is there something fundamentally flawed in the approach? Are the assumptions realistic and does that matter? Is the method new? Can it be generalized to other problems? Again, electronic libraries are most useful. Is the actual execution of the research correct? Are the mathematics and statistics correct? Check! Have the simulations been described in sufficient detail for replication? What about the boundary conditions? Do the results make sense? This part may require considerable effort from the reviewer... Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper: analysis (cont’d) Are the conclusions correct? Wht are the applications or implications of the results and are the results analysed to an adequate depth? Is the presentation satisfactory? Is the paper readable? Is it structured according to the convenstions of scientific publications? What did you as the reviewer learn? If you didn’t learn anything, then the paper is not publishable (provided that you understood the paper) Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper: analyzing the references It is researcher’s professional obligation to cite prior work the manuscript being reviewed includes claims of novelties; regularly citing prior research the reviewer needs to check the validity of the claims most efficient to carry out the analysis using electronic libraries At minimum: Check what is found using the key words of the article Study the references you don’t know beforehand Check which recent papers cite the same references Check the references of those recent papers Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Reviewing a research paper: reference analysis example Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Writing the referee report No fixed rules exist, the following ones are according to (Smith 1990) Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions State the recommendation and its justification; the five point scale part of the evaluation form is not enough Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers Evaluate the significance and validity of the research goal Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here Make a clear recommendation for or against publication with justifications Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Compiling the recommendations Classification of papers (Smith 1990) Very significant; includes major results (<1% of all papers) Interesting work, a good contribution (<10%) Minor positive contribution (10-30%) Elegant and technically correct, but useless Neither elegant nor useful, but not wrong Wrong and misleading Unreadable, impossible to evaluate The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4) Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Research proposals A research proposal is a request for funding submitted to, e.g, Finnish Academy of Sciences European Commission Tekes other funding organization such as a foundation The key difference to reviewing research papers is that the reviewers also evaluate the proposers Not all organization use peer review as a means for selecting proposals for funding Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Evaluating research proposals The evaluation criteria vary between funding organizations Key criteria: Is the research topic significant? Are the goals realistic? Has the proposer sufficient expertice and facilities to reach the goals? Is the requested funding reasonable? Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Sidenote: Evaluation of EU RTD proposals Often arranged as evaluation meetings lasting for several days 3-4 reviewers evaluate each proposal about 30 questions to be answered with at least one sentence in addition a 5 point evaluation of each area consolidation meetings to analyze differences of opinion around 2 hours per proposal, each ~30 pages To be selected, none of the evaluated issues should fall below 4 points Self evaluation possible; evaluation instructions are public Key criteria to reject proposals: Lacking significance Quality of the consortium Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Acting as an editor or program chairperson The editor maintains correspondence with authors and referees finds new referees if the ones assigned fail to act in given time decides on acceptance, rejection or a revision round based on 2-4 review statements. should distribute all review statements to the referees receives occasional negative feedback Review is not a vote! The editor is likely to line himself according to the best justified recommendations Conference program committees often rely on the numerical evaluations, occasionally resorting to vote Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

How to become a referee Writing a publication that is cited is the most certain way to become a referee Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations Refereeing is very rewarding, helps to keep up-to-date and aware of developments in fields adjacent to ones own specialty Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu

Final words Good referee reports are valuable and free of charge help in improving the paper help in improving as a researcher help in improving as a referee Refereeing is a learning experience Scientific progress rests heavily on peer reviews Machine Vision and Media Processing Unit Department of Electrical Engineering and Infotech Oulu