The peer review process and the task of a referee Olli Silvén University of Oulu Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Introduction A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field Number of scientific papers and articles (2004): > 630 000 (ISI) About 50% in the fields of science and technology ~ 4000 pages/day The number of papers and articles submitted for publication is much larger refereeing process selects the ones to be published Examples of acceptance rates after refereeing: IEEE journals: ~10-20% (large variance) IEEE conferences: ~10-50% (very large variance) IEEE workshops: ~30%-90% Refereeing is also used in selecting research projects to be funded Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Introduction (1) What is a sufficient contribution? new result, theoretical or experimental e.g ”we derive a method for estimating cross-correlations in complex lapped transform (CLT) domain” new insight novel synthesis of ideas e.g. magnetic resonance imaging useful survey useful tutorial Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Introduction (2) What is not a sufficient contribution badly written new, novel, useful badly written erroneous data non-sensical results (that occasionally get published) (for a famous published example see: Alan D. Sokal, Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity, Social/Text #46/47, pp. 217-252 (spring/summer 1996)) MPI = Minimum Publishable Increment depends on the forum Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Peer review process Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution novelty, significance, correctness, readability Refereeing is public service to the scientific community professional obligation, carried out on volunteer basis requires high expertice helps in improving one’s own expertice ensures the integrity of science Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Peer review process of a journal submission publish editor author accept reject revise selection of associate editor Three ”API’s”! reviews recommendations associate editors referees selection of referees checking of revised papers Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Peer review process of a conference submission program committee program chair author accept/ reject/ accept with revisions selection of the referees checking of revisions Two ”API’s”! accept/reject/minor revision recommendations referees extra referees Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Peer review process of a workshop submit program committee program chair author accept/ reject refereeing checking of revisions A single ”API” extra referees Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

The tasks of a referee The reviewer grades a paper based on its novelty, significance, correctness, and readability In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly Both positive and negative findings are summarized in a referee report * confidential part only for the editor/program committee: information that could reveal the identity of the reviewer or in minor conflicts of interest * non-confidential part for the author/program committee Learn from the other reviews, if they are sent to you after the process Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

The right attitude: I can learn something! Humbleness and an open mind needed; 100% self-confidence absolutely harmful Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided an elegantly written paper may have zero actual contribution a paper with broken English may contain a major new idea The papers recommended for acceptance should have novelty and be correct If the reviewer can’t check a fact or is unsure, this should be stated in the review report But don’t waste your time on analysing in detail a paper that is never publishable a single crucial error is enough Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Reviewing a research paper The paper to be reviewed is typically accompanied with a review form fill the five point scale questions last it is most important to write an itemized review report Relevance [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Originality Background knowledge of the subject and references Technical content Presentation Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Reviewing a research paper: analysis The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990) What is the purpose of the paper Is the problem clearly stated and have the key issues been pointed out? Is it clear what has been accomplished? Is the paper appropriate for the intended forum? If it is not, what could be a better choice? Is the goal significant = has the work been worth doing? Are the results just trivial variations or extensions of previous results? Are there any new ideas, or novelties in research methodology? Citation analysis using electronic libraries are a big help! Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example of analysing a research paper (1) Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Background of the paper Most video coding standards rely on hybrid block based transform coding methods. A key technique in achieving good compression efficiency is motion estimation. The best method is ”full-search” via correlation, but it is very expensive. Other techniques compromise compression efficiency, but reduce computational complexity. Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example (2) Understand the purpose of the paper = read the abstract and introduction Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example (3) Is the goal significant = has the work been worth doing? O(N2*M2)->O(M2), N typically 16, M typically 48*48 Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Reviewing a research paper: analysis (cont’d) Is the method of approach clear and valid? Is there something fundamentally flawed in the approach? Are the assumptions realistic and does that matter? Is the method new? Can it be generalized to other problems? Again, electronic libraries are most useful. Is the actual execution of the research correct? Are the mathematics and statistics correct? Check! Have the simulations been described in sufficient detail for replication? What about the boundary conditions? Do the results make sense? This part may require considerable effort from the reviewer... Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example (4) Is the method of approach clear and valid? Now, it is a mathematical one: Chapter II, Equation (3) Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Reviewing a research paper: analysis (cont’d) Are the conclusions correct? Are he results analysed to an adequate depth? What are the applications or implications of the results? Is the presentation satisfactory? Is the paper readable? Is it structured according to the conventions of scientific publications? What did you as the reviewer learn? If you didn’t learn anything, then the paper is not publishable (provided that you understood the paper) Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Reviewing a research paper: analyzing the references It is researcher’s professional obligation to cite prior work the manuscript being reviewed includes claims of novelties; regularly citing prior research the reviewer needs to check the validity of the claims most efficient to carry out the analysis using electronic libraries At minimum: Check what is found using the key words of the article Study the references you don’t know beforehand Check which recent papers cite the same references Check the references of those recent papers Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

References (start of a list of 41…) Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example (5) Koc & Liu 1998: DCT based motion estimation Analysis of the references: 1 Check what is found using the key words of the article: not much... Study the key references you don’t know beforehand: Transform domain methods: Kuglin & Hines 1975, Thomas 1987, Girod 1993 Young & Kingsbury 1993, (this reviewer would add Lees & Henshaw 1986 Proc. SPIE 730, that has its foundation in optics) Check which recent papers (untill 1997) cite the same references No similar ideas found, just the ordinary ”whatever you do, I can do better” = perhaps 1 MPI unit contributions Check the references of those recent papers Make your conclusions on novelty Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

A look at the most relevant references (in Young & Kingsbury 1993) We observe the same foundation as with Koc & Liu 1998 Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Example (6) Who has cited Kuglin & Hines 1975? (… can’t find this, ISI database only from 1986…) Who has cited Thomas 1987 (… can’t find this, it is a conference paper) Who has cited Girod 1993? (seven references till 1997) Examples Girod B, Steinbach E, Farber N Performance of the H.263 video compression standard J VLSI SIG PROCESS S 17 (2-3): 101-111 NOV 1997 Ku CW, Chen LG, Chen CH, et al. Investigation of a visual telephone prototyping on personal computers IEEE T CONSUM ELECTR 42 (3): 750-759 AUG 1996 Kuo CM, Hsieh CH, Jou YD, et al. Motion estimation for video compression using Kalman filtering IEEE T BROADCAST 42 (2): 110-116 JUN 1996 KOKARAM AC, MORRIS RD, FITZGERALD WJ, et al. INTERPOLATION OF MISSING DATA IN IMAGE SEQUENCES IEEE T IMAGE PROCESS 4 (11): 1509-1519 NOV 1995 ………No similar contributions as in the paper we are refereeing were found……… Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Writing the referee report No fixed rules exist, the following ones are according to (Smith 1990) Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions State the recommendation and its justification; the five point scale part of the evaluation form is not enough Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers Evaluate the significance and validity of the research goal Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here Make a clear recommendation for or against publication with justifications This analysis is exactly what you need for the literature review part of your thesis! Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

Compiling the recommendations Classification of papers (Smith 1990) Very significant; includes major results (<1% of all papers) Interesting work, a good contribution (<10%) Minor positive contribution (10-30%) Elegant and technically correct, but useless Neither elegant nor useful, but not wrong Wrong and misleading Unreadable, impossible to evaluate The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4) Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

What to do with the referee reports on your own paper? The quality of the referee reports vary a lot... a good report provides additional references, and guidance to writing and further research If you feel dissatisfied with the quality of the referee report, especially, if it is a vague one, you may write to the editor, politely asking for further clarifications as the subject of researh is important to you first, however, very carefully check the facts in the report the editor may select an additional reviewer Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

(Almost) Final Words Writing a publication that is cited is the most certain way to become a referee Good referee reports are valuable and free of charge help in improving the paper help in improving as a researcher help in improving as a referee Refereeing is a learning experience: helps to keep the referee up-to-date Scientific progress rests heavily on peer reviews Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

How to avoid the reviewer’s axe (1) Alternative guidelines for authors and (reviewers) by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003) (Almost) Nothing is New = ”First, figure out what you have done. Then go to the library and find it there!” ….and tell this to the reviewers Rely on the Believability Index = write the paper in order of decreasing believability Watch for Gambling Words =”obviously, probably, certainly, undoubtedly” = you don’t know what you are talking about Machine Vision and Media Processing Unit Department of Electrical and Information Engineering

How to avoid the reviewer’s axe (2) Guidelines by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003) Don’t be a Longfellow (a famous poet/storyteller) = don’t make conclusions of your results prematurely in your paper, that is, before ”Discussion” Don’t Pull Rabbits out of Hats = scientific articles are no detective stories Mine All the Gold = try to extract everything you can from your data Remember: Reviewers are inarticulate and authors (somewhat) paranoid ~ next time write a better paper Machine Vision and Media Processing Unit Department of Electrical and Information Engineering