1. The peer review process and the task of a referee
Olli Silvén
University of Oulu
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

2. Introduction
A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field
Number of scientific papers and articles (2004): > (ISI)
About 50% in the fields of science and technology ~ 4000 pages/day
The number of papers and articles submitted for publication is much larger
refereeing process selects the ones to be published
Examples of acceptance rates after refereeing:
IEEE journals: ~10-20% (large variance)
IEEE conferences: ~10-50% (very large variance)
IEEE workshops: ~30%-90%
Refereeing is also used in selecting research projects to be funded
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

3. Introduction (1) What is a sufficient contribution?
new result, theoretical or experimental
e.g ”we derive a method for estimating cross-correlations in complex lapped transform (CLT) domain”
new insight
novel synthesis of ideas
e.g. magnetic resonance imaging
useful survey
useful tutorial
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

4. Introduction (2) What is not a sufficient contribution badly written
new, novel, useful
badly written
erroneous data
non-sensical results (that occasionally get published)
(for a famous published example see: Alan D. Sokal, Transgressing
the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity,
Social/Text #46/47, pp (spring/summer 1996))
MPI = Minimum Publishable Increment depends on the forum
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

5. Peer review process
Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution
novelty, significance, correctness, readability
Refereeing is public service to the scientific community
professional obligation,
carried out on volunteer basis
requires high expertice
helps in improving one’s own expertice
ensures the integrity of science
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

6. Peer review process of a journal
submission
publish
editor
author
accept
reject
revise
selection of associate editor
Three ”API’s”!
reviews
recommendations
associate
editors
referees
selection of referees
checking of revised papers
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

7. Peer review process of a conference
submission
program
committee
program chair
author
accept/
reject/
accept with revisions
selection of the
referees
checking of
revisions
Two ”API’s”!
accept/reject/minor revision recommendations
referees
extra referees
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

8. Peer review process of a workshop
submit
program
committee
program chair
author
accept/
reject
refereeing
checking of
revisions
A single ”API”
extra referees
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

9. The tasks of a referee
The reviewer grades a paper based on its novelty, significance,
correctness, and readability
In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly
Both positive and negative findings are summarized in a referee report
* confidential part only for the editor/program committee:
information that could reveal the identity of the reviewer or
in minor conflicts of interest
* non-confidential part for the author/program committee
Learn from the other reviews, if they are sent to you after the process
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

10. The right attitude: I can learn something!
Humbleness and an open mind needed; 100% self-confidence absolutely harmful
Early assumptions on the correctness of the paper or the sufficiency
of its references should be avoided
an elegantly written paper may have zero actual contribution
a paper with broken English may contain a major new idea
The papers recommended for acceptance should have novelty and be correct
If the reviewer can’t check a fact or is unsure, this should be
stated in the review report
But don’t waste your time on analysing in detail a paper that is never publishable
a single crucial error is enough
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

11. Reviewing a research paper
The paper to be reviewed is typically accompanied with a review form
fill the five point scale questions last
it is most important to write an itemized review report
Relevance
[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent
Originality
Background knowledge of the subject and references
Technical content
Presentation
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

12. Reviewing a research paper: analysis
The analysis of a paper can be done by generating explanations to the
following eight points (Smith 1990)
What is the purpose of the paper
Is the problem clearly stated and have the key issues been pointed out?
Is it clear what has been accomplished?
Is the paper appropriate for the intended forum?
If it is not, what could be a better choice?
Is the goal significant = has the work been worth doing?
Are the results just trivial variations or extensions of previous results?
Are there any new ideas, or novelties in research methodology?
Citation analysis using electronic libraries are a big help!
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

13. Example of analysing a research paper (1)
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

14. Background of the paper
Most video coding standards rely on hybrid block based transform coding methods. A key technique in achieving good compression efficiency is motion estimation. The best method is ”full-search” via correlation, but it is very expensive. Other techniques compromise compression efficiency, but reduce computational complexity.
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

15. Example (2)
Understand the purpose of the paper = read the abstract and introduction
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

16. Example (3)
Is the goal significant = has the work been worth doing? O(N2*M2)->O(M2),
N typically 16, M typically 48*48
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

17. Reviewing a research paper: analysis (cont’d)
Is the method of approach clear and valid?
Is there something fundamentally flawed in the approach?
Are the assumptions realistic and does that matter?
Is the method new? Can it be generalized to other problems?
Again, electronic libraries are most useful.
Is the actual execution of the research correct?
Are the mathematics and statistics correct? Check!
Have the simulations been described in sufficient detail for replication?
What about the boundary conditions?
Do the results make sense?
This part may require considerable effort from the reviewer...
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

18. Example (4)
Is the method of approach clear and valid? Now, it is a mathematical one: Chapter II, Equation (3)
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

19. Reviewing a research paper: analysis (cont’d)
Are the conclusions correct?
Are he results analysed to an adequate depth?
What are the applications or implications of the results?
Is the presentation satisfactory?
Is the paper readable? Is it structured according to the conventions
of scientific publications?
What did you as the reviewer learn?
If you didn’t learn anything, then the paper is not publishable
(provided that you understood the paper)
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

20. Reviewing a research paper: analyzing the references
It is researcher’s professional obligation to cite prior work
the manuscript being reviewed includes claims of novelties; regularly citing prior research
the reviewer needs to check the validity of the claims
most efficient to carry out the analysis using electronic libraries
At minimum:
Check what is found using the key words of the article
Study the references you don’t know beforehand
Check which recent papers cite the same references
Check the references of those recent papers
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

21. References (start of a list of 41…)
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

22. Example (5) Koc & Liu 1998: DCT based motion estimation
Analysis of the references:
1 Check what is found using the key words of the article: not much...
Study the key references you don’t know beforehand:
Transform domain methods: Kuglin & Hines 1975, Thomas 1987, Girod 1993 Young & Kingsbury 1993, (this reviewer would add Lees & Henshaw 1986 Proc. SPIE 730, that has its foundation in optics)
Check which recent papers (untill 1997) cite the same references
No similar ideas found, just the ordinary ”whatever you do, I can do better” = perhaps 1 MPI unit contributions
Check the references of those recent papers
Make your conclusions on novelty
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

23. Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

24. A look at the most relevant references
(in Young & Kingsbury 1993)
We observe the same foundation as with Koc & Liu 1998
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

25. Example (6)
Who has cited Kuglin & Hines 1975? (… can’t find this, ISI database only from 1986…)
Who has cited Thomas 1987 (… can’t find this, it is a conference paper)
Who has cited Girod 1993? (seven references till 1997) Examples
Girod B, Steinbach E, Farber N Performance of the H.263 video compression standard J VLSI SIG PROCESS S 17 (2-3): NOV 1997
Ku CW, Chen LG, Chen CH, et al. Investigation of a visual telephone prototyping on personal computers IEEE T CONSUM ELECTR 42 (3): AUG 1996
Kuo CM, Hsieh CH, Jou YD, et al. Motion estimation for video compression using Kalman filtering IEEE T BROADCAST 42 (2): JUN 1996
KOKARAM AC, MORRIS RD, FITZGERALD WJ, et al. INTERPOLATION OF MISSING DATA IN IMAGE SEQUENCES IEEE T IMAGE PROCESS 4 (11): NOV 1995
………No similar contributions as in the paper we are refereeing were found………
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

26. Writing the referee report
No fixed rules exist, the following ones are according to (Smith 1990)
Most important: make your opinions clear; avoid ”perhaps” and ”maybe”;
evaluate the paper, not the author; itemize the contributions
State the recommendation and its justification; the five point scale part of the evaluation form is not enough
Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers
Evaluate the significance and validity of the research goal
Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here
Make a clear recommendation for or against publication with justifications
This analysis is exactly what you need for the literature review part of your thesis!
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

27. Compiling the recommendations
Classification of papers (Smith 1990)
Very significant; includes major results (<1% of all papers)
Interesting work, a good contribution (<10%)
Minor positive contribution (10-30%)
Elegant and technically correct, but useless
Neither elegant nor useful, but not wrong
Wrong and misleading
Unreadable, impossible to evaluate
The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

28. What to do with the referee reports on your own paper?
The quality of the referee reports vary a lot...
a good report provides additional references, and guidance to writing and further research
If you feel dissatisfied with the quality of the referee report,
especially, if it is a vague one, you may write to the editor, politely asking for
further clarifications as the subject of researh is important to you
first, however, very carefully check the facts in the report
the editor may select an additional reviewer
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

29. (Almost) Final Words
Writing a publication that is cited is the most certain way to become a referee
Good referee reports are valuable and free of charge
help in improving the paper
help in improving as a researcher
help in improving as a referee
Refereeing is a learning experience: helps to keep the referee up-to-date
Scientific progress rests heavily on peer reviews
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

30. How to avoid the reviewer’s axe (1)
Alternative guidelines for authors and (reviewers) by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003)
(Almost) Nothing is New
= ”First, figure out what you have done. Then go to the library and find it there!” ….and tell this to the reviewers
Rely on the Believability Index
= write the paper in order of decreasing believability
Watch for Gambling Words
=”obviously, probably, certainly, undoubtedly” = you don’t know what you are talking about
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering

31. How to avoid the reviewer’s axe (2)
Guidelines by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003)
Don’t be a Longfellow (a famous poet/storyteller)
= don’t make conclusions of your results prematurely in your paper, that is, before ”Discussion”
Don’t Pull Rabbits out of Hats
= scientific articles are no detective stories
Mine All the Gold
= try to extract everything you can from your data
Remember: Reviewers are inarticulate and authors (somewhat) paranoid
~ next time write a better paper
Machine Vision and Media Processing Unit
Department of Electrical and Information Engineering