1. The peer review process and the task of a referee
Olli Silvén
University of Oulu
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

2. Contents Introduction Peer review process Journals Conferences
Research programmes
The tasks of a referee
Reviewing a research paper
Preparing the referee report & recommendations
Evaluating a research proposal
Acting as an editor or program chairperson
How to become a referee?
Final words
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

3. Introduction
A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field
Number of scientific papers and articles (2000): > (ISI)
About 50% in the fields of science and technology
The number of papers and articles submitted for publication is much larger
refereeing process selects the ones to be published
Examples of acceptance rates after refereeing:
IEEE journals: ~10-20% (large variance)
IEEE conferences: ~10-50%
Workshops: ~30%-90%
Refereeing is also used in selecting research projects to be funded
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

4. Introduction What is a sufficient contribution?
new result, theoretical or experimental
new insight
novel synthesis of ideas
useful survey
useful tutorial
What is not a sufficient contribution
new, novel, useful
badly written
erroneous data
MPI = Minimum Publishable Increment depends on the forum
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

5. Peer review process
Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution
novelty, significance, correctness, readability
Refereeing is public service to the scientific community
professional obligation,
carried out on volunteer basis
requires high expertice
helps in improving one’s own expertice
ensures the integrity of science
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

6. Peer review process of a journal
submission
publish
editor
author
accept
reject
revise
selection of associate editor
reviews
recommendations
associate
editors
referees
selection of referees
checking of revised papers
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

7. Peer review process of a conference
submission
program
committee
program chair
author
accept/
reject/
accept with revisions
selection of the
referees
checking of
revisions
accept/reject/minor revision recommendations
referees
extra referees
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

8. Peer review process of a workshop
submit
program
committee
program chair
author
accept/
reject
refereeing
checking of
revisions
extra referees
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

9. Peer review process of a research programme
submission
steering
committee
proposer
accept with partial funding/
reject
referees
Notice: not representative of TEKES or EU research programmes
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

10. The tasks of a referee
The reviewer grades a paper based on its novelty, significance,
correctness, and readability
In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly
Both positive and negative findings are summarized in a referee report
confidential part only for the editor/program committee:
information that could reveal the identity of the reviewer or
in minor conflicts of interest
non-confidential part for the author/program committee
Learn from the other reviews, if they are sent to you after the process
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

11. The right attitude: I can learn something!
Humbleness and an open mind needed; 100% self-confidence can be harmful
Early assumptions on the correctness of the paper or the sufficiency
of its references should be avoided
an elegantly written paper may have zero actual contribution
a paper with broken English may contain a major new idea
The papers recommended for acceptance should have novelty and be correct
If the reviewer can’t check a fact or is unsure, this should be
stated in the review report
But don’t waste your time on analysing in detail a paper that is never publishable
a single crucial error is enough
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

12. Reviewing a research paper
The paper to be reviewed is typically accompanied with a review form
fill the five point scale questions last
it is most important to write an itemized review report
Relevance
[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent
Originality
Background knowledge of the subject and references
Technical content
Presentation
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

13. Reviewing a research paper: analysis
The analysis of a paper can be done by generating explanations to the
following eight points (Smith 1990)
What is the purpose of the paper
Is the problem clearly stated and have the key issues been pointed out?
Is it clear what has been accomplished?
Is the paper appropriate for the intended forum?
If it is not, what could be a better choice?
Is the goal significant = has the work been worth doing?
Are the results just trivial variations or extensions of previous results?
Are there any new ideas, or novelties in research methodology?
Citation analysis using electronic libraries are a big help!
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

14. Reviewing a research paper: analysis (cont’d)
Is the method of approach clear and valid?
Is there something fundamentally flawed in the approach?
Are the assumptions realistic and does that matter?
Is the method new? Can it be generalized to other problems?
Again, electronic libraries are most useful.
Is the actual execution of the research correct?
Are the mathematics and statistics correct? Check!
Have the simulations been described in sufficient detail for replication?
What about the boundary conditions?
Do the results make sense?
This part may require considerable effort from the reviewer...
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

15. Reviewing a research paper: analysis (cont’d)
Are the conclusions correct?
Wht are the applications or implications of the results and are the results
analysed to an adequate depth?
Is the presentation satisfactory?
Is the paper readable? Is it structured according to the convenstions
of scientific publications?
What did you as the reviewer learn?
If you didn’t learn anything, then the paper is not publishable
(provided that you understood the paper)
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

16. Reviewing a research paper: analyzing the references
It is researcher’s professional obligation to cite prior work
the manuscript being reviewed includes claims of novelties; regularly citing prior research
the reviewer needs to check the validity of the claims
most efficient to carry out the analysis using electronic libraries
At minimum:
Check what is found using the key words of the article
Study the references you don’t know beforehand
Check which recent papers cite the same references
Check the references of those recent papers
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

17. Reviewing a research paper: reference analysis example
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

18. Writing the referee report
No fixed rules exist, the following ones are according to (Smith 1990)
Most important: make your opinions clear; avoid ”perhaps” and ”maybe”;
evaluate the paper, not the author; itemize the contributions
State the recommendation and its justification; the five point scale part of the evaluation form is not enough
Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers
Evaluate the significance and validity of the research goal
Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here
Make a clear recommendation for or against publication with justifications
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

19. Compiling the recommendations
Classification of papers (Smith 1990)
Very significant; includes major results (<1% of all papers)
Interesting work, a good contribution (<10%)
Minor positive contribution (10-30%)
Elegant and technically correct, but useless
Neither elegant nor useful, but not wrong
Wrong and misleading
Unreadable, impossible to evaluate
The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

20. Research proposals
A research proposal is a request for funding submitted to, e.g,
Finnish Academy of Sciences
European Commission
Tekes
other funding organization such as a foundation
The key difference to reviewing research papers is that
the reviewers also evaluate the proposers
Not all organization use peer review as a means for selecting proposals for funding
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

21. Evaluating research proposals
The evaluation criteria vary between funding organizations
Key criteria:
Is the research topic significant?
Are the goals realistic?
Has the proposer sufficient expertice and facilities to reach the goals?
Is the requested funding reasonable?
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

22. Sidenote: Evaluation of EU RTD proposals
Often arranged as evaluation meetings lasting for several days
3-4 reviewers evaluate each proposal
about 30 questions to be answered with at least one sentence
in addition a 5 point evaluation of each area
consolidation meetings to analyze differences of opinion
around 2 hours per proposal, each ~30 pages
To be selected, none of the evaluated issues should fall below 4 points
Self evaluation possible; evaluation instructions are public
Key criteria to reject proposals:
Lacking significance
Quality of the consortium
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

23. Acting as an editor or program chairperson
The editor
maintains correspondence with authors and referees
finds new referees if the ones assigned fail to act in given time
decides on acceptance, rejection or a revision round based on 2-4 review statements.
should distribute all review statements to the referees
receives occasional negative feedback
Review is not a vote! The editor is likely to line himself according to
the best justified recommendations
Conference program committees often rely on the numerical
evaluations, occasionally resorting to vote
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

24. How to become a referee
Writing a publication that is cited is the most certain way to become a referee
Coordination or technical coordination of an EU RTD project is a direct
road to proposal evaluations
Refereeing is very rewarding, helps to keep up-to-date and
aware of developments in fields adjacent to ones own specialty
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu

25. Final words Good referee reports are valuable and free of charge
help in improving the paper
help in improving as a researcher
help in improving as a referee
Refereeing is a learning experience
Scientific progress rests heavily on peer reviews
Machine Vision and Media Processing Unit
Department of Electrical Engineering and Infotech Oulu