1. Rob Holte University of Alberta holte@cs.ualberta.ca
ICML’2003 Minitutorial on Research, ‘Riting, and Reviews
Reviews
Rob Holte
University of Alberta
January 11, 2019

2. Purpose Evaluate the paper’s scientific merit
Check the validity of the paper’s claims and evidence
Judge the paper’s relevance and significance
Provide constructive feedback to the author
January 11, 2019

3. Example: IJCAI’99 review form
1. How RELEVANT is this paper to AI researchers?
2. How SIGNIFICANT is this paper?
3. How ORIGINAL is this paper?
4. Is this paper technically SOUND?
5. How well is this paper PRESENTED?
Further comments, advice or explanations (Please be specific and
constructive, especially with respect to any negative judgements
above. Point to the section(s) where an error occurs, cite omitted
references, etc.)
January 11, 2019

4. Excerpts from ICML’03 form
GOALS/MOTIVATION. Does the submission state the goals of the research, including the criteria by which readers should evaluate the results?
Is the learning problem well-motivated?
CLAIMS/EVIDENCE. Do the authors make explicit claims or draw clear conclusions, and do they present reasonable evidence to support their position?
If the claims are theoretical, are the proofs correct?
If the claims are empirical, are the experiments appropriate, and do the results reveal the underlying reasons or causes for phenomena?
How might the authors strengthen their claims, evidence, or reasoning?
January 11, 2019

5. Example: COLT’97 instructions
FOUNDATIONAL/CONCEPTUAL CONTRIBUTION: Note things like a new model, new notion, new definition, new approach. Note the significance and reasons for this novelty (and note the absence of such a novelty).
TECHNICAL DEVELOPMENT:
- Introduces a new technique
- Novel use of known technique
- Talented use of known technique
- Traditional use of known technique
- Trivial technically
January 11, 2019

6. COLT’97 continued RELATION TO OPEN PROBLEMS:
Does the paper solve completely/partially/ or does it address an open question?
How important is this question?
(Central/ important/ interesting/ legitimate/ stupid).
How much effort has been invested before in solving it and by whom?
January 11, 2019

7. COLT’97 continued SOCIAL INTEREST IN PAPER:
Is it potentially interesting to the whole COLT community, to a major subarea, to everyone in a restricted area, interesting only to the authors.
HOW WILL IT CONTRIBUTE:
Fertilization, satisfy curiosity, who knows?
PAPER TYPE: Is it a
- First step (opens a new area)
- Last step (closes an important area)
- Giant step (makes essential progress)
- None of the above.
January 11, 2019

8. Good Reviews Polite Fair Concise Clear Constructive Specific
Well-documented
Represent the scientific community
January 11, 2019

9. Anonymity Reviewers are supposed to be anonymous
BUT, sometimes their identity becomes known or is at least guessed by the authors
Write your reviews so that you would not be embarrassed if your identity was revealed
January 11, 2019

10. “In my opinion…” Reviews are necessarily subjective
A paper must convince its readers that its claims are valid and significant
A reviewer is more careful, thorough and patient than an average reader
January 11, 2019

11. Reviews that “miss the point”
Not uncommon, very frustrating
Don’t blame the reviewer. Fix your paper.
January 11, 2019

12. Conference reviews short time frame
each reviewer has several papers to review
Expect reviews to be terse, less thorough, less satisfying
January 11, 2019