CMPUT Teaching and Research Methods1 CMPUT603 - Fall 2005 Topic2: Refereeing (After Alan J. Smith, “The Task of the Referee”, IEEE Computer, April, 1989, pp ) José Nelson Amaral et al.

CMPUT Teaching and Research Methods2 Role of the Referee zWhat is the role of the referee? yFor an acceptance recommendation: xIs the referee responsible for the correctness of the paper? yFor a rejection: xWhat should be the goal of the report? yDoes the quality of the report affect the referee reputation? How?

CMPUT Teaching and Research Methods3 Referee Report Structure zSummarize your recommendation zSummarize the point of the paper zEvaluate the significance of the research zEvaluate the quality of the research yMethodology, techniques, accuracy, presentation zOverall recommendation yOn a rejection, clearly state the reasons yMake the strength of your opinion clear yOn acceptance, list required and suggested changes.

CMPUT Teaching and Research Methods4 Evaluation of a Research Paper zWhat is the purpose of the paper? zIs the paper appropriate? zIs the goal significant? zIs the method of approach valid? zIs the execution of the research correct? zAre the conclusions supported by the data? zIs the presentation satisfactory? zWhat did you learn?

CMPUT Teaching and Research Methods5 Evaluation of a Research Paper (categories) zMajor results zGood, solid, interesting work zMinor, but positive, contribution to knowledge zElegant and technically correct but useless zNeither elegant nor useful, but not wrong zWrong and misleading zSo badly written that technical evaluation is impossible

CMPUT Teaching and Research Methods6 Comparative Evaluation zWhat are the standards of this journal or conference? yIf you recommend a revision (minor or major), will you see the paper again? yIf you do, what should you check for?

CMPUT Teaching and Research Methods7 Evaluating a Tutorial zAre the title and abstract adequate? zIs the scope too wide, too narrow, too bizarre? zDoes the paper have a consistent theme? zIs the material correct? zIs the coverage too simple or too sophisticated? zIs the paper well-written and clear?

CMPUT Teaching and Research Methods8 Evaluating a Survey zIs the author an expert in the field? zIs the material integrated in a consistent manner? (Annotated bibliographies are not interesting) zIs the coverage balanced and thorough? zIs all the important literature cited? zIs the presentation biased, slanted, selective?

CMPUT Teaching and Research Methods9 Other Important Issues zSimultaneous submissions, prior publication, and unrevised retries zAcknowledgments and plagiarism zTimely response and returning a paper zThe author’s reputation zConfidentiality zConflicts of interest zThe editor’s role zThe program chair’s role