1. Christoph F. Eick: ML Project2 2013 Post-Analysis 1 Project2 Post Analysis —General Things Reviewing is about voicing your opinion about the paper! Reviews that do not express any opinion—except for their summarization value—are useless. However, if you make evaluations you should do your best to justify your evaluation; here are some bad examples: “ the paper fails to review important related work.” instead of specifically listing what important papers were not referenced! “ the paper experimental evaluation is not convincing ” without saying why. “ the paper fails to evaluate alternative approaches” without saying which approaches should be evaluated. You should not be afraid to mention difficulties you had in reading understanding the paper! A lot of reviews were incomplete.

2. Christoph F. Eick: ML Project2 2013 Post-Analysis 2 Project2 Post Analysis —Paper Summaries The paper summary should briefly introduce the research area of the paper and then make a clear statement about the paper’s contribution and why the author believes this paper improves the state of the art. Paper summaries should be stand alone and readers should be able to understand those without reading the paper itself. 50% of the individual paper summaries are very short, not specific enough, and about 40% are not easy to comprehend. Paper summary and paper evaluation should be clearly separated. Paper summaries should be specific and should not lose themselves in generalities. Do not use sentences like that “ The paper introduces a novel data mining technique and applies it to the challenging application ”. Do not introduce terms without explaining those e.g. “…introduces a cocktail approach…” “…uses a hybrid approach…”—explain what approaches are combined

3. Christoph F. Eick: ML Project2 2013 Post-Analysis 3 Project2 Post Analysis —Paper Evaluation Most reviews did a poor job with assessing the novelty of the paper, as they were lacking a clear statement about which extend a paper improves the state of the art, if it creates new bridge between research fields and extends the research field’s knowledge base. Reviews should be clearly structured; some students did not use subsections and as their writing is not always the best, it was often not clear what a particular paragraph is currently assessing; e.g. is this about about novelty or technical. Voice your own opinion, but do not discriminate: “…One author is a professor in the MIT Math Department. So it’s unnecessary to challenge all the formulas.” “… The references cited in the paper are a bit old ” Comment mainly on the specific paper and not the research field in general! Education impact: many reviews fail to make a clear statement about what “ you have learnt from reading the paper ” and what other groups could learn from it.

4. Christoph F. Eick: ML Project2 2013 Post-Analysis 4 Project2 Post Analysis —Other Issues Websearch and in general: Have a reference list at the end of your review; also if you found other relevant papers, explicitly mention if they are not referenced in the paper. Most reviews did a poor job in summarizing websearch results and other did not do much websearch Some students are confused what “broader impact” is all about. Avoid writing trivialities: “ The author has acknowledged what he has accomplished.” “ The author summarized the paper’s accomplishment in the paper’s abstract ” Do not paste large portions of the paper into your review; summarize it instead. Do not trust anything the authors claim to be the case! Look for inconsistencies which point to the fact that there is something fundamentally wrong with the paper.

5. Christoph F. Eick: ML Project2 2013 Post-Analysis 5 Project2 Post Analysis —Other Issues Some reviews do not follow the suggested template! Some individual reviews are very hard to understand; the writing of the group reviews was somewhat better Specific Comments, centering on Any inconsistencies Parts you did not understand Parts were you disagree with the authors’ conclusion or assessment Errors and things that are missing Asking authors specific questions Making suggestions on how to improve the paper are an important part of a good review. Moreover, assess the paper from the a more emotional point of view; e.g. does the paper does a good job to get people excited about the research field and the paper it self. Finally, consider the paper’s entertainment value and if it is high, add a positive paragraph to your review.