How to Really Review Papers CS 8803 AIC. Prvulovic: CORD 2 Paper Reviewing Algorithm Read the paper Think about it Take a look at related work Leave it.

Slides:



Advertisements
Similar presentations
How to write a review. Outline What is a review? Why should you review? How do you review a paper? What not to do? What are the dilemmas? Case study.
Advertisements

Problem Solving in Science
Foundations and Strategies Attention Investment CS352.
Solving Quadratic Equations by Factoring
Turnaround Interview® Turn off your people problems, not your people.
Study What’s that? Why? How?. School does not “do stuff” to you You do stuff to school – it is active You do stuff to school – it is active This is about.
4. Evaluating a paper 1Prof. Talal Aburjai. A thorough understanding and evaluation of a paper involves answering several questions: a. What questions.
Working on a Mini-Project Anders P. Ravn/Arne Skou Computer Science Aalborg University February 2011.
SENG 531: Labs TA: Brad Cossette Office Hours: Monday, Wednesday.
Reviewing the work of others Referee reports. Components of a referee report Summary of the paper Overall evaluation Comments about content Comments about.
CS 197 Computers in Society Fall, Welcome, Freshmen!
How to present your project?. A simple 3-part template A.Introduction B.Body C.Conclusion.
Supplementing lectures with additional online materials Matthew Juniper, CUED June 2007.
How to read and comment on a paper? 1.How to read English paper? 2.How to comment on a paper? 3.What are the bad practices you should avoid 4.Your feedback?
Project Workshops Results and Evaluation. General The Results section presents the results to demonstrate the performance of the proposed solution. It.
Peer Review for Addiction Journals Robert L. Balster Editor-in-Chief Drug and Alcohol Dependence.
Advanced Research Methodology
Making quant tutorials work! DOs and DON'Ts of classroom inversion Dr Ralf Becker Economics, School of Social Sciences The University of Manchester.
The Research Process Interpretivist Positivist
Hypothesis Testing. A Research Question Everybody knows men are better drivers than women. Hypothesis: A tentative explanation that accounts for a set.
How to Write a Literature Review
ASPLOS 2015 Debate Onur Mutlu
Essay and Report Writing. Learning Outcomes After completing this course, students will be able to: Analyse essay questions effectively. Identify how.
’Fess up Taking Responsibility for your Actions. Everyone makes mistakes. You wouldn't be human if you didn't. The fear of making mistakes can prevent.
An Introduction to Empirical Investigations. Aims of the School To provide an advanced treatment of some of the major models, theories and issues in your.
Using Visual Aids in a Speech. Visual Aids can be powerful when giving a speech. However, make sure they will improve your speech. Ask yourself the following.
The Conclusion and The Defense CSCI 6620 Spring 2014 Thesis Projects: Chapters 11 and 12 CSCI 6620 Spring 2014 Thesis Projects: Chapters 11 and 12.
Software Engineering Experimentation Rules for Reviewing Papers Jeff Offutt See my editorials 17(3) and 17(4) in STVR
Discussion: So Who Won. Announcements Looks like you’re turning in reviews… good! – Some of you are spending too much time on them!! Key points, what.
Topic of Your Group Names of Group Member (alphabetical order) 1.
Making Decisions uCode: October Review What are the differences between: o BlueJ o Java Computer objects represent some thing or idea in the real.
New Junior Cycle. Aims To understand Assessment for learning (AFL) strategies. To understand two types of feedback. To be aware of the importance of descriptive.
Activity 5: Lead this Group! Instructions: There are many ways for a leader to respond to situations that have developed within a group. How would you.
Fundamentals of Scientific Research Dr. Samer Odeh Hanna (Ph.D.)
© 2015 albert-learning.com Phrases For Business English.
Life Science Chapter 1 Section 1.
Problem Solving. o You notice something, and wonder why it happens. o You see something and wonder what causes it. o You want to know how or why something.
 An article review is written for an audience who is knowledgeable in the subject matter instead of a general audience  When writing an article review,
Thomas HeckeleiPublishing and Writing in Agricultural Economics 1 Observations on assignment 4 - Reviews General observations  Good effort! Some even.
Opening. Two heads Before proceeding find a partner to go through the next slides with you. If you don’t have a partner be sure to share this later with.
Guide for AWS Reviewers Lois A. Killewich, MD PhD AWS AJS Editorial Board.
Writing a Science or Engineering Paper: It is just a story Frank Shipman Department of Computer Science Texas A&M University.
INFOMGP Student names and numbers Papers’ references Title.
SLAMS To write a good answer to a question, SLAM it!!
 Problem solving involves a number of well- defined steps, which are as follows:  Define the problem.  Analyze the problem.  Identify and evaluate.
Midterm/Final Presentation Project Name Students: [Name1], [Name2] Supervisor: [SV Name] Context: Project [A/B/Special] Semester: Winter/Spring, Year Date:
Questioning as Formative Assessment: GRECC Math Alliance February 4 th - 7 th, 2008.
1 Taking Notes. 2 STOP! Have I checked all your Source cards yet? Do they have a yellow highlighter mark on them? If not, you need to finish your Source.
ATTACKING THE (SAR) OPEN ENDED RESPONSE. Get out a sheet of paper(or 2?)! Your responses to the questions on this power point will be your SAR test grade.
Pitfalls of your first paper Shu Cai Institute of Computing Technology, Chinese Academy of Sciences
Sept 17, 2007C.Watters 1 Reviewing Published Articles.
Danger Signs of Reading From pages LEARNING TARGET I can identify the danger signs of reading. I can practice ways to improve reading comprehension.
Revising Your Paper Paul Lewis With thanks to Mark Weal.
Objection Handling. Agenda Seven Steps to handle objections 10 Common objections Questions.
Separation Lab overview  Given a mixture of Sand, Salt, Poppy Seeds and Iron Filings, design and execute a separation method.  Write a proposed procedure.
CS 664 Sample Presentation
Scientific Method What is the Scientific Method?
Writing for Academic Journals
How to Get Your Paper Rejected
CAPE INFORMATION TECHNOLOGY
How to Really Review Papers
Ms. Esquibel Science Teacher Upper Grades
How to Get Your Paper Rejected
The Historical Investigation
Strategic Presentation
CAPE INFORMATION TECHNOLOGY
How to Read a Paper (Practice: CCS’14)
Presentation and project
CS 6640 Sample Presentation
Presentation and project
Presentation transcript:

How to Really Review Papers CS 8803 AIC

Prvulovic: CORD 2 Paper Reviewing Algorithm Read the paper Think about it Take a look at related work Leave it alone for a few days Read it again Write the review

Prvulovic: CORD 3 Thinking about it What is the paper trying to do –Observation/Motivation Programs often do X… In the future, Y won’t work… –Implementation Here’s how to exploit this observation –Evaluation We get great perf/power/reliaility/… improvement, or It didn’t work and here’s why

Prvulovic: CORD 4 Contributions Did you gain an important insight –A new problem? –A new way to look at a problem? –Counter-intuitive or unexpected? Did it propose a good solution to a problem? –Explanation why solution should work –Explanation why it’s a better solution Did it provide a good quantitative evaluation –Should work in real processors? –Should it gain or lose relevance in the future? –Should work on real applications?

Prvulovic: CORD 5 Summary Like a paper abstract –Short and self-contained (no undefined acronyms) –Short high-level description of contribution But without the “sales pitch” –Avoid quoting numbers –What is really the contribution

Prvulovic: CORD 6 Good points Successful contributions –If you learned something, that’s a good thing –If could be useful for real systems, that’s good List the most important things –If you only list “Well written”, paper is really bad There are ALWAYS good points

Prvulovic: CORD 7 Bad points Motivation problems –Unclear whether there is a problem –Unclear why it’s an important problem Implementation problems –Unclear why solve the problem this way –Unclear what tradeoffs are Evaluation problems –We all know about these Organization problems –Difficult to follow? List the most important bad points –If “difficult to read” is the only problem, it’s a great paper

Prvulovic: CORD 8 Questions to answer Important questions only –If answer can’t change your recommendation don’t ask the question here (see next slide) Can it be answered in a few days? –If lots of work needed to answer, don’t ask here Unsure if you got something right? –If decision depends on something, and you are not sure if your understanding is correct DO ask it here

Prvulovic: CORD 9 “General Comments” Section Help the authors improve the paper Always be constructive –Even if paper is hopeless: “The problem of cache conflicts in 1024-way set associative caches is not very important, and the authors should focus on lower-associativity caches” All grammar/writing/organization issues go here!

Prvulovic: CORD 10 Confidential Comments Only put things that are important and would disclose your identity –“My group did this already in 1999 and we published it in ISCA that year”

Prvulovic: CORD 11 Scoring (numbers) Compare to other papers in the same journal/conference/class So, compare to other papers in this class Weak accept –Similar in quality to lower half of accepted papers Strong accept –Similar to upper half of accepted papers Neutral/Undecided –Would be in bottom 10% of accepted papers Strong reject –You would argue against accepting it

Prvulovic: CORD 12 Always the big picture Don’t focus on minor issues –Very easy to find lots of these Focus on make-or-break issues –If X wasn’t there, it would be a much worse paper –If X done right, it would be a much better paper Always consider the space –Don’t ask for it if you don’t think something else should be removed or made shorter