1. The Reviewing Process Marie desJardins AAAI-13 Panel Conference Reviewing Best Practices

2. Conference Reviewing u Program committees  Selection process  Different models/roles: n Senior vs. area chair vs. regular members u Paper assignments  Keyword-based  Similarity-based (compare to reviewer’s published papers)  Self-selection (bidding) u Discussion and author response u Decisions 7/17/13 2 Review Process and Quality

3. Purpose of a Review u Evaluate the paper’s scientific merit  Check the validity of the paper’s claims and evidence  Judge the paper’s relevance and significance u Provide constructive feedback to the author 7/17/13 3 Review Process and Quality

4. AAAI-13 Review Criteria u Goal: identify “interesting but flawed” papers and make sure they are reviewed fairly  Different ways in which a paper can make a contribution  Avoid a conference full of “technically acceptable but uninteresting and incremental” papers u Quality criteria:  Technical quality  Experimental analysis  Formal analysis  Clarity/presentation u Innovation criteria:  Novelty/innovation of question / solution u Impact criteria:  Breadth of interest  Potential for impact 7/17/13 Review Process and Quality 4

5. Good Reviews Are... u Polite u Fair u Concise u Clear u Constructive u Specific u Well documented u Represent the scientific community 7/17/13 5 Review Process and Quality