ICSE 2016 Reviewing Process Laurie and Willem

Welcome Thanks for serving! 28 Program Board members – In charge of +-22 papers – Full review for +-6 of those 89 Program Committee members – +-16 papers to review

General Philosophy Program Committee does the main reviewing Program Board keeps process on track and leads the discussions PB then communicates PC decisions at the face- to-face meeting – Extra PB member does a checks&balances review Main purpose of the meeting is to determine where the bar is for acceptance – For some problem cases that had no consensus during the PC reviewing period, decisions will be made Chairs sit back and relax!

Author view All papers get at least 3 reviews – Some will get 4 (extra from PB member) All papers will get a chance at a rebuttal – Right after PC reviewing, i.e. will come about 3 days after the start of online discussion All papers will get a summary of the decision Author feedback will be solicited

Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Overall Process

Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Program Board Involvement

Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Program Committee Involvement Online Discussion

Program Board Actions Overview Bidding Monitor Reviewing [Chairs will as well] – Make sure reviews are on time Drive online discussion PB Reviews – Full review, i.e. filling in the reviewing form PB Meeting

Program Board Actions Online Discussion Drive online discussion – Summarize views (including rebuttal) and kick off the conversation – Eliminate clear rejects first No support is a reject – Try to find consensus No consensus is fine at this point – Consensus is based on all 3 reviewers and the PB member’s opinion i.e. PB member is allowed an opinion, but it doesn’t carry more weight than the PC Write a discussion summary at the end of the discussion period – Select one of two outcomes: reject or move on Either consensus accept or no consensus moves on – Rejection summaries will be included in author feedback Summaries of papers moving on will be superseded by edited summaries later in the process

Program Board Actions Phase 2 PB Reviews – Full review, i.e. filling in the reviewing form PB review is visible to PC and might lead to more online discussion – If there is disagreement between the consensus or if there was no consensus before another brief online discussion is recommended PB Meeting – PB member summarizes the paper and the reviews/discussion, 2 nd Reader comments briefly and the complete PB discusses whether the paper is above the bar for the meeting – Edit the final summary that will be included in the feedback to the authors

Program Committee Actions Bidding – This is important and without it you might get things you don’t like or can’t even review properly Reviewing – Intermediate deadline when 1/2 the papers will be done This feature saves everyone pain Online Discussions – Mandatory Participation – Check rebuttals and amend reviews where required – Check that PB member summaries are accurate – Be available for last minute discussions early December

Paper Flow Desk Reject Online Discussion PC Reviewing PB Reviewing PB Meeting per PC (PB) per PB ??

PB Meeting Raleigh, North Carolina 9 and 10 December 2015 – 2 full days Sheraton Downtown Raleigh Raleigh-Durham Airport (RDU) – Direct flights from all major US hubs – London