1. ICSE 2016 Reviewing Process Laurie and Willem

2. Welcome Thanks for serving! 28 Program Board members – In charge of +-22 papers – Full review for +-6 of those 89 Program Committee members – +-16 papers to review

3. General Philosophy Program Committee does the main reviewing Program Board keeps process on track and leads the discussions PB then communicates PC decisions at the face- to-face meeting – Extra PB member does a checks&balances review Main purpose of the meeting is to determine where the bar is for acceptance – For some problem cases that had no consensus during the PC reviewing period, decisions will be made Chairs sit back and relax!

4. Author view All papers get at least 3 reviews – Some will get 4 (extra from PB member) All papers will get a chance at a rebuttal – Right after PC reviewing, i.e. will come about 3 days after the start of online discussion All papers will get a summary of the decision Author feedback will be solicited

5. Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Overall Process

6. Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Program Board Involvement

7. Desk Rejects 28/8 1/9 Bidding 6/9 Assignments Rebuttals 23/10 (3 days) Online Discussion 23/10 PC Reviewing 11/9 PB Overseeing 20/11 2 nd Reader Assignment 24/11-6/12 PB Review 9-10/12 PB Meeting 15/12 Notifications Program Committee Involvement Online Discussion

8. Program Board Actions Overview Bidding Monitor Reviewing [Chairs will as well] – Make sure reviews are on time Drive online discussion PB Reviews – Full review, i.e. filling in the reviewing form PB Meeting

9. Program Board Actions Online Discussion Drive online discussion – Summarize views (including rebuttal) and kick off the conversation – Eliminate clear rejects first No support is a reject – Try to find consensus No consensus is fine at this point – Consensus is based on all 3 reviewers and the PB member’s opinion i.e. PB member is allowed an opinion, but it doesn’t carry more weight than the PC Write a discussion summary at the end of the discussion period – Select one of two outcomes: reject or move on Either consensus accept or no consensus moves on – Rejection summaries will be included in author feedback Summaries of papers moving on will be superseded by edited summaries later in the process

10. Program Board Actions Phase 2 PB Reviews – Full review, i.e. filling in the reviewing form PB review is visible to PC and might lead to more online discussion – If there is disagreement between the consensus or if there was no consensus before another brief online discussion is recommended PB Meeting – PB member summarizes the paper and the reviews/discussion, 2 nd Reader comments briefly and the complete PB discusses whether the paper is above the bar for the meeting – Edit the final summary that will be included in the feedback to the authors

11. Program Committee Actions Bidding – This is important and without it you might get things you don’t like or can’t even review properly Reviewing – Intermediate deadline when 1/2 the papers will be done This feature saves everyone pain Online Discussions – Mandatory Participation – Check rebuttals and amend reviews where required – Check that PB member summaries are accurate – Be available for last minute discussions early December

12. Paper Flow Desk Reject Online Discussion PC Reviewing PB Reviewing PB Meeting 500 30 470 16 per PC (PB) 300 170 6 per PB 2030 120 60??

13. PB Meeting Raleigh, North Carolina 9 and 10 December 2015 – 2 full days Sheraton Downtown Raleigh Raleigh-Durham Airport (RDU) – Direct flights from all major US hubs – London