Two Issues Concerning Research Conferences Dave Patterson October 2004

1) Big Idea/New Directions Papers Problem Trying to Solve: Given avalanche of submissions to conferences, Big Idea / New Directions papers rarely accepted at many conferences Program Committee (PC) must reject papers, easy to find bullet holes in such papers, they get a few low scores, and PC rejects them Occasionally hero PC member gets 1 thru, but not sustainable (not just call for papers, PC chair) Numbers making many conferences conservative Given important stature of conferences today, Big Ideas / New Directions papers even more important now

1) Big Idea/New Directions Papers Proposal: Try a 3 – 4 year experiment to attract such papers at some conferences Mechanism: Separate Big Idea / New Directions PC Former PC chairs + people with reputation for Big Idea / New Directions select ~ 3 papers Dedicate 1 session to such papers Evaluate value of Big Idea papers to decide if discard/keep/expand this approach: attendance at session, good discussions at conference, lead to subsequent papers, …

1) Big Idea/New Directions Papers Our conference already accepts them Great! Please share your experience, but no need to try this idea. But we lose slots for 3 good papers? For 3 years, lets drop 1 panel session or 1 keynote (often nil in my experience) Discuss! Suggestions for conferences and/or people to contact as likely match?

2) Is large scale a threat to conference quality? Background – Number of paper submissions increasing over time while number of papers presented roughly constant

2) Is large scale a threat to conference quality? Papers Submitted

2) Is large scale a threat to conference quality? Acceptance Rate

2) Is large scale a threat to conference quality? Impact on Program Committee – Either more papers per member or larger PC Either affects quality of reviews and of decision? Impact on Authors of Rejected Papers – Inaccurate reviews decision feels arbitrary, unfair Resubmit them to related conferences or submit more papers per conference more papers per conference Impact on Conference,Field – Losing good ideas? Are papers ranked really that bad? (Its research) Researchers dont try riskier ideas since less chance?

2) Is large scale a threat to conference quality? Proposal – Create an ad hoc committee to write report (& talk in Feb) on whether / when a problem including: Why are numbers increasing? (More people in field, resubmiting, multiple papers/conf) What size is a problem? (150? 250? 350?) What have some conferences tried, and did it work or not? (best/worst practices) New good ideas to help to cope with issue

2) Is large scale a threat to conference quality? Examples of Best/Worst Practice STOC – Went to dual tracks PODC – 5 minute Brief Announcements (1 page in proceedings) + 25-min papers 75 talks in 3 days, 50% - 50%, single track SIGMOD – lower bound to accept rate, so more papers submitted more presented SIGGRAPH – add 5-minute recent result papers with later deadline, quick review DAC? – Hierarchical Program Committees (divide by topic, allocate by track)

2) Is large scale a threat to conference quality? Examples of New Ideas No downside to submitting, so create downside by publishing authors batting averages to Limit number of papers one can submit to a single conference (as often a limit for PC members) Since now many related conferences, if some submit papers to each, go to a common paper deadline to force authors to decide target Journals cope with surge by having editors reject papers outright without further review, for otherwise surge would break review system: give PC chair same authority smaller PC

2) Is large scale a threat to conference quality? Request – If you think its a good idea, suggest names of people who might be willing to volunteer for ad hoc committee, possibly a chair? Important to get people from diverse communities to collect practices, ideas