© 2006 ACM/IEEEHealth of Conferences Committee Health of Conferences Committee Update February 10, SIG Governing Board Meeting Mark D. Hill (Committee Chair) U. WisconsinACM/SIGARCH & IEEE Jean-Luc GaudiotUC IrvineIEEE Mary HallUSC/ISIACM/SIGPLAN Joe MarksMitsubishi Electric Research Labs ACM/SIGGRAPH Paolo PrinettoPolitecnico di TorinoIEEE Donna BaglioACM HeadquartersACM

Health of Conferences Committee © 2006 ACM/IEEE 2 Executive Summary Working for six weeks to find some best practices Five questions yielded responses from 30 SIGS –Reviewer load –Non-incremental –Program committees –Workshops, etc. –Catch all Have some first-impression results Want feedback or

Health of Conferences Committee © 2006 ACM/IEEE 3 Talk Outline Set Up –Charge –Process –Response Sources so Far –Caveats Questions & Example Responses Appendix: Full text of questions

Health of Conferences Committee © 2006 ACM/IEEE 4 Charge from ACM President David Patterson The idea is to collect the best practices onto a web page so that conference organizers can see innovative ways to cope with the demands of paper submissions, refereeing, and presentations as the number of papers increase. The hope is that organizers will either try good new ideas or at least avoid the mistakes of others.

Health of Conferences Committee © 2006 ACM/IEEE 5 Process Dec committee formed; Baglio & Hill start up Jan. 5 call to divide responsibly & set questions –Marks – ACM SIGs with BIG conferences (> 1K attendees) –Hall – ACM SIGs with MEDIUM –Baglio – ACM SIG with SMALL (< 100) –Gaudiot/Prinetto – IEEE in manner to be done Iterated on questions; sent out; got many responses Feb 6 call to check status & Feb 8/9 info to Hill Feb 10 status to SGB (this talk) Feb 27 call to review data on Wiki (we hope)

Health of Conferences Committee © 2006 ACM/IEEE 6 Response Sources so Far ACM Big: –DAC, ICSE, OOPSLA, SIGCSE, SIGGRAPH ACM Medium: –SIGART, SIGARCH, SIGCHI, SIGIR, SIGACT, SIGPLAN, SIGMOD, SIGCOMM, SIGKDD ACM Small: –SIGAda, SIGAPP, SIGBED, SIGACCESS, SIGCSE, SIGDA, SIGDOC, SIGecom, SIGITE, SIGMETRICS, SIGMICRO, SIGMIS, SIGSAC, SIGSAM, SIGSIM, SIGUCCS, SIGWEB IEEE: To be done,

Health of Conferences Committee © 2006 ACM/IEEE 7 Regarding First-Impression Results Goal –Unearth actionable ideas –Reveal failed ideas Non-Goal –Determine summary statistics, because audience interested in SIGs like theirs Caveat –Data arrived to me yesterday & day before

Health of Conferences Committee © 2006 ACM/IEEE 8 Q1: Reviewer Load (example of full text) Has your community recently adopted new practices to deal with growing reviewer load, such as: –tracking reviews of rejected papers from conference to conference as is done in journal reviewing –increasing program committee size –charging a review fee –others? For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

Health of Conferences Committee © 2006 ACM/IEEE 9 Q1: Reviewer Load Tracking reviews? Increasing PC size? Fee? Others? Big (SIGCSE): Maintain reviewer database Medium (SIGART): Senior PC supervises whole PC Small (SIGUCCS): tracking reviews among conferences (like journals)

Health of Conferences Committee © 2006 ACM/IEEE 10 Q2: Non-Incremental Big ideas sessions? More papers? Shorter papers? Deemphasizing detailed evaluation? Others? Big (ICSE): fun flames over beer & snacks Medium (SIGKDD): KDD Cup competition to evaluate alternative approaches to the same problem Small (SIGDOC): Earlier international acceptances to ease visa problems

Health of Conferences Committee © 2006 ACM/IEEE 11 Q3: Program Committees Double blind submissions? Program committee submission restrictions? Rebuttals? Large PCs? Program subcommittees? Others? Big (SIGCSE): Encourage large PCs Medium (SIGCHI): Rebuttals avoid compounding misunderstandings, make authors feel better, & help get reviews on time Small (SIGITE): Single-person PC

Health of Conferences Committee © 2006 ACM/IEEE 12 Q4: Workshops, etc. Workshop co-located at conferences? Stand-alone workshops? Panels? Crazy idea sessions? Big (OOPSLA): 5-minute lightning talks Medium (SIGARCH): Some workshops get special issues in its newsletter Small (SIGAda): Extended abstracts & experience reports

Health of Conferences Committee © 2006 ACM/IEEE 13 Q5: Catch All Are there other approaches your community has tried or abandoned that the rest of us can learn from? Big (SIGGRAPH): Don’t be afraid to try; don’t be afraid to stop Medium (SIGPLAN): Professionally supported PC software is big win Small (SIGMICRO): Need better ways to handle conflicts of interest

Health of Conferences Committee © 2006 ACM/IEEE 14 Executive Summary Working for six weeks to find some best practices Five questions yielded responses from 30 SIGS –Reviewer load –Non-incremental –Program committees –Workshops, etc. –Catch all Have some first-impression results Want feedback or

Health of Conferences Committee © 2006 ACM/IEEE 15 Appendix: Full Text of Questions

Health of Conferences Committee © 2006 ACM/IEEE 16 Q1: Reviewer Load Has your community recently adopted new practices to deal with growing reviewer load, such as: –tracking reviews of rejected papers from conference to conference as is done in journal reviewing –increasing program committee size –charging a review fee –others? For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

Health of Conferences Committee © 2006 ACM/IEEE 17 Q2: Non-Incremental Has your community recently adopted new practices to promote non-incremental new ideas? –big ideas sessions –more papers –shorter papers –deemphasizing detailed evaluation –others? For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

Health of Conferences Committee © 2006 ACM/IEEE 18 Q3: Program Committees Does your community practice: –double blind submissions –program committee submission restrictions –rebuttals (author responses) –large program committees –program subcommittees –others? Do these practices seem to help or hurt promoting your field?

Health of Conferences Committee © 2006 ACM/IEEE 19 Q4: Workshops, etc. Does your community provide venue for work not mature enough for your major conferences, such as: –workshop co-located at conferences –stand-alone workshops –Panels –crazy idea sessions On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination?

Health of Conferences Committee © 2006 ACM/IEEE 20 Q5: Catch All Are there other approaches your community has tried or abandoned that the rest of us can learn from?