© J. Christopher Beck 2007 Commentary on Session A J. Christopher Beck Department of Mechanical & Industrial Engineering University of Toronto, Canada Scheduling a Scheduling Competition, Providence, Sept 2007

2 © J. Christopher Beck Papers Ghersi et al. focuses on an operational question –how do we judge the results? –important, technical issue –valuable for whoever runs the competition Other 3 papers focus on a more strategic question –what problem types should the competition address?

3 © J. Christopher Beck 2007 Some Operational Issues Automated verification of results Entries need to run on the same platform Not just source code vs. binary –License issues?

4 © J. Christopher Beck 2007 Integration vs. Focus Rich problems (Le Pape; Guerri et al.) vs. a single fundamental issue (Cicirello) Things to consider: –industry impact –potential for research progress/breakthroughs will we understand the results? –barriers to entry easier to participate in Cicirello-style track –is this an OR vs. AI issue?

5 © J. Christopher Beck 2007 Multiple Tracks? Perhaps of increasing difficulty Things to consider: –spreading the competition too much one entry per track is not very interesting –“granularity” of tracks –creating a challenge –barriers to entry

6 © J. Christopher Beck 2007 Robustness as a Criteria Bias evaluation toward good performance on all instances (Le Pape) Best all-round single machine scheduler across different opt. funcs (Cicirello) Things to consider: –“jack of all trades, master of none” give up on being “the” best on a given problem marketing –industry vs. research –another OR vs. AI issue?

7 © J. Christopher Beck 2007 Competition or “Challenge” Competition –like SAT or Planning competition –multiple tracks Challenge –one problem type (e.g., one of Guerri et al.’s, one of Le Pape’s) –long work horizon (e.g., 6 months – 1 year) –like the CP Modeling Challenge (2005)

8 © J. Christopher Beck 2007 Competition or “Challenge” Things to consider: –marketing –(end user) industry interest and commitment –barriers to entry –potential for lack of community interest –organizational overhead