1. T. Messelis, S. Haspeslagh, P. De Causmaecker B. Bilgin, G. Vanden Berghe

2.  Introduction  Method  Our work  Conclusions  Future work T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 2

3.  predict performance ◦ of one or more algorithms ◦ on a specific problem instance  to be able to ◦ know in advance how good an algorithm will do ◦ choose the ‘best’ algorithm out of a portfolio ◦ choose the ‘best’ parameter setting T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 3

4.  Build empirical hardness models ◦ empirical: performance of some algorithm ◦ hardness: measured by some performance criteria  time spent by an algorithm searching for a solution  quality of an (optimal) solution  gap between found and optimal solution  Model hardness as a function of features ◦ computationally inexpensive ‘properties’  e.g. clauses-to-variables ratio (SAT)  e.g. maximum consecutive working days (NRP) T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 4

5.  Introduced by K. Leyton-Brown et al. 1.Select problem instance distribution 2.Select one or more algorithms 3.Create a set of features 4.Generate an instance set, calculate features and determine the algorithm performances 5.Eliminate redundant or uninformative features 6.Use machine learning techniques to select functions of the features that approximate the algorithm’s performances K. Leyton-Brown, E. Nudelman, Y. Shoham. Learning the empirical hardness of optimisation problems: The case of combinatorial auctions. In LNCS, 2002 T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 5

6.  This strategy has been successful in different areas: ◦ combinatorial auction: winner determination problem ◦ uniform random 3-SAT  accurate algorithm performance prediction  algorithm portfolio approach (SATzilla)  won several gold medals in SAT competitions  Apply it to Nurse Rostering! T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 6

7.  problem of assigning nurses to shifts, given a set of hard and soft constraints  Performance: ◦ time spent by a complete search algorithm to find the optimal roster ◦ quality of this optimal roster ◦ quality of a roster obtained by a heuristic algorithm, ran for some fixed period of time ◦ quality gap between both solutions T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 7

8. Translate NRP instances to SAT instances and use existing SAT features to build models  translation based on numberings  solve instances to optimum using CPLEX  run a metaheuristic for 10 seconds T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 8

9.  Regression results on predicting CPLEX objective T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 9 Regression Statistics Multiple R0,994111737 R Square0,988258146 Adjusted R Square0,986086846 Standard Error3,055746115 Observations500 ANOVAdfSSMSFSignificance F Regression7387449,570955349,9387 5927,65075 20 Residual4934603,4290699,337584318 Total500392053 CoefficientsStandard Errort StatP-valueLower 95%Upper 95% Intercept0 ----- VCG CN variation549,453243726,5599116120,687314466,6305E-69497,2686628601,6378245 VCG CN max16,144934110,51068830931,614066421,1547E-12015,1415401417,14832809 VCG VN min-27,683744040,867307099 - 31,919194584,7809E-122 - 29,38781814 - 25,97966995 CG mean-126,06510553,028079093 - 41,632038521,6608E-163 - 132,0146372 - 120,1155737 BAL PL/C variation14819,45307331,758279844,669429441,9828E-17514167,6185715471,28757 BAL 1-20743,13734555,6276197 - 37,332804578,8022E-146 - 21834,82751 - 19651,44717 BAL 3-1719,07290546,94112572 - 36,621893469,3813E-143-1811,30224 - 1626,843571

10. Come up with a feature set specifically for NRP and build models from these features  on the same set of NRP instances  for the same performance indicators T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 10

11.  very simple set of features: ◦ some problem parameters  max & min number of assignments  max & min number of consecutive working days  max & min number of consecutive free days ◦ and ratio’s of those parameters  max cons working days / min cons working days  max num assignments / min cons working days  availability / coverage requirements (tightness) ... T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 11

12.  Regression results on predicting CPLEX objective T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 12 Regression Statistics Multiple R0,970029662 R Square0,940957544 Adjusted R Square0,940117509 Standard Error2,589352208 Observations500 ANOVAdfSSMSFSignificance F Regression752571,815537510,259362 1120,14096 41,2709E-297 Residual4923298,7344696,704744855 Total49955870,55 CoefficientsStandard Errort StatP-valueLower 95%Upper 95% Intercept66,154822661,5519558542,626742682,6721E-16763,1055440469,20410128 max num assignments (6 - 10) - 6,5221495680,134955581 - 48,328120425,9072E-189 - 6,787309925 - 6,256989211 min cons working days (2 - 6)2,7839805390,24971756111,148517286,86075E-262,2933361573,274624921 max cons working days (3 - 8) - 2,6165552520,078539247 - 33,315257853,2938E-128 - 2,770868949 - 2,462241554 min cons free days (1 - 3)7,9715202130,65876745412,100658841,09346E-296,6771757179,26586471 max cons free days (2 - 3) - 2,1533266360,589449923 - 3,653112085 0,00028689 1 - 3,311476237 - 0,995177036 max num assignments / min cons work days1,5890556550,3374487614,7090279683,24188E-060,9260372492,252074061 max cons FD / min cons FD2,6464762380,6388655234,1424621334,0444E-051,3912350013,901717476

13.  We can build accurate models to predict algorithm performance, based on very basic properties of NRP instances. ◦ objective values  models for the objective values of both CPLEX and the metaheuristic are fairly accurate ◦ gap  less accurate predictions, however with a standard error of 0.95 ◦ CPLEX running time  not very accurate, due to very high variability in the running time T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 13

14.  building models on a larger scale ◦ now only a very limited dataset  more (sophisticated) features for NRP instances ◦ now only a very basic set with some aggregate functions of it  combining both SAT features and NRP features T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 14

15. T. Messelis, S. Haspeslagh, P. De Causmaecker, B. Bilgin, G. Vanden Berghe 15