On the Potential of Automated Algorithm Configuration Frank Hutter, University of British Columbia, Vancouver, Canada. Motivation for automated tuning Experimental Setup Algorithm design: new application/algorithm: Restart parameter tuning from scratch Waste of time both for researchers and practicioners Algorithm analysis: comparability Is algorithm A faster than algorithm B because they spent more time tuning it ?  Algorithm use in practice: Want to solve MY problems fast, not necessarily the ones the developers used for parameter tuning Automated tuning: related work Best default parameter settings for instance set: Full factorial/fractional factorial experimental design Racing algorithms [ Birattari, Stuetzle, Paquete, and Varrentrapp, GECCO-02] Sequential Parameter Optimization: response surface based [Bartz-Beielstein, 2006] CALIBRA: experimental design & local search [Adenso-Diaz and Laguna, OR Journal 2006] Best parameter setting per instance: Predict runtime with each parameter setting, pick best one [Hutter, Hamadi, Hoos, and Leyton-Brown, CP-06] Best sequence of parameter settings during algorithm run: Reactive search [Battiti and Brunato, 2007] Reinforcement learning [ Lagoudakis and Littman, AAAI ‘00 ] Given: A parametric algorithm A with possible parameter configurations C An instance distribution D, A cost function assigning a cost to each run of A on an instance of D A function combining multiple costs Find: A parameter configuration c  C s.t. algorithm A achieves its lowest overall combined cost on instances from distribution D. Example: (Spear algorithm by Domagoj Babic) New tree search algorithm for SAT Design choices made explicit as parameters 26 parameters, including many categorical and continuous ones SAT-encoded software verification problems Minimize mean runtime across instances Training vs. test instances: Disjoint training and test instances (split 50-50) Want generalization to unseen instances Standard in machine learning Unfortunately not yet standard in optimization Stochastic Optimization Problem: A single instance gives a noisy estimate of a configurations’s cost Reduce noise by performing multiple runs Automated Algorithm Configuration Solvers and benchmark data sets Search in parameter configuration space Evaluation of parameter configuration c Run algorithm with configuration c on a number of instances Comparison of parameter configurations With respect to their performance on the same instances, and with the same random seeds (variance reduction) Conclusions Do not tune your parameter manually anymore Automatic approach is easier, more principled, and yields better results Potentially large potential for improvement over default parameter configuration Large speedups in our domains, even based on simple random sampling Higher potential for higher-dimensional tuning problems: humans cannot grasp high-dimensional parameter interactions well Analysis of tuning scenarios based on random sampling Performance variation across random configurations. Training performances, compared to default. Training performance of random sampling (keeping incumbent). Variations across different orders of configurations. Hardness variation across instances. CDFs for five random configurations and default. Test performance of random sampling (N=100 for training). Variations across different orders of configurations. Solvers: (Dynamic) Local Search for SAT: SAPS 4 continuous parameters Tree Search for SAT: Spear 26 categorical/binary/continuous/integer parameters Branch and Cut for MIP: CPLEX 80 categorical/ordinal/continous/integer parameters Benchmark data sets: SAT-encoded Quasigroup completion SAT-encoded graph colouring based on small-world graphs Winner determination in combinatorial auctions Test performance of random sampling for different sizes of training benchmark set. Percent ‘correct’ pairwise comparisons of parameter configurations, based on varying training cutoff and number of training instances, N Comparison to Iterated Local Search BasicILS and FocusedILS from [Hutter, Hoos, and Stuetzle, AAAI-07]. Both Random and ILS enhanced by novel pruning technique. Future work Continuous parameters (currently discretized) Statistical tests (c.f. racing) Learning approaches (sequential experimental design) Per-instance tuning Automated algorithm design