1. Rachel T. Johnson Douglas C. Montgomery Bradley Jones
An Empirical Study of the Prediction Performance of Space-filling Designs
Rachel T. Johnson
Douglas C. Montgomery
Bradley Jones

2. Widely used in engineering design Use continues to grow
Computer Models
Widely used in engineering design
Use continues to grow
May have lots of variables, many responses
Can have long run times
Complex output results
Need efficient methods for designing the experiment and analyzing the results
Johnson QPRC 2009

3. Computer Simulation Types
Computer Simulation Models
Stochastic Simulation Models
Deterministic Simulation Models
Discrete Event Simulation
(DES)
Computational Fluid Dynamics
(CFD)
Johnson QPRC 2009

4. Space-filling designs compared
Comparing Designs
Space-filling designs compared
Sphere packing (SP)
Latin Hypercube (LH)
Uniform (U)
Gaussian Process Integrated Mean Square Error (GP IMSE)
Assumed surrogate model
Gaussian Process model
Comparisons based on prediction
Johnson QPRC 2009

5. Designs Sphere Packing: Johnson et al. (1990) Latin Hypercube:
Uniform
Sphere Packing:
Johnson et al. (1990)
Maximizes the minimum distance between pairs of design points
Latin Hypercube:
McKay et al. (1979)
A random n x s matrix, in which columns are a random permutation of {1, . . ., n}
Latin Hypercube
Uniform:
Feng (1980)
A set of n points uniformly scattered within the design space
Maximum Entropy
GASP IMSE
Maximum Entropy:
Shewry and Wynn (1987)
Maximizes the information contained in the distribution of a data set
GP IMSE:
Sacks et al. (1989)
Minimizes the integrated mean square error of the Gaussian Process model
I - Optimal:
Box and Draper (1963)
Minimizes the average prediction variance (of a linear regression model) over a design region
I - Optimal
Johnson QPRC 2009

6. Model Fitting Techniques
Gaussian Process Model
Assumes a normal distribution
Interpolator based on correlation between points
Prediction variance calculated as
Johnson QPRC 2009

7. Evaluation of Designs for the GP Model
Recall the prediction variance:
Prediction variance dependent on:
Design
Value of unknown θ
Sample size
Dimension of x
Design of Experiments can be used to evaluate the designs
Johnson QPRC 2009

8. Example FDS Plots Maximum Entropy Sphere Packing Uniform
Latin Hypercube
GP IMSE
Johnson QPRC 2009

9. Research Questions Does it matter in practice what design you choose?
Is there a dominating experimental design that performs better in terms of model fitting and prediction?
What is the role of sample size in experimental designs used to fit the GASP model?
At what point does prediction error variance and other measures of prediction performance become reasonably small with respect to N, the sample size chosen?
Johnson QPRC 2009

10. Used test functions to act as surrogates to simulation code
Empirical Comparison
Used test functions to act as surrogates to simulation code
Evaluated designs based on RMSE and AAPE
Interested in effect of:
Design type
Sample size
Dimension of design
*Gaussian Correlation Function
Johnson QPRC 2009

11. Comparison Procedure Step 1: Choose a test function
Step 2: Choose a sample size and space-filling design
Step 3: Create the design with the number of factors equal to the number in the test function chosen in step 1) and the specifications set in step 2)
Step 4: Using the test function in 1) find the values that correspond to each row in the design
Step 5: Fit the GASP model
Step 6: Generate a set of 40,000 uniformly random selected points in the design space and compare the predicted value (generated by the fitted GASP model) to the actual value (generated by the test function) at these points. Compute the Root mean square error (RMSE).
Johnson QPRC 2009

12. Test Functions Test Function #1 Test Function #2 Test Function #4
Johnson QPRC 2009

13. Test Function #1 Results
Johnson QPRC 2009

14. Test Function #2 Results
Johnson QPRC 2009

15. Test Function #3 Results
Johnson QPRC 2009

16. Test Function #2 Results
Johnson QPRC 2009

17. ANOVA Analysis
Johnson QPRC 2009

18. Jackknife Plots Test Function #2 – LHD with a sample size of 20
Johnson QPRC 2009

19. Conclusions No one design type is better than the others
Increasing sample size decreases RMSE
There is a strong interaction between sample size and test function type – the more complex the function the more runs required
Jackknife plot is an excellent indicator of a “good fit”
Johnson QPRC 2009

20. References
Fang, K.T. (1980). “The Uniform Design: Application of Number-Theoretic Methods in Experimental Design,” Acta Math. Appl. Sinica. 3, pp. 363 – 372.
Johnson, M.E., Moore, L.M. and Ylvisaker, D. (1990). “Minimax and maxmin distance design,” Journal for Statistical Planning and Inference 26, pp. 131 – 148.
McKay, N. D., Conover, W. J., Beckman, R. J. (1979). “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code,” Technometrics 21, pp. 239 – 245.
Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and Analysis of Computer Experiments,” Statistical Science 4(4), pp. 409 – 423.
Shewry, M.C. and Wynn, H.P. (1987). “Maximum entropy sampling,” Journal of Applied Statistics 14, pp. 898 – 914.
Johnson QPRC 2009

21. Correlation Function Gaussian Correlation Function
Cubic Correlation Function
Johnson QPRC 2009