1. A Primer on Running Deterministic Experiments
Bradley Jones
Principal Research Fellow
JMP Division/SAS

2. Plan for this talk
Introduce an internet-based simulator of a garden sprinkler.
Investigate and compare several strategies for experimental design and analysis using the simulator.

3. Computer Experiments vs. Physical Experiments
Fast
Cheap
Predictive of Real World?
Low Accuracy Experiments LE
Time to build prototypes
Cost of prototypes
Empirical Validity
High Accuracy Experiments HE

4. Current Applications (partial list)
Circuit Simulation (SPICE)
Stress Analysis (FEA)
Hurricane Tracking
Weather Prediction
Fluid Flow (FLUENT)

5. How are computer experiments different?
No random error!
So replication and randomization are useless.
More complicated functions
So low-order polynomial approximations may be inadequate.
Statistical designs based on minimizing variance may not perform well based on their incorrect assumption.

6. Why design computer experiments?
Individual runs may be expensive. (hours, days)
They may depend on many variables.
Need to develop a surrogate for the computer simulation model.

7. What do you want in a surrogate model?
Adequate approximation to the computer simulation.
Fast predictions for new points.

8. Standard Surrogate - Gaussian Process (GP)
Pros of GP Models
Flexibility
Parsimonious in the number of unknowns
No error for input settings where you have observed responses (i.e. GP models interpolate the data.)

9. Cons of GP models
Estimating the GP parameters has technical challenges.
Problems inverting near singular matrices
Model fitting is slow compared to polynomial models.
Interpolating the data does not mean perfect prediction for other factor combinations!
Model is based on random functions but the modeled function is not random.

10. Alternative Surrogate Models
Polynomial Regression
Neural Nets
Both using cross validation to avoid over-fitting.

11. Internet simulation tool

12. Sprinkler Simulation

13. Design Strategies for Comparison
Maximin Latin Hypercube Design
Covering Array
Multi-level Orthogonal Design
Fast Flexible Filling (FFF) Design

14. Maximin Latin Hypercube Design (Mm LHD)
Construction of a random LHD with n runs. Each column is a random permutation of 1 to n Center and scale to match the desired low and high values Construction of an Mm LHD. Choose the column permutations so that the minimum distance between any two points in the design is maximized (i.e. spread the points apart).

15. Mm LHD with two factors and 11 runs

16. Properties of the Mm LHD
Design points are equally spaced for every factor.
No replication in projection.
Design points are spread apart from each other more than for any other LHD

17. Covering Array
New in JMP 12 Pro

18. Multi-level orthogonal design
New in JMP 11 in the screening design tool

19. Fast Flexible Filling Designs
New in JMP 11 in the space filling design tool Improved in JMP 12 with MaxPro optimization criterion

20. Fast Flexible Filling Design – MaxPro Criterion
n is the number of runs and p is the number of factors
Note that when xil– xjl gets small, m gets big.
The goal of the design is to make m as small as possible.

21. FFF Design – Predicted Optimal Settings

22. Demonstrations using JMP
Showing how to
use JMP to create designs
use the garden sprinkler simulator to generate responses
use JMP to analyze the data and optimize operating settings