A Primer on Running Deterministic Experiments Bradley Jones Principal Research Fellow JMP Division/SAS

Plan for this talk Introduce an internet-based simulator of a garden sprinkler. Investigate and compare several strategies for experimental design and analysis using the simulator.

Computer Experiments vs. Physical Experiments Fast Cheap Predictive of Real World? Low Accuracy Experiments LE Time to build prototypes Cost of prototypes Empirical Validity High Accuracy Experiments HE

Current Applications (partial list) Circuit Simulation (SPICE) Stress Analysis (FEA) Hurricane Tracking Weather Prediction Fluid Flow (FLUENT)

How are computer experiments different? No random error! So replication and randomization are useless. More complicated functions So low-order polynomial approximations may be inadequate. Statistical designs based on minimizing variance may not perform well based on their incorrect assumption.

Why design computer experiments? Individual runs may be expensive. (hours, days) They may depend on many variables. Need to develop a surrogate for the computer simulation model.

What do you want in a surrogate model? Adequate approximation to the computer simulation. Fast predictions for new points.

Standard Surrogate - Gaussian Process (GP) Pros of GP Models Flexibility Parsimonious in the number of unknowns No error for input settings where you have observed responses (i.e. GP models interpolate the data.)

Cons of GP models Estimating the GP parameters has technical challenges. Problems inverting near singular matrices Model fitting is slow compared to polynomial models. Interpolating the data does not mean perfect prediction for other factor combinations! Model is based on random functions but the modeled function is not random.

Alternative Surrogate Models Polynomial Regression Neural Nets Both using cross validation to avoid over-fitting.

Internet simulation tool https://perswww.kuleuven.be/~u0059569/doe/sprinkler/default.php

Sprinkler Simulation

Design Strategies for Comparison Maximin Latin Hypercube Design Covering Array Multi-level Orthogonal Design Fast Flexible Filling (FFF) Design

Maximin Latin Hypercube Design (Mm LHD) Construction of a random LHD with n runs. Each column is a random permutation of 1 to n Center and scale to match the desired low and high values Construction of an Mm LHD. Choose the column permutations so that the minimum distance between any two points in the design is maximized (i.e. spread the points apart).

Mm LHD with two factors and 11 runs

Properties of the Mm LHD Design points are equally spaced for every factor. No replication in projection. Design points are spread apart from each other more than for any other LHD

Covering Array New in JMP 12 Pro

Multi-level orthogonal design New in JMP 11 in the screening design tool

Fast Flexible Filling Designs New in JMP 11 in the space filling design tool Improved in JMP 12 with MaxPro optimization criterion

Fast Flexible Filling Design – MaxPro Criterion n is the number of runs and p is the number of factors Note that when xil– xjl gets small, m gets big. The goal of the design is to make m as small as possible.

FFF Design – Predicted Optimal Settings

Demonstrations using JMP Showing how to use JMP to create designs use the garden sprinkler simulator to generate responses use JMP to analyze the data and optimize operating settings