Using Physical and Computer Based Models to Teach Design of Experiments Sam Gardner, PStat®, Senior Research Scientist Elanco Abstract Teaching Approach Unguided Experimentation Teaching Design of Experiments (DOE) to scientists and engineers has two crucial elements: knowledge and motivation Using models (physical or simulated) to illustrate the application of DOE to understanding and optimizing a system provides the opportunity to cover both of these elements effectively An overview of several models and how they can be used along with JMP to teach DOE are presented The key aspect of this approach is how it motivates the student to want to use DOE in their research Begin with an unguided experimentation exercise that involves the entire class Divide the class into 4 groups Briefly describe the model that is going to be used, the goals for the experiment, and demonstrate a single run of the model Let each group, in succession, choose a setting for the experimental factors and then perform the experimental run, recording the factor levels and the measured response(s) Conduct 4 rounds of experiments for each of the 4 groups (for a total of 16 experiments) At the end of the experiments, use JMP to interactively explore the experimental space and factor/response relationships The following teaching agenda is used for a 4-8 hour course with up to 30 students. JMP is utilized through to illustrate concepts and to show how to analyze experimental data. Team based, unguided experimentation with the physical/simulated model Illustrates (usually) the lamppost effect Introduce 2 level designs for screening Factorial Fractional Factorial Explaining concepts in design analysis and evaluation Replication Confounding Lack of fit / curvature Introduce Response Surface and Definitive Screening Designs Conclude with planning, executing, and analyzing a RSM or DS design on the physical/simulated model

A Selection of Models that are Useful for Teaching DOE Desired Features of a Simulation Model Ease of Use Size – amount of space needed to conduct experiments Speed – amount of time it takes to setup and execute each experimental run Realistic, understandable, engaging Must have interactions and curvature Must have random error in the response (not completely deterministic) References simulation models Piston Simulator Kennett, Zacks, Modern Industrial Statistics, John Wiley & Sons http://bit.ly/20sIKjc Virtual Trebuchet (www.virtualtrebuchet.com) Paper Helicopter (www.paperhelicopter.com)

Using The Virtual Trebuchet Model Usage Notes The JMP script is used to facilitate executing experimental runs with the Virtual Trebuchet Selecting “Simulate” creates a URL and opens the web browser with the specified trebuchet settings Wind speed is set at random, exploiting this as a nuisance variable, inducing random variation in the response After running the trebuchet launch simulation in the browser, the distance is entered into the JMP form and submitted to the data table A tabulation of the experimental results is appended to the report window Using the script increases the speed of executing the experimental runs Script available on request, email author at sjgardner@elanco.com

Why Start with Unguided Experimentation? Actual In-Class Results The Lamp Post Effect Late at night, a police officer finds a drunk man crawling around on his hands and knees under a lamp post. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street. “Then why are you looking over here?”, the befuddled officer asks. “Because the light is better here”, explains the drunk man. It (almost) always highlights the perils and pitfalls of not following a structured experimental plan Typically factors are changed “1-factor at a time” Once a “good” setting has been found, participants rarely look at factor combinations that are dramatically different from the previous experiments (the Lamp Post Effect) The emotional aspect of experimentation are also highlighted No one wants a “failing” result or a worse outcome than a previous result Not understanding a system leads to hesitancy to explore and anxiety The inefficiency of unstructured experimentation can also be highlighted How much of the experimental space was explored? How many times were different factor settings replicated? Was the goal achieved? Final Design has moderate to high confounding, poor efficiency, and poor power Actual In-Class Results Very Little Variation in Factor Settings from Run-to-Run

Finish with a JMP Experimental Design! Augmented Design RSM Model Fit Results Shows the efficiency and clarity that can be achieved using DOE Contrast with the “natural” unstructured approach that most experimenters take Increases the desire and motivation to use DOE in scientific work Teach the concepts in Experimental Design, and show how to use JMP to create and analyze designs Guide the class to determine factors and levels and statistical design Repeat the experimentation with new design Screening Design Initial Model Augmented Design RSM Model Fit Model shows Lack-of-fit/curvature