1. Using Physical and Computer Based Models to Teach Design of Experiments
Sam Gardner, PStat®, Senior Research Scientist
Elanco
Abstract
Teaching Approach
Unguided Experimentation
Teaching Design of Experiments (DOE) to scientists and engineers has two crucial elements: knowledge and motivation
Using models (physical or simulated) to illustrate the application of DOE to understanding and optimizing a system provides the opportunity to cover both of these elements effectively
An overview of several models and how they can be used along with JMP to teach DOE are presented
The key aspect of this approach is how it motivates the student to want to use DOE in their research
Begin with an unguided experimentation exercise that involves the entire class
Divide the class into 4 groups
Briefly describe the model that is going to be used, the goals for the experiment, and demonstrate a single run of the model
Let each group, in succession, choose a setting for the experimental factors and then perform the experimental run, recording the factor levels and the measured response(s)
Conduct 4 rounds of experiments for each of the 4 groups (for a total of 16 experiments)
At the end of the experiments, use JMP to interactively explore the experimental space and factor/response relationships
The following teaching agenda is used for a 4-8 hour course with up to 30 students.
JMP is utilized through to illustrate concepts and to show how to analyze experimental data.
Team based, unguided experimentation with the physical/simulated model
Illustrates (usually) the lamppost effect
Introduce 2 level designs for screening
Factorial
Fractional Factorial
Explaining concepts in design analysis and evaluation
Replication
Confounding
Lack of fit / curvature
Introduce Response Surface and Definitive Screening Designs
Conclude with planning, executing, and analyzing a RSM or DS design on the physical/simulated model

2. A Selection of Models that are Useful for Teaching DOE
Desired Features of a Simulation Model
Ease of Use
Size – amount of space needed to conduct experiments
Speed – amount of time it takes to setup and execute each experimental run
Realistic, understandable, engaging
Must have interactions and curvature
Must have random error in the response (not completely deterministic)
References simulation models
Piston Simulator
Kennett, Zacks, Modern Industrial Statistics, John Wiley & Sons
Virtual Trebuchet (
Paper Helicopter (

3. Using The Virtual Trebuchet Model
Usage Notes
The JMP script is used to facilitate executing experimental runs with the Virtual Trebuchet
Selecting “Simulate” creates a URL and opens the web browser with the specified trebuchet settings
Wind speed is set at random, exploiting this as a nuisance variable, inducing random variation in the response
After running the trebuchet launch simulation in the browser, the distance is entered into the JMP form and submitted to the data table
A tabulation of the experimental results is appended to the report window
Using the script increases the speed of executing the experimental runs
Script available on request, author at

4. Why Start with Unguided Experimentation? Actual In-Class Results
The Lamp Post Effect
Late at night, a police officer finds a drunk man crawling around on his hands and knees under a lamp post. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street.
“Then why are you looking over here?”, the befuddled officer asks.
“Because the light is better here”, explains the drunk man.
It (almost) always highlights the perils and pitfalls of not following a structured experimental plan
Typically factors are changed “1-factor at a time”
Once a “good” setting has been found, participants rarely look at factor combinations that are dramatically different from the previous experiments (the Lamp Post Effect)
The emotional aspect of experimentation are also highlighted
No one wants a “failing” result or a worse outcome than a previous result
Not understanding a system leads to hesitancy to explore and anxiety
The inefficiency of unstructured experimentation can also be highlighted
How much of the experimental space was explored?
How many times were different factor settings replicated?
Was the goal achieved?
Final Design has moderate to high confounding, poor efficiency, and poor power
Actual In-Class Results
Very Little Variation in Factor Settings from Run-to-Run

5. Finish with a JMP Experimental Design! Augmented Design RSM Model Fit
Results
Shows the efficiency and clarity that can be achieved using DOE
Contrast with the “natural” unstructured approach that most experimenters take
Increases the desire and motivation to use DOE in scientific work
Teach the concepts in Experimental Design, and show how to use JMP to create and analyze designs
Guide the class to determine factors and levels and statistical design
Repeat the experimentation with new design
Screening Design
Initial Model
Augmented Design RSM Model Fit
Model shows Lack-of-fit/curvature