Limitations of ANOVA ©2005 Dr. B. C. Paul

The Data Size Effect We Did ANOVA with one factor We Did ANOVA with one factor We Did it with two factors (Driver and Car) We Did it with two factors (Driver and Car) We could have used the same procedure to do three way (maybe add country or city driving) We could have used the same procedure to do three way (maybe add country or city driving) Or Four Way (maybe add season of the year) Or Four Way (maybe add season of the year) What happens to our test data requirements? What happens to our test data requirements?

Data Set Size Illustration Lets say we tested 4 cars with pairs of before and after MPG Lets say we tested 4 cars with pairs of before and after MPG Minimum we would need 2 drivers on each car Minimum we would need 2 drivers on each car Ie need 8 paired driving tests Ie need 8 paired driving tests Why 8 tests for four cars? Why 8 tests for four cars? Need a way to measure variability not accounted for by the test Need a way to measure variability not accounted for by the test If all differences are accounted for you have no SS error If all differences are accounted for you have no SS error With no denominator you cannot do the F test to decide if your effect is significant With no denominator you cannot do the F test to decide if your effect is significant

The Exponential Explosion Now lets look for a Driver Effect Now lets look for a Driver Effect Lets try 4 drivers Lets try 4 drivers Each driver drives each car twice Each driver drives each car twice 8 pairs of data is now 32 pairs 8 pairs of data is now 32 pairs Now lets look for a Country, Urban Effect Now lets look for a Country, Urban Effect We’ll run the tests in downtown of a large city, in a suburb, and in the country We’ll run the tests in downtown of a large city, in a suburb, and in the country Now we need 96 pairs of data Now we need 96 pairs of data Now lets see if it depends on season Now lets see if it depends on season We’ll do Winter, Spring, Summer, Fall We’ll do Winter, Spring, Summer, Fall Now we need 384 pairs of data Now we need 384 pairs of data

Problem of ANOVA As you get more and more effects to study the amount of data needed grows exponentially As you get more and more effects to study the amount of data needed grows exponentially There are practical limits to how much you can do at once There are practical limits to how much you can do at once There are specialized techniques that can be done There are specialized techniques that can be done We had every driver test every car under every condition. We had every driver test every car under every condition. Eng 540 Design of Experiments does a lot for elegant alternatives Eng 540 Design of Experiments does a lot for elegant alternatives

Relief from SPSS at a Price Our experimental design called for equal numbers of tests under all conditions Our experimental design called for equal numbers of tests under all conditions Actually the procedure I showed with SPSS will run the test without equal numbers of tests under every condition. Actually the procedure I showed with SPSS will run the test without equal numbers of tests under every condition. The Price The Price If I do not check every driver in every car I will loose my ability to measure interaction effects (it will go into the SS error) If I do not check every driver in every car I will loose my ability to measure interaction effects (it will go into the SS error) If I have equal numbers of cases in every cell the test tends to be forgiving (“Robust”) against violations of the normal distribution assumption If I have equal numbers of cases in every cell the test tends to be forgiving (“Robust”) against violations of the normal distribution assumption If my cells are uneven my model will start spitting out more poorly fit answers if I violate the normal distribution assumption. If my cells are uneven my model will start spitting out more poorly fit answers if I violate the normal distribution assumption.

The Who Done It Mystery ANOVA will easily tell you whether an effect exists ANOVA will easily tell you whether an effect exists When it says the driver makes a difference When it says the driver makes a difference Did you have 2 wacko drivers and the other 8 are all the same? Did you have 2 wacko drivers and the other 8 are all the same? It tells you whether an effect exists, but it might still come from only part of the data set It tells you whether an effect exists, but it might still come from only part of the data set

Coping With Who Done It You can run all sorts of plots to see if its just a few results that are different. You can run all sorts of plots to see if its just a few results that are different. You can run statistical tests to test different subsets of the data against each other You can run statistical tests to test different subsets of the data against each other The little options button on the SPSS field where you said Ok leads to a menu of optional tests and plots The little options button on the SPSS field where you said Ok leads to a menu of optional tests and plots Not going to deal with them right now other than telling you they are there. Not going to deal with them right now other than telling you they are there.

Now I Know – What Does It Mean? We found that the MPG improvement from the Red Rooster Carburetor varied with the driver We found that the MPG improvement from the Red Rooster Carburetor varied with the driver We put an ‘individual results may vary” disclaimer on our advertizing We put an ‘individual results may vary” disclaimer on our advertizing Ok, but how much do individual results vary? Ok, but how much do individual results vary? ANOVA doesn’t tell us ANOVA doesn’t tell us For some types of engineering works we have to know how big a difference something will make (Yes-No doesn’t always cut it) For some types of engineering works we have to know how big a difference something will make (Yes-No doesn’t always cut it)

Dealing with large numbers of possible causes May have a large, but more randomly organized set of data and conditions that might have influenced it. May have a large, but more randomly organized set of data and conditions that might have influenced it. Trying to do ANOVA for 15 affect variables would be unwieldy Trying to do ANOVA for 15 affect variables would be unwieldy Solution to the I need to quantify the effect and for maxing out the computer memory (and corporate budget) from doing a 15 way ANOVA is a method called “Regression” Solution to the I need to quantify the effect and for maxing out the computer memory (and corporate budget) from doing a 15 way ANOVA is a method called “Regression” Our next exciting topic!!! Our next exciting topic!!!