Some Notes on the Design and Analysis of Experiments

Formal experiments are … n Cons –extremely expensive (time & money) –usually not representative of the real world (cf. natural observation, field studies, surveys) n Pros –highly controlled –replicable –sometimes the only way to measure small effects or to identify interactions

Designed experiments are used to … n address a research question n to test a hypothesis or a model

Some Definitions: n Independent Variable- the variable which the experimenter has direct control over and is purposely manipulated to test a hypothesis (presence vs. absence, amount, type) n Dependent Variable- what’s being measured

Definitions part 2 n Factor, Treatment- a controlled variable in an experiment (fixed & random) n Level- a particular setting of a factor n Main effect- the effect of a independent variable on experiment n randomize- errr, random?

Definitions part 3 n within subjects experiment - all subjects receive the same treatments n between subjects experiment - subject are randomly divided into groups, and different groups receive different treatments n asymmetrical transfer - when the effect of doing A then B is different then doing B then A

Definitions part 4 n confounding - where the effect of variable has not been separated from the effect of another a variable n control group - a group that does not receive a treatment n factorial design - a designed experiment where two or more independent variables are studied simultaneously

Fractional Factorial Designs n Number of trials gets very large as one increases the number of factors & levels n higher order interactions are actually quite rare n therefore, it makes sense to confound the higher order interactions n example: fractional factorial design

SOURCE: grand mean AA LA N MEAN SD SE SOURCE: AA AA LA N MEAN SD SE SOURCE: LA AA LA N MEAN SD SE

SOURCE: AA LA AA LA N MEAN SD SE

FACTOR : Subject AA LA Res LEVELS : TYPE : RANDOM WITHIN WITHIN DATA SOURCE SS df MS F p ================================================ mean *** S/ AA *** AS/ LA *** LS/ AL ALS/

Interaction An interaction exist when the effect of one variable depends on the level of another variable n Example: 2x2 factorial design has 7 possibilities for significant effects

A nice way to specify a design: “The experiment was a within subjects 5 X 3 X 3 factorial, repeated measures design 10 subjects X 5 limb conditions X 5 limb conditions X 3 target amplitudes X 3 target amplitudes X 3 target widths X 3 target widths X 5 blocks X 5 blocks X 20 trials per amplitude-width condition X = 45,000 total trials”

Some basic rules … n You should always think you know what you’re going to find BEFORE you run the experiment (which doesn’t mean that you are always right, only that you have a hypothesis) n Everything that is tested statistically should also be graphed n If your graphs and your stat analysis don’t CLEARLY agree, something is wrong

Some basic rules part.2 n You should always know exactly how you are going to analyze your data BEFORE you collect it. (the statistical methods) n Remember the difference between statistical significance and the magnitude of the effect