1. Experiment Basics: Designs
Psych 231: Research Methods in Psychology

2. Announcements Exam 2 a week from today
Review sessions after labs this week
Thursday, DEG 19, 5:30-6:30
Friday, DEG 13, 2:00-3:00
First draft of Class Experiment APA paper due in Labs this week
Piloting our group projects next week, so be prepared to do a complete run through of your group experiment
Announcements

3. Experimental designs Some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

4. 1 factor - 2 levels Good design example
How does anxiety level affect test performance?
Two groups take the same test
Grp1(low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough
Grp2 (moderate anxiety group): 5 min lecture on the importance of good grades for success
What are our IV and DV?
1 Factor (Independent variable), two levels
Basically you want to compare two treatments (conditions)
The statistics are pretty easy, a t-test
1 factor - 2 levels

5. 1 factor - 2 levels Good design example
How does anxiety level affect test performance?
participants
Low
Moderate
Test
Random
Assignment
IV: Anxiety
Dependent Variable
1 factor - 2 levels

6. 1 factor - 2 levels Good design example
How does anxiety level affect test performance?
One factor
Use a t-test to see if these points are statistically different
low
moderate
test performance
anxiety
anxiety
Two levels
low
moderate 60 80
Observed difference between conditions
T-test =
Difference expected by chance
1 factor - 2 levels

7. 1 factor - 2 levels Advantages:
Simple, relatively easy to interpret the results
Is the independent variable worth studying?
If no effect, then usually don’t bother with a more complex design
Sometimes two levels is all you need
One theory predicts one pattern and another predicts a different pattern
1 factor - 2 levels

8. 1 factor - 2 levels Interpolation Extrapolation Disadvantages:
“True” shape of the function is hard to see
Interpolation and Extrapolation are not a good idea
Interpolation
Extrapolation
What happens within of the ranges that you test?
What happens outside of the ranges that you test?
test performance
high
low
moderate
1 factor - 2 levels
anxiety

9. Experimental designs Some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

10. 1 Factor - multilevel experiments
For more complex theories you will typically need more complex designs (more than two levels of one IV)
1 factor - more than two levels
Basically you want to compare more than two conditions
The statistics are a little more difficult, an ANOVA (Analysis of Variance)
1 Factor - multilevel experiments

11. 1 Factor - multilevel experiments
Good design example (similar to earlier ex.)
How does anxiety level affect test performance?
Groups take the same test
Grp1(low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough
Grp2 (moderate anxiety group): 5 min lecture on the importance of good grades for success
Grp3 (high anxiety group): 5 min lecture on how the students must pass this test to pass the course
1 Factor - multilevel experiments

12. 1 factor - 3 levels participants Low Moderate Test Random Assignment
IV: Anxiety
Dependent Variable
High
1 factor - 3 levels

13. 1 Factor - multilevel experiments
low
mod
test performance
anxiety
anxiety
low
mod
high
high 80 60 60
1 Factor - multilevel experiments

14. 1 Factor - multilevel experiments
Advantages
Gives a better picture of the relationship (functions other than just straight lines)
Generally, the more levels you have, the less you have to worry about your range of the independent variable
low
moderate
test performance
anxiety
2 levels
high
low
mod
test performance
anxiety
3 levels
1 Factor - multilevel experiments

15. 1 Factor - multilevel experiments
Disadvantages
Needs more resources (participants and/or stimuli)
Requires more complex statistical analysis (ANOVA [Analysis of Variance] & follow-up pair-wise comparisons)
The ANOVA just tells you that not all of the groups are equal.
If this is your conclusion (you get a “significant ANOVA”) then you should do further tests to see where the differences are
High vs. Low
High vs. Moderate
Low vs. Moderate
1 Factor - multilevel experiments

16. Experimental designs Some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

17. Factorial experiments
Two or more factors
Some vocabulary
Factors - independent variables
Levels - the levels of your independent variables
2 x 4 design means two independent variables, one with 2 levels and one with 4 levels
“Conditions” or “groups” is calculated by multiplying the levels, so a 2x4 design has 8 different conditions A1 A2 B1 B2 B3 B4
Factorial experiments

18. Factorial experiments
Two or more factors
Main effects - the effects of your independent variables ignoring (collapsed across) the other independent variables
Interaction effects - how your independent variables affect each other
Example: 2x2 design, factors A and B
Interaction:
At A1, B1 is bigger than B2
At A2, B1 and B2 don’t differ A A1 A2
Dependent Variable B1 B2
Everyday interaction = “it depends on …”
Factorial experiments

19. Rate how much you would want to see a new movie (1 no interest, 5 high interest):
Hail, Caesar! – new Cohen Brothers movie in 2016
Ask men and women – looking for an effect of gender
Not much of a difference: no effect of gender
Interaction effects

20. Maybe the gender effect depends on whether you know who is in the movie. So you add another factor:
Suppose that George Clooney or Scarlett Johansson might star. You rate the preference if he were to star and if he were not to star.
Effect of gender depends on whether George or Scarlett stars in the movie or not
This is an interaction
Interaction effects
A video lecture from ThePsychFiles.com podcast

21. Factorial designs Consider the results of our class experiment X ✓ ✓
Main effect of cell phone X
Main effect of site type ✓
An Interaction between cell phone and site type ✓
-0.50
0.04
Factorial designs
Report for each main effect and the interaction
Resource: Dr. Kahn's reporting stats page
Means (& SDs) from the table
ANOVA, alpha level 0.05
E.g., “F(1,246) = 52.6, p < .05”

22. Results of a 2x2 factorial design
The complexity & number of outcomes increases:
A = main effect of factor A
B = main effect of factor B
AB = interaction of A and B
With 2 factors there are 8 basic possible patterns of results:
1) No effects at all
2) A only
3) B only
4) AB only
5) A & B
6) A & AB
7) B & AB
8) A & B & AB
Results of a 2x2 factorial design

23. 2 x 2 factorial design Interaction of AB A1 A2 B2 B1 Marginal means
What’s the effect of A at B1?
What’s the effect of A at B2?
Condition
mean
A1B1
Condition
mean
A2B1
Marginal means
B1 mean
B2 mean
A1 mean
A2 mean
Main effect of B
Condition
mean
A1B2
Condition
mean
A2B2
Main effect of A
2 x 2 factorial design

24. Examples of outcomes Main effect of A ✓ Main effect of B
Dependent Variable B1 B2 30 60 45 60 45 30 30 60
Main Effect
of A
At A1: B1 = B2
At A2: B1 = B2
The effect of A doesn’t depend on level of B
Main effect of A ✓
Main effect of B X
Interaction of A x B X
Examples of outcomes

25. Examples of outcomes Main effect of A Main effect of B ✓
Dependent Variable B1 B2 60 60 60 30 30 30 45 45
Main Effect
of A
At A1: B1 - B2 = 30
At A2: B1 - B2 = 30
The effect of A doesn’t depend on level of B
Main effect of A X
Main effect of B ✓
Interaction of A x B X
Examples of outcomes

26. Examples of outcomes Main effect of A Main effect of B
Dependent Variable B1 B2 60 30 45 60 45 30 45 45
Main Effect
of A
At A1: B1 - B2 = +30
At A2: B1 - B2 = -30
The effect of A does depend on level of B
Main effect of A X
Main effect of B X
Interaction of A x B ✓
Examples of outcomes

27. Examples of outcomes Main effect of A ✓ Main effect of B ✓
Dependent Variable B1 B2 30 60 45 30 30 30 30 45
Main Effect
of A
At A1: B1 - B2 = 0
At A2: B1 - B2 = 30
The effect of A doesn’t depend on level of B
Main effect of A ✓
Main effect of B ✓
Interaction of A x B ✓
Examples of outcomes

28. Anxiety and Test Performance
Let’s add another variable: test difficulty.
anxiety
low
mod
high 80 35 50 70 80
main effect
of difficulty
test performance
high
low
mod
anxiety
easy
easy
medium
hard
Test difficulty 80 80 80
medium 65 80
hard 65 80 60
main effect
of anxiety
Yes: effect of anxiety depends on level of test difficulty
Interaction ?
Anxiety and Test Performance

29. Factorial designs Consider the results of our class experiment X ✓ ✓
Main effect of cell phone X
Main effect of site type ✓
An Interaction between cell phone and site type ✓
-0.50
0.04
Factorial designs
Resource: Dr. Kahn's reporting stats page

30. Factorial Designs Advantages Interaction effects
Can’t “see” interaction effects without factorial design
Consider the interaction effects before trying to interpret the main effects
Adding factors decreases the variability
Because you are controlling more of the variables that influence the dependent variable
This increases the statistical Power (your ability to detect an effect) of the statistical tests
Increases generalizability of the results
Because you have a situation closer to the real world (where all sorts of variables are interacting)
Factorial Designs

31. Factorial Designs Disadvantages Action Movie Romantic Comedy
Experiments become very large, and unwieldy
The statistical analyses get much more complex
Interpretation of the results can get hard
In particular for higher-order interactions
Higher-order interactions (when you have more than two interactions, e.g., ABC).
Action Movie
Romantic Comedy
Here there is a three way interaction: gender X who stars X type of movie
Factorial Designs

32. Experimental designs Some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

33. What is the effect of presenting words in color on memory for those words?
Clock
Chair
Cab
So you present lists of words for recall either in color or in black-and-white.
Two different designs to examine this question
Example

34. Between-Groups Factor
2-levels
Each of the participants is in only one level of the IV
levels
Clock
Chair
Cab
Colored
words
participants
Test
Clock
Chair
Cab BW
words

35. Within-Groups Factor levels participants Colored words BW Test Clock
Sometimes called “repeated measures” design
2-levels, All of the participants are in both levels of the IV
levels
participants
Colored
words BW
Test
Clock
Chair
Cab
Clock
Chair
Cab

36. Between vs. Within Subjects Designs
All participants participate in all of the conditions of the experiment.
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

37. Between vs. Within Subjects Designs
All participants participate in all of the conditions of the experiment.
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

38. Between subjects designs
Advantages:
Independence of groups (levels of the IV)
Harder to guess what the experiment is about without experiencing the other levels of IV
Exposure to different levels of the independent variable(s) cannot “contaminate” the dependent variable
Sometimes this is a ‘must,’ because you can’t reverse the effects of prior exposure to other levels of the IV
No order effects to worry about
Counterbalancing is not required
participants
Colored
words BW
Test
Clock
Chair
Cab
Between subjects designs

39. Between subjects designs
participants
Colored
words BW
Test
Clock
Chair
Cab
Disadvantages
Individual differences between the people in the groups
Excessive variability
Non-Equivalent groups
Between subjects designs

40. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Individual differences

41. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Excessive variability due to individual differences
Harder to detect the effect of the IV if there is one R NR
Individual differences

42. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Non-Equivalent groups (possible confound)
The groups may differ not only because of the IV, but also because the groups are composed of different individuals
Individual differences

43. Dealing with Individual Differences
Strive for Equivalent groups
Created equally - use the same process to create both groups
Treated equally - keep the experience as similar as possible for the two groups
Composed of equivalent individuals
Random assignment to groups - eliminate bias
Matching groups - match each individuals in one group to an individual in the other group on relevant characteristics
Dealing with Individual Differences

44. Matching groups Group A Group B Matched groups
Trying to create equivalent groups
Also trying to reduce some of the overall variability
Eliminating variability from the variables that you matched people on
Red
Short
21yrs
matched
Red
Short
21yrs
matched
Blue
tall
23yrs
Blue
tall
23yrs
matched
Green
average
22yrs
Green
average
22yrs
Color
Height
Age
matched
Brown
tall
22yrs
Brown
tall
22yrs
Matching groups

45. Between vs. Within Subjects Designs
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
Within-subjects designs
All participants participate in all of the conditions of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

46. Within subjects designs
Advantages:
Don’t have to worry about individual differences
Same people in all the conditions
Variability between conditions is smaller (statistical advantage)
Fewer participants are required
Within subjects designs

47. Within subjects designs
Disadvantages
Range effects
Order effects:
Carry-over effects
Progressive error
Counterbalancing is probably necessary to address these order effects
Within subjects designs

48. Within subjects designs
Range effects – (context effects) can cause a problem
The range of values for your levels may impact performance (typically best performance in middle of range).
Since all the participants get the full range of possible values, they may “adapt” their performance (the DV) to this range.
Within subjects designs

49. Order effects Carry-over effects
Transfer between conditions is possible
Effects may persist from one condition into another
e.g. Alcohol vs no alcohol experiment on the effects on hand-eye coordination. Hard to know how long the effects of alcohol may persist.
test
Condition 2
Condition 1
How long do we wait for the effects to wear off?
Order effects

50. Order effects Progressive error
Practice effects – improvement due to repeated practice
Fatigue effects – performance deteriorates as participants get bored, tired, distracted
Order effects

51. Dealing with order effects
Counterbalancing is probably necessary
This is used to control for “order effects”
Ideally, use every possible order
(n!, e.g., AB = 2! = 2 orders; ABC = 3! = 6 orders, ABCD = 4! = 24 orders, etc).
All counterbalancing assumes Symmetrical Transfer
The assumption that AB and BA have reverse effects and thus cancel out in a counterbalanced design
Dealing with order effects

52. Counterbalancing Simple case Two conditions A & B
Two counterbalanced orders: AB BA
participants
Colored
words BW
Test
Counterbalancing

53. Often it is not practical to use every possible ordering
Partial counterbalancing
Latin square designs – a form of partial counterbalancing, so that each group of trials occur in each position an equal number of times
Counterbalancing

54. Partial counterbalancing
Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
1) Unbalanced Latin square: each condition appears in each position (4 orders) D C B A
Order 1
Order 2
Order 3
Order 4 A D C B B A D C C B A D
Partial counterbalancing

55. Partial counterbalancing
Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
2) Balanced Latin square: each condition appears before and after all others (8 orders) A B C D A B D C
Partial counterbalancing

56. Mixed factorial designs
Treat some factors as within-subjects (participants get all levels of that factor) and others as between-subjects (each level of this factor gets a different group of participants).
This only works with factorial (multi-factor) designs
Mixed factorial designs