1. Control problems in experimental research
Week E

2. Today’s questions Last week, we understood what experiments are (not)
A robust experiment allows strong conclusions to be made.
How do we design robust experiments?
We control for variables which can upset or compromise our conclusions!

3. Two basic research designs
Experiment
Between-subjects
Within-subjects

4. Design & control problems
Each design poses its own (unique) problems with controls
Between subjects
Creating equivalent groups
Within subjects
Minimizing sequence effects

5. Control problems in between-subjects designs

6. Recall: Between-subjects
Independent groups are formed.
Participants in Group A are not the same participants as those in Group B
Conclusions are valid if before the manipulation, Group A ≡ Group B

7. How are independent groups formed?
Randomly assigned
Matched on key variables
Naturally occurring
E.g., males vs. females; young vs. old; patients vs. normals

8. Recall from SRM I: Making population inferences from samples

9. Creating equivalence groups: Random assignment
Every person volunteering for the study has an equal chance of being placed in any groups to be formed
The goal is to take any individual difference factors that could bias the study and spread them evenly throughout the different groups
No guarantee of an even spread, but the larger number of participants, the better your chances

10. Ideally, both groups should be equivalent on all traits

11. How to randomly assign participants?
Use a random number generator
Coin flip
Table of random numbers
Pre-program randomization into your computer script (if you’re running experiments on computers)

12. Creating equivalence groups: Matching
Random assignment requires large number of subjects to even out individual differences
Sometimes this is impossible or impractical
Patients with rare diseases are difficult to find, whereas healthy controls are abundant
Solution: Match the controls on key variables that theoretically may confound the experiment

13. Matched groups design 49, married 39, single 28, single 59, married
Treatment group (trauma to left prefrontal cortex)
Healthy controls, matched on age and marital status
49, married
39, single
28, single
59, married
57, single
34, single
23, single
49, married
39, single
28, single
59, married
57, single
34, single
23, single
How do you know what variables are important to match?

14. Matching There are infinite number of variables that you can match
The variable you choose to match should be the one that is theoretically most important.
In a study of memory functions after brain trauma, which would be more important to match: age, marital status, education, sexual preference,?

15. A statistical alternative to matching: Using covariates
(not in your exam/syllabus, but useful to know)
Matching is a methodological way of creating equivalent groups.
An alternative is to collect data of potential covariates and then use them as control variables during data analysis
The procedure is known as ANCOVA (analysis of covariance), but can also be implemented in regression

16. Control problems in within-subjects designs

17. Within-subjects design
Controlling sequence effects
Testing once per condition
Complete counterbalancing
Partial counterbalancing
Testing more than once per condition
Reverse counterbalancing
Block randomization
Problems with counterbalancing

18. Sequence effects
In within-subjects design, multiple measurements are taken from the same participant.
It is possible that a prior measurement will influence a later measurement

19. Testing once per condition
In a face perception study, participants judged the beauty of each face
What would be the problem if Face A is always presented first, and Face B is always presented second? A B

20. Counterbalancing: Two stimuli
Subject 1: AB
Subject 2: BA
Subject 3: AB
Subject 4: BA … A B

21. Counterbalancing: Three stimuli
[AB, BA] combination is simple for two stimuli.
What if you have three faces? B A C

22. Counterbalancing: Three stimuli
Subject 1: ABC Subject 2: BAC Subject 3: CAB Subject 4: BCA Subject 5: CBA Subject 6: ABC
Number of combinations = C!
(C! means “three factorial” in this case)
That is, 3! = 1 × 2 × 3
Ideally, you should have multiples of # of orders, i.e., 6, 12, 18, 24, etc.

23. Counterbalancing: >3 stimuli
3! is small, but when you have 4!, 5!, etc., the number of combinations quickly outnumber your resources (participants)
Consider
Fully randomizing all presentations
Using Latin square (see textbook description)

24. Block counterbalancing
Sometimes, stimuli can be grouped into “blocks”, for example, pink vs. black dress A B C D E F X Y
You have 6! = 720 combinations

25. Block randomization
You can divide stimuli into blocks, such that psychologically, you assume that responses within one block are uncontaminated by those of the other block
Subject 1: X, Y
Subject 2 : Y, X …
Sometimes these blocks can happen to be your IV of interest e.g., pink dress vs. black dress

26. Block randomization
You can then randomize (or counterbalance) the order of presentation within each block
Subject 1: ABC DEF
Subject 2: EFD BAC
Subject 3: CAB FED
Subject 4: DEF BCA … A B C D E F

27. Issues in longitudinal design

28. Longitudinal design
Think of longitudinal design as an ultra-long within subjects design
How long? Days, weeks, months, years, decades.
Some research questions are inherently longitudinal
Romantic relationship dynamics
Aging
Clinical drug trials (sometimes)

29. Example: Memory in young and old people
Does age affect memory?
Between-subjects method:
Recruit old vs. young
What is the problem with any conclusions that recruited old and young people?

30. Problem: Cohort effects
Back then…
Now…
With a cross-sectional design (recruiting old vs. young), you cannot rule out cohort effects.

31. Longitudinal design
The same question (“Does age affect memory?”) can be investigated by tracking individuals over time.
Two problems with longitudinal design
Expensive
Attrition

32. Control groups

33. Control groups
A variable in an experiment which is held constant in order to assess the relationship between other variables, is the control variable.
Members possessing control variables are known as control groups.
Provides a basis of comparison, and thus sometimes known as “comparison groups” (we will discuss this again later)

34. Control groups
Are control groups necessary in all experimental research?
Let’s look at the definition again:
“A systematic research in which the investigator directly varies some factor (or factors), holds all else constant, and observes the results of the systematic variation.”
Experimental research does not require control groups.

35. Control groups rule out confounds
But having control groups help to refine explanations, to rule out alternative explanations (confounds):
Study A
Study B

36. Confounds Also known as “extraneous variables”
Any variable that is not of interest to the researcher but which might influence the conclusions.

37. Is there a confound?
The first 10 people to arrive in class were assigned to the Experimental Group, and the last 10 people to arrive were assigned to the Control group. At the end of the experiment, the experimenter finds differences between the Experimental group and the Control group, and claims these differences are a result of the experimental procedure.
Yes No

38. Is there a confound?
Participants read a passage about romantic love (or jealousy), and then rated sweetness of candies after tasting them.
Is positive emotion a confound?
Yes No
Chan et al. (2012). What do love and jealousy taste like? Emotion.

39. Is there a confound?
Participants watched Mr Bean (or a video about an orphan girl) and then rated their mood.
Is number of laughter a confound?
Note: So-called ‘confounds’ are sometimes precisely the characteristics of the independent variable!
Yes No

40. Sidetrack: Control groups in daily life
Consider this statement:
“Wives should give daily massages to their husbands, cook for them, and make them happy.”
K. Q. Chan
Is K. Q. Chan a chauvinist?

41. Sidetrack: Control groups in daily life
Chauvinist
Humanist
“Wives should give daily massages to their husbands, cook for them, and make them happy. Husbands should just sit back and relax, and let their wives serve them.”
“Wives should give daily massages to their husbands, cook for them, and make them happy. Husbands should do the same too.”

42. Advanced issues in control groups

43. What exactly is a control group?
Some researchers think that we should instead rename it as “comparison group”
A researcher wanted to examine gender differences in cursing behavior. What is the appropriate control group for gender?
Males vs. vs. Females vs. Neutral gender (?!?!)

44. How many variables should you control for?
There are infinite number of confounds in a research
Some are more important to address than others
Control for those that severely threaten the internal validity of your conclusion

45. What is a good control group?
A researcher tested two drug treatments: Drug A vs. No drug.
Is ‘No drugs’ a good control group?
The problem of expectancy effects

46. Control groups should ideally be meaningful
An extremely well-controlled research gives strong conclusions about the theory (strong internal validity)
But may lead to poor external validity – the degree to which conclusions can be generalized beyond the settings which it is tested.
This problem is exacerbated when meaningless control groups are used – groups that do not reflect the ‘natural’ way of behaving

47. Discussion
A researcher wanted to perform an experiment to study the effect of happiness on helping behavior. What would be the appropriate control/comparison group?

48. Take home messages You can’t control for everything.
But you do need to control for important things so that you can make strong conclusions.
Control groups need to be meaningful.