Control problems in experimental research Week E

Today’s questions Last week, we understood what experiments are (not) A robust experiment allows strong conclusions to be made. How do we design robust experiments? We control for variables which can upset or compromise our conclusions!

Two basic research designs Experiment Between-subjects Within-subjects

Design & control problems Each design poses its own (unique) problems with controls Between subjects Creating equivalent groups Within subjects Minimizing sequence effects

Control problems in between-subjects designs

Recall: Between-subjects Independent groups are formed. Participants in Group A are not the same participants as those in Group B Conclusions are valid if before the manipulation, Group A ≡ Group B

How are independent groups formed? Randomly assigned Matched on key variables Naturally occurring E.g., males vs. females; young vs. old; patients vs. normals

Recall from SRM I: Making population inferences from samples

Creating equivalence groups: Random assignment Every person volunteering for the study has an equal chance of being placed in any groups to be formed The goal is to take any individual difference factors that could bias the study and spread them evenly throughout the different groups No guarantee of an even spread, but the larger number of participants, the better your chances

Ideally, both groups should be equivalent on all traits

How to randomly assign participants? Use a random number generator Coin flip Table of random numbers Pre-program randomization into your computer script (if you’re running experiments on computers)

Creating equivalence groups: Matching Random assignment requires large number of subjects to even out individual differences Sometimes this is impossible or impractical Patients with rare diseases are difficult to find, whereas healthy controls are abundant Solution: Match the controls on key variables that theoretically may confound the experiment

Matched groups design 49, married 39, single 28, single 59, married Treatment group (trauma to left prefrontal cortex) Healthy controls, matched on age and marital status 49, married 39, single 28, single 59, married 57, single 34, single 23, single 49, married 39, single 28, single 59, married 57, single 34, single 23, single How do you know what variables are important to match?

Matching There are infinite number of variables that you can match The variable you choose to match should be the one that is theoretically most important. In a study of memory functions after brain trauma, which would be more important to match: age, marital status, education, sexual preference,?

A statistical alternative to matching: Using covariates (not in your exam/syllabus, but useful to know) Matching is a methodological way of creating equivalent groups. An alternative is to collect data of potential covariates and then use them as control variables during data analysis The procedure is known as ANCOVA (analysis of covariance), but can also be implemented in regression

Control problems in within-subjects designs

Within-subjects design Controlling sequence effects Testing once per condition Complete counterbalancing Partial counterbalancing Testing more than once per condition Reverse counterbalancing Block randomization Problems with counterbalancing

Sequence effects In within-subjects design, multiple measurements are taken from the same participant. It is possible that a prior measurement will influence a later measurement

Testing once per condition In a face perception study, participants judged the beauty of each face What would be the problem if Face A is always presented first, and Face B is always presented second? A B

Counterbalancing: Two stimuli Subject 1: AB Subject 2: BA Subject 3: AB Subject 4: BA … A B

Counterbalancing: Three stimuli [AB, BA] combination is simple for two stimuli. What if you have three faces? B A C

Counterbalancing: Three stimuli Subject 1: ABC Subject 2: BAC Subject 3: CAB Subject 4: BCA Subject 5: CBA Subject 6: ABC Number of combinations = C! (C! means “three factorial” in this case) That is, 3! = 1 × 2 × 3 Ideally, you should have multiples of # of orders, i.e., 6, 12, 18, 24, etc.

Counterbalancing: >3 stimuli 3! is small, but when you have 4!, 5!, etc., the number of combinations quickly outnumber your resources (participants) Consider Fully randomizing all presentations Using Latin square (see textbook description)

Block counterbalancing Sometimes, stimuli can be grouped into “blocks”, for example, pink vs. black dress A B C D E F X Y You have 6! = 720 combinations

Block randomization You can divide stimuli into blocks, such that psychologically, you assume that responses within one block are uncontaminated by those of the other block Subject 1: X, Y Subject 2 : Y, X … Sometimes these blocks can happen to be your IV of interest e.g., pink dress vs. black dress

Block randomization You can then randomize (or counterbalance) the order of presentation within each block Subject 1: ABC DEF Subject 2: EFD BAC Subject 3: CAB FED Subject 4: DEF BCA … A B C D E F

Issues in longitudinal design

Longitudinal design Think of longitudinal design as an ultra-long within subjects design How long? Days, weeks, months, years, decades. Some research questions are inherently longitudinal Romantic relationship dynamics Aging Clinical drug trials (sometimes)

Example: Memory in young and old people Does age affect memory? Between-subjects method: Recruit old vs. young What is the problem with any conclusions that recruited old and young people?

Problem: Cohort effects Back then… Now… With a cross-sectional design (recruiting old vs. young), you cannot rule out cohort effects.

Longitudinal design The same question (“Does age affect memory?”) can be investigated by tracking individuals over time. Two problems with longitudinal design Expensive Attrition

Control groups

Control groups A variable in an experiment which is held constant in order to assess the relationship between other variables, is the control variable. Members possessing control variables are known as control groups. Provides a basis of comparison, and thus sometimes known as “comparison groups” (we will discuss this again later)

Control groups Are control groups necessary in all experimental research? Let’s look at the definition again: “A systematic research in which the investigator directly varies some factor (or factors), holds all else constant, and observes the results of the systematic variation.” Experimental research does not require control groups.

Control groups rule out confounds But having control groups help to refine explanations, to rule out alternative explanations (confounds): Study A Study B

Confounds Also known as “extraneous variables” Any variable that is not of interest to the researcher but which might influence the conclusions.

Is there a confound? The first 10 people to arrive in class were assigned to the Experimental Group, and the last 10 people to arrive were assigned to the Control group. At the end of the experiment, the experimenter finds differences between the Experimental group and the Control group, and claims these differences are a result of the experimental procedure. Yes No

Is there a confound? Participants read a passage about romantic love (or jealousy), and then rated sweetness of candies after tasting them. Is positive emotion a confound? Yes No Chan et al. (2012). What do love and jealousy taste like? Emotion.

Is there a confound? Participants watched Mr Bean (or a video about an orphan girl) and then rated their mood. Is number of laughter a confound? Note: So-called ‘confounds’ are sometimes precisely the characteristics of the independent variable! Yes No

Sidetrack: Control groups in daily life Consider this statement: “Wives should give daily massages to their husbands, cook for them, and make them happy.” K. Q. Chan Is K. Q. Chan a chauvinist?

Sidetrack: Control groups in daily life Chauvinist Humanist “Wives should give daily massages to their husbands, cook for them, and make them happy. Husbands should just sit back and relax, and let their wives serve them.” “Wives should give daily massages to their husbands, cook for them, and make them happy. Husbands should do the same too.”

Advanced issues in control groups

What exactly is a control group? Some researchers think that we should instead rename it as “comparison group” A researcher wanted to examine gender differences in cursing behavior. What is the appropriate control group for gender? Males vs. vs. Females vs. Neutral gender (?!?!)

How many variables should you control for? There are infinite number of confounds in a research Some are more important to address than others Control for those that severely threaten the internal validity of your conclusion

What is a good control group? A researcher tested two drug treatments: Drug A vs. No drug. Is ‘No drugs’ a good control group? The problem of expectancy effects

Control groups should ideally be meaningful An extremely well-controlled research gives strong conclusions about the theory (strong internal validity) But may lead to poor external validity – the degree to which conclusions can be generalized beyond the settings which it is tested. This problem is exacerbated when meaningless control groups are used – groups that do not reflect the ‘natural’ way of behaving

Discussion A researcher wanted to perform an experiment to study the effect of happiness on helping behavior. What would be the appropriate control/comparison group?

Take home messages You can’t control for everything. But you do need to control for important things so that you can make strong conclusions. Control groups need to be meaningful.