Experimental Confounds Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American psychologist, 17(11), Poulton, E. C. (1975). Range effects in experiments on people. The American Journal of Psychology, 3-32.

On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications Presented by Michael Hammes

Nuremberg Code 1964 Declaration of Helsinki 1966 Public Health Service issues policy requiring informed consent 1974 National Research Act 1978 Belmont Report Respect for persons, beneficence, and justice 1981 Common Rule (45 CFR 46) IRB, informed consent, additional protections for vulnerable populations

What Are Demand Characteristics? The concept of “demand characteristics” refers to participants being aware of what the researcher is trying to investigate, or anticipates finding, and what this implies for how participants are expected to behave. McCambridge, J., De Bruin, M., & Witton, J. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: a systematic review. PLoS One, 7(6), e39116.

Experimental Design in Behavioral Science Difficult to meet two necessary requirements for meaningful experimentation Reproducibility Ecological validity (External Validity, Generalizability) Why? Thinking, conscious participants, not passive responders to stimuli Subjects react not only to stimuli but to the experimental situation This reaction is a factor in reproducibility and ecological validity

Qualities Peculiar To Psychological Experiments Special form of social interaction known as “taking part in an experiment” Explicit agreement of participant to participate Participant and Experimenter roles Well understood Well-defined mutual role expectations

Participant Role Implicitly agrees to perform a wide range of actions on request without inquiring as to their purpose, and frequently without inquiring as to their duration. Agrees to tolerate a considerable degree of discomfort, boredom, or actual pain, if required to do so by the experimenter Assumes a legitimate purpose will be served (shared with experimenter). Q: Is this still true today?

The “Good” Participant Hope and expectation that study will contribute materially to science and perhaps human welfare in general Has a stake in the study outcome Tries to help validate the experimental hypothesis by being a “good” participant Attempts to ascertain true purpose (assuming true purpose not shared) Will form hypothesis based on cues Cues will create demand characteristics

Demand Characteristics Totality of cues give rise to participant hypothesis Rumors Information Experimenter Setting Explicit and Implicit Communications Demand characteristics will vary for each participant Demand characteristics will influence participant behavior

Demand Characteristics as a Confound Participant behavior in any experiment will be determined by two sets of variables: Experimental (Independent) variables Demand characteristics of the experimental situation The extent to which participant behavior is related to demand characteristics rather than experimental variables will affect replication and generalizability

The Problem of Demand Characteristics Demand characteristics cannot be eliminated Response to demand characteristics is both conscious and subconscious Resulting bias can be positive or negative

Controlling for Demand Characteristics Post-experimental inquiry Open-ended questions to account for “pact of ignorance” Has correlation problems and is subject to demand characteristics Pre-experimental inquiry Exposure to experimental information and materials without participation Inquiry participants cannot be subjected to experiment Eliminate experimental variables Participants simulate exposure to the experimental variables “Blind” researcher evaluates participants for simulation Psychological placebo Ascertain whether subject perceives treatment to be experimental or control

Summary Participants are actively involved in experiment Demand characteristics are a confounding variable Determine when and to what extent demand characteristics have an influence Four suggested techniques The Takeaway: Consider demand characteristics in all research, and control if possible

Discussion

Poulton, E. C. (1975). Range effects in experiments on people. The American Journal of Psychology, Presented by Michael Hammes

What Is A Range Effect? With a range effect, participant responses to stimuli are influenced by the range of stimuli, the range of responses used by (available to) the participant, or both. Range effects generally involve a central tendency, but not always. Poulton, E. C. (1973). Unwanted range effects from using within-subject experimental designs. Psychological Bulletin, 80(2), 113.

Defining the Problem Range effects occur when each participant receives more than one experimental condition (within-subjects design) No way of discovering whether a within-subjects design has introduced an unwanted range effect except by repeating parts of the experiment using a separate-groups (between-subjects) design. Poulton, E. C. (1973). Unwanted range effects from using within-subject experimental designs. Psychological Bulletin, 80(2), 113.

Range Effect Examples Central Tendency Asymmetrical training/learning transfer Scanning alphanumeric matrices Short-term memory scanning Time-order errors Ratings of sensory magnitude Direct magnitude estimates Preferences and choices Shape constancy Size constancy Step tracking Stimulus generalization Absolute judgments

Range Effect Examples Continued Middle Range Confusion Stimulus generalization Absolute judgments Serial learning Probability (often studied deliberately) Stimulus and response probability Probability learning Maximizing, probability matching, averaging

Avoiding Range Effects Many range effects occur because participants learn the range of stimuli in the experiment Many, but not all, stimulus-determined range effects can be prevented by restricting each participant to a single stimulus “No psychologist should use a within-subjects design, except for a special purpose, without combining it with a separate-groups design”

Unavoidable Experimental Range Effects Shape constancy – influenced by range of stimuli based on initial shape Probability – participant knows probabilities range from 0 to 1 and will select probability near middle Subjective sensory magnitudes – very first judgment influenced by idea of range of possible stimuli Range of possible responses affects participant response

Special Case: Prior Experience Participant may bring range effect to experiment Possible solution is to select participants with appropriate (read: similar) past experience Retraining not substitute for selection Can suffer from asymmetrical transfer

Conclusions Within-subjects experimental designs produce range effects when they are used to measure almost any kind of human behavior Sometimes range effects give false support to unacceptable theoretical formulations Researchers must decide to what extent experimental results are determined by experiment design Evaluate conflicting results against experimental design Do not select from conflicting results to fit theory When results of between- and within-subjects designs conflict, accept between-subjects design result

Discussion