Pitfalls in Experimental Research Threats to Internal and External Validity

overview of threats to validity conceptual slippage faulty manipulation loose procedures order effects experimenter effects fudging researcher personal attribute effect researcher unintentional expectancy effect sensitization recording errors history and maturation basement and ceiling effects Hawthorne effect social desirability bias selection bias (sampling) intersubject bias mortality (attrition)

conceptual slippage a gap between how a variable is conceptualized and how it is actually operationalized in an experiment –example: prejudice It is difficult for a scale to capture the full richness of a concept remedies: –limit the scope of the variables to be studied –don't over-claim the nature and/or purpose of the investigation –triangulation; use multiple outcome measures

faulty manipulation failure to manipulate a variable, or create a stimulus condition as intended –example: Janis & Feshbach's (1953) study on fear appeals remedies: –include a manipulation check: Teven & Comadena (1996) office aesthetics and teacher credibility –cover story can disguise true purpose of an experiment –use neutral observers to judge stimulus materials

loose procedures ambiguity or imprecision in the experimental protocol –example: Raffetto (1967) study on sensory deprivation and hallucinations remedies: –run a pilot study –script out all instructions to participants

order effects the order, sequence, or placement of items in a series can influence participants' perceptions –example: Schuman, Presser, & Ludwig (1981) survey on attitudes toward abortion remedies: –shuffle the order of items within a questionnaire, e.g. use multiple versions of the questionnaire –alter the order if different questionnaires are completed together

experimenter effects fudging: –example: Stephen Gould "IQ Argument" researcher expectations –example: intercessory prayer cueing –example: Rosenthal (1966) unintentional paralinguistic and kinesic cues by experimenters demand characteristics of the experimental situation

experimenter effects, continued researcher observational biases –problem of “observer drift,” e.g., changing standards or criteria –problem of “observer bias” remedies: –minimize interaction with participants –use multiple experimenters (different genders, different ethnicities, etc.) –use experimenters who are “blind” to the purpose of the study

sensitization pre-test measurements may bias post-test measurements solution: –avoid pre-tests, if possible –use different pre- and post-test measures

recording errors errors in entering, translating, or transforming data –example: Kennedy & Uphoff (1939) extrasensory perception study remedies: –use double-checks during data entry –pre-code surveys, questionnaires, etc.

history and maturation history: events happening outside the experiment that influence participants' responses within the experiment maturation: internal changes in participants during an experiment (developmental, emotional, etc.) remedies: –keep experiments short –sequester participants if possible

ceiling or basement effects ceiling effect: responses exceed capacity of measuring instrument; responses are “off the scale.” floor or basement effect: responses are below threshold of measuring instrument remedies: –use sensitive, precise measures –use established scales, instruments

Hawthorne effect the mere knowledge by a participant that he/she is being observed may alter his/her natural behavior –example: Western Electric plant in Illinois (circa late 1970's) remedies –use a cover story –use unobtrusive measures

social desirability bias Self reports are often distorted because respondents want to present themselves in the best possible light –Examples: in person interviews involving embarrassing topics; infidelity, spouse abuse, cheating, etc. Remedies –Emphasize anonymity of responses, importance for obtaining genuine responses, no correct answers –use of indirect questioning: asking respondents to project how they think other persons would answer

selection bias failure to gather a random or representative sample, or a reliance on self-selected groups –example: the proverbial "college freshman" used in many social science studies remedies: –use a random, representative, sufficient sample (easier said than done) –use random assignment within the experiment

inter-participant bias Participant interaction may taint responses, both inside and outside of the experiment –example: social proof and laughter –example: surveys done in large lecture halls remedies: –isolate participants from one another –ask participants not to disclose the purpose of an investigation to others

mortality or attrition Subjects may drop-out or cease participating in a study. The ones who drop out may be different from the ones who remain. –example: long-term “longitudinal” research –example: studies on intimates remedies: –keep studies short –provide incentives for participating