Chapter 8 Construct and External Validity in Experimental Research ♣ ♣ Construct Validity   External Validity   Cautions in Evaluating the External Validity of Experiments   Relationship between Internal and External Validity    Back to Brief Contents

8.1 Construct Validity -1 Definition — extent to which we can infer higher-order constructs for our operations Fig 8.1 Constructs are used for  Research participants  Independent variable  Dependent variable  Experimental setting  Back to Chapter Contents

8.1 Construct Validity -2 Assessing Construct Validity   Threats to Construct Validity    Reactivity to the Experimental Situation    Experimenter Effect    Back to Chapter Contents

Assessing Construct Validity -1 Have a clear definition of the construct of interest  Problem is identifying prototypical features of the constructs  Affects ability to identify the concrete operations used to represent the construct  Back to Chapter Contents

Assessing Construct Validity -2(end) Assess the match between the constructs and the operations used to represent them  Ways to assess Content validity Criterion-related validity — Predictive — concurrent Any other source of evidence  Back to Chapter Contents

Threats to Construct Validity Inadequate explanation of the construct Tab 8.1 Tab 8.1 Construct confounding Mono-operation bias Mono-method bias Confounding constructs with level of constructs Treatment-sensitive factorial structure Reactive self-report changes Reactivity to the experimental situation Experimenter effects Novelty and disruption effects Compensatory equalization Compensatory rivalry Treatment diffusion  Back to Chapter Contents

Threats to Construct Validity Inadequate explanation of the construct If a construct is not adequately explained and analyzed, it can lead to a set of operations that do not represent the construct adequately Construct confounding the operations used in a study represent more than one construct Mono-operation bias a study uses only one operationalization of a construct This typically results in an underrepresentation of the construct and lowers construct validity  Back to Chapter Contents

Threats to Construct Validity Mono-method bias a study uses only one method (e.g., physiological recording) to operationalize a construct The method used may influence the results Confounding constructs with level of constructs a study investigates only a few levels of a construct (e.g., three doses of a drug), but makes inferences about the overall construct (e.g., the overall effect of the dose) Treatment-sensitive factorial structure an instrumentation change that occurs because of the experimental treatment  Back to Chapter Contents

Threats to Construct Validity Reactive self-report changes changes that a research participant may make on self-report measures as a result of a motivational shift after being included in the experimental study Reactivity to the experimental situation research participants’ perceptions and motives can affect the responses they make to the dependent variable and these responses can be interpreted as part of the treatment construct being tested  Back to Chapter Contents

Threats to Construct Validity Experimenter effects the experimenter’s attributes and expectancies can influence the responses made by the research participants and these responses can be interpreted as part of the treatment construct being tested Novelty and disruption effects Research participants usually respond better to a new and novel situation and poorly to one that disrupts their routine These effects are part of the overall treatment effect  Back to Chapter Contents

Threats to Construct Validity Compensatory equalization individuals try to provide the same benefits or services to the control group that are received by the experimental group Compensatory rivalry individuals resent being assigned to the control group and respond more negatively than would be expected, because of the resentment they feel Treatment diffusion individuals in one treatment group receive some or all of another group’s treatment  Back to Chapter Contents

Reactivity to the Experimental Situation 1/3 Refers to research participants ’ motives and perceptions influencing the response to the DV Participant Effect  Demand characteristics Any of the cues available in an exp, such as instructions, experimenter, rumors, experimental settings  Primary motive — positive self-presentation  Back to Chapter Contents

Reactivity to the Experimental Situation 2/3 Conditions producing a positive self-presentation motive  Tedeschi, Schlenker, & Bonoma (1971) P believe that others view their behavior as indicative of their true intentions, beliefs, or feelings Exp. constructed  P believe that others think their behavior is externally determined  Back to Chapter Contents

Reactivity to the Experimental Situation 3/3 Implication for research  Intertreatment interaction Perception by P in different treatment groups that they can fulfill the positive self-presentation motive by responding in different ways  Intratreatment interaction Perception by P in the same treatment group that they can fulfill the positive self-presentation motive by responding in different ways  Back to Chapter Contents

Experimenter Effect 1/3 Experimenter Effect Fig 8.2 Fig 8.2  Experimenter has motive of supporting the study hypothesis  Can unintentionally lead to recording errors  Back to Chapter Contents

Experimenter Effect 2/3 Ways experimenter may bias the study  Experimenter attributes Biosocial attributes (e.g.) age, sex, race Psychosocial attributes psychometrically determined characteristics (e.g.) anxiety level, hostility, authoritarianism, intelligence, dominance, warmth Situational factors (e.g.) prior contact of P, naive or experienced  Experimenter expectancies (cont.)  Back to Chapter Contents

Experimenter Effect 3/3 Ways experimenter may bias the study  Experimenter attributes  Experimenter expectancies Effect on experimenter (e.g.) recording errors (biased, but low error rate) Effect on research participant Tab 8.2 Tab 8.2 Mediation of expectancies — Handling in animal research — Nonverbal communication in human studies Magnitude of expectancies — can exist in animal and human research and can be greater than the IV Fig 8.3 Fig 8.3  Back to Chapter Contents

8.2 External Validity 1/4 Definition—generalizing across people, settings, treatment variations, outcomes and times Represents a test of interactions Threats to external validity  Population validity  Ecological validity  Temporal validity  Treatment variation validity  Outcome validity  Back to Chapter Contents

8.2 External Validity 2/4 Population Validity The extent to which the results of a study can be generalized to the larger population  Two-step inferential process Fig 8.4 Tab 8.3 Fig 8.4Tab 8.3 Sample  Experimentally accessible population  Target population  College students as P less emotionally and impulsively in laboratory studies  Selection by Treatment Interaction if interaction exists  cannot generalize to the target population  Back to Chapter Contents

8.2 External Validity 3/4 Ecological Validity The extent to which the results of a study can be generalized across settings or environmental conditions Temporal Validity The extent to which the results of an experiment can be generalized across time  Seasonal variation A variation that occurs at regular time intervals  Cyclical variation A regular variation that occurs within people and other organisms (ex) circadian rhythm, MC  Back to Chapter Contents

8.2 External Validity 4/4 Treatment Variation Validity (IV) The generalizability of results across variation of the treatment Outcome Validity (DV) The generalizability of results across different but related dependent variables  Back to Chapter Contents

8.3 Cautions in Evaluating the External Validity of Experiments Some studies conducted to increase knowledge and not to generalize to a real-life situation Some studies assess a theoretical process Moving out of the laboratory does not insure generalization  Back to Chapter Contents

8.4 Relationship between Internal and External Validity Relationship between internal and external validity — often an inverse relationship  Back to Chapter Contents