Reliability and Validity in Experimental Research ♣ Chapter 7 Back to Brief Contents Introduction  Relationship between Reliability and Validity  Experimental Reliability  Reliability and the Independent Variable  Reliability and the Dependent Variable  Measurement Error  Methods of Assessing Reliability  Experimental Validity  Statistic Conclusion Validity  Internal Validity 

7.0 Introduction -1 Reliability Experimental reliability Back to Chapter Contents Reliability Consistency, stability, or repeatability Experimental reliability Consistency, stability, or repeatability of the results of an experimental study

7.0 Introduction -2(end) Validity Experimental validity Back to Chapter Contents Validity The correctness or truthfulness of an inference Experimental validity The correctness or truthfulness of an inference that is made from the results of an experiment

7.1 Relationship between Reliability and Validity Back to Chapter Contents Experimental reliability can exist without experimental validity Experimental validity cannot exist without experimental reliability (e.g.) speedometer, tires

7.2 Experimental Reliability -1 Back to Chapter Contents Number of research participants More Participants  more reliable Large N: less likely to be influenced by random or chance factors Reliability of identifying the type of participant needed for the study Indicators used select P with the same characteristics (e.g.) Beck Depression Inventory

7.2 Experimental Reliability -2 Back to Chapter Contents Characteristics of the researchers (e.g.) sleep deprivation on heart rate: male P, male vs. attractive blond Maximize reliability: the same researcher collects data from all P Replication: researcher with the same characteristics should be employed

7.2 Experimental Reliability -3(end) Back to Chapter Contents Ability of the researcher Use the same procedure Well trained  good recording (e.g.) EEG

7.3 Reliability and the Independent Variable Back to Chapter Contents Consistency: a psychological construct is created in the P Concrete operations must produce the same effect in P for reliability to exist (e.g.) Velten (1968): mood state read a set of statements and to feel and place themselves in that situation

7.4 Reliability and the Dependent Variable Back to Chapter Contents The effect measured should be about the same regardless of when it is measured

7.5 Measurement Error -1 Back to Chapter Contents Definition — inaccuracy that exists when measuring a psychological characteristic or behavior Factors causing measurement error — P respond differently on different occasions (e.g.) fatigue, emotionally upset Relationship between measurement error and reliability Measurement error   Reliability 

7.5 Measurement Error -2(end) Back to Chapter Contents Measurement = true score + error score Most experiments: single-occasion events Epstein (1981): low reliability Averaged over several days Epstein (1979) Fig 7.1 Repeat and Average (but most exp. cannot) Replication of the results of single-occasion studies

7.6 Methods of Assessing Reliability Back to Chapter Contents Successive measurement Test-retest, Equivalent forms Simultaneous measurement Interrater reliability: Consistency of ratings made by the different observers (continuous v.) Interobserver agreement: Percentage of times different raters agree (discrete v.) Internal consistency Split-half, Cronbach’s alpha

7.7 Experimental Validity Back to Chapter Contents Definition—accuracy or correctness of an inference Types of validity that maximize obtaining experimental validity (Shadish et. Al., 2002) Tab 7.1 Statistical conclusion validity Internal validity Construct validity External validity

7.8 Statistical Conclusion Validity Back to Chapter Contents Refers to the accuracy of an inference that the IV and DV covary Threats insufficient research participants -- power Many others but beyond this course to discuss

7.9 Internal Validity -1 Back to Chapter Contents Refers to the accuracy of the inference that the IV caused the effect observed in the DV Primary threat — confounding extraneous variables (EV) Confounding: systematically vary with IV EV: variable other than IV that influences DV

7.9 Internal Validity -2 Threats to Internal Validity Back to Chapter Contents Threats to Internal Validity Shadish, Cook & Campbell (2002) History Maturation Instrumentation Testing Regression Artifact Attrition Selection Additive and Interactive Situation

7.9 Internal Validity -3 Threats to Internal Validity History Fig 7.2 Back to Chapter Contents Threats to Internal Validity History Fig 7.2 Any event occurring after the experimental treatment is introduced that could produce the observed outcome Maturation Changes in biological and psychological conditions that occur with the passage of time Instrumentation Changes in the assessment of the DV — human observers: more than one (check) training program

7.9 Internal Validity -4 Threats to Internal Validity Testing Back to Chapter Contents Threats to Internal Validity Testing Changes in a person’s score on the second administration of a test as a result of previously having taken the test Regression Artifact Tab 7.2 Fig 7.3 Attrition 耗損 Some people do not show up for the study or do not complete it

7.9 Internal Validity -5(end) Back to Chapter Contents Threats to Internal Validity Selection The choice of P for the various treatment groups is based on different criteria (e.g.) type of psychosis Additive and Interactive Situation The combined effect of several threats to internal validity — selection-maturation effect Fig 7.4 — selection-history effect — selection-instrumentation effect