1. Class 4 Experimental Studies: Validity Issues Reliability of Instruments
Chapters 7 Spring 2017

2. Dependent Variables: Continuous
Psychometric Properties
Reliability
Validity

3. Reliability
The extent to which scores show true variance in attributes across participants as opposed to error variance
Assumption: error variance is random, therefore it does not correlate with anything
More items of good quality = higher reliability
One-item scales have very low reliability; estimated around r =.25

4. Sources of Measurement Error
Specific error
something unique about the instrument (or observational task) that differs from what the researcher intended (e.g. social desirability; reading level; idioms)
Transient error
some temporary factor that affects the measurement (e.g. order of instruments; historical events; noise while observations occur; tiredness; inattention)

5. Types of Measures Observational Measures
Self-Report Paper-and-Pencil Measures
Content tests- right/wrong answers
Likert-scales (3 to 7 response options)

6. Types of Reliability Inter-Scorer Agreement - Observation Measures
Test/Re-Test
Alternate Forms (achievement)
Internal Consistency
how items correlate with each other

7. Internal Consistency Reliability
Split-Half
Kuder-Richardson
Dichotomous items- right /wrong; Yes/No
Chronbach Alpha α
Average correlation of all possible split-half reliabilities

8. Internet AddictionMeasure: Q 3a-c
12-item measure rx= .70 to rx= .95 among college students
Expected reliability estimates among the adolescent sample
Three-item measure: rx=.40 to rx=.55

9. Internet Addiction: Q 2a
12-item measure: internal consistency reliability coefficients rx= .70 to rx= .95 among college students
70% to 95% of variability in respondents’ scores is due to and the rest is due to .

10. Reliability Estimates: Q 2a
Extent to which variability is due to true variation versus error
Cronbach alpha =. 70;
70% of variation in scores is due to true differences in internet addiction & 30% is due to error
Cronbach alpha =. 95;
95% of variation in scores is due to true differences in internet addiction & 5% is due to error

11. Internet Addiction Measure: Q 2b
12-item measure rx= .70 to rx= .95 among college-age samples
Expected reliability estimates among the adolescent sample

12. Wellbeing Measure: Q 2b
12-item measure rx= .70 to rx= .95 among college samples
Expected reliability coefficient among adolescents =>.70
Yes/No ?
Reliability refers to scores with specific and .
It’s a property of not of the instruments.

13. Internet Addiction Measure: Q 2b
12-item measure rx= .70 to rx= .95 among College students
Expected reliability coefficient among adolescents =>.70
No. Reliability refers to scores with specific respondents and conditions
It’s a property of scores, not of instruments

14. More accurate?
The internal consistency of the Internet Addiction Scale (IAS) has ranged from to .95.
The internal consistency of scores in the Internet Addiction Scale (IAS) with college students in the U.S. has ranged from .70 to .95.

15. Wellbeing Measure: Q 2c Three-item measure: rx=.40 to rx=.45.
To improve the scores’ reliability just add 5 or 6 items:
True False Not sure

16. Internet Addiction Measure: Q 2c
Three-item measure: rx=.40 to rx=.45 –
New items increase reliability only if they are of good quality

17. Reliability and Correlation:
In correlational research, how does the reliability of two scores (e.g. depression and self -esteem) affect the probability that the observed correlation coefficient between scores in two variables approximates the “true” correlation coefficient among the constructs?

18. Internal Reliability and Correlation
Depression 1 Cronbach α = .45
Depression 2 Cronbach α = .90
Same sample: which r dep-se below will be larger?
DEP1 (α = .45) correl. Self Esteem
DEP2 (α = .90) correl. Self Esteem

19. Validity of Measures Construct Validity Predictive Validity
Factor structure -- latent constructs
Convergent and Discriminate Validity
Correlation with similar/dis-similar measures
Predictive Validity
Correlation among different constructs based on expected relations
Cross-sectional or Longitudinal

20. Reliability vs. Validity
Observed correlation coefficient will be smaller and less accurate with the less reliable measure
Correlations between constructs are attenuated by the (internal) reliability of the measures
The reliability of a measure puts a ceiling on its validity

21. Validity Of Experimental Designs
Do inferences from an outcome study results reflect how things actually are?
Does the manipulation (treatment) causes the observed outcome?
Threats to Validity

22. Threats to Statistical Conclusion Validity
Are the observed relations among IV (Manipulation) and DV (Outcome) variables accurate?
Power
Fishing
Unreliability of measures
Unreliability of treatment
implementation
Extraneous variables
Heterogeneity of
participants

25. Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
Power
Not mentioned -
Sample relatively small N per group = 29 -
Experimental design + Measures reliability +
2. Fishing
Adjusted P values (.01) for 6 measures +
Table 3 Effect sizes +
3. Unreliability
of Measures
HAM-A 3 trained, blind raters, consensus agrmnt +
Measures are well established +
Unreliability of Treatment Implementation
Specificity- Active ingredients in manual+
Monthly supervision + Competence ?
Fidelity blind reviewers 87% to 88% +
Experienced th. + CBT more exp. manual -
Extraneous Variance in Experimental Setting
Not known-
Where was therapy conducted - th’s private office?
Th./inv Allegi controlled thpst of both orientations +
Heterogeneity of Participants
Most Females 81% + & partnered 79% +
Comorbidity other anx. & dep ect 72% -
Many exclusions p

26. Threats to Internal Validity
Can we conclude that there is a causal relation between the IV and the DV?
Selection to Treatment Groups
History
Attrition
Repeated Testing Effects
Reaction to Control Group Assignment

27. Threats to Internal Validity
Can we conclude that there is a causal relation between the IV and the DV?
Selection to Treatment Groups
Clear inclusion/exclusion criteria+
Random assignment to T groups+
Treat Groups = Dep pre-test +
History
Therapy appeared to occur for everyone at once + (?)
30 -sessions
Attrition
Low attrition CBT29/27/ Psycho D 28/25/23 similar reasons (1 each requ more treat.) +
Repeated Testing Effects
Moderate - measures at 3 points in time +
Reaction to Control Group Assignment
Two therapies +

28. Threats to Construct Validity
To what extent variables (DV & IV) capture desired constructs
Inadequate Explications
Mono-Operation Bias
Mono-Method Bias
Reactivity to Exp. Conditions
Experimenter Expectancies

29. Threats to Construct Validity
To what extent variables (DV & IV) capture desired constructs
Inadequate Explications
Clear explanations of treatments’ ingredients +
Confronting Fear overlapping variables -
IVs credible measures +
Mono-Operation Bias
Outcome measures +
Mono-Method Bias
Hamilton Anx . clinician administered +
5/6 self-report -
Reactivity to Exp. Conditions
Volunteers sought fro treat - Purpose clear
Hawthorne effect -
Increased symptoms at intake ?
Placebo effect - ?
Experimenter Expectancies
Favored Psychodynamic -

30. Threats to External Validity
Can we generalize observed relations across persons, settings and times
1. Person-Units
2. Treatments
3. Outcome Measures
4. Settings

31. Threats to External Validity
Can we generalize observed relations across persons, settings and times
1. Person-Units
Mostly Females 81% -& partnered 79% -
Comorbidity other anx. & dep ect 72% -+
Many exclusions p
Random sample not drawn from population -
Ethnicity mot noted ? -
Social class participants ?
2. Treatments
Clinicians tend to not use manuals -
Lack of trng. , sup. & monitoring in field -
Well known treatments +
3. Outcome Measures
Measures have shown good external validity
Two types of meas.- self-report and clin ad +
Behaviors, report s sign. others not assessed -
Recovery rates/clinical significance not meas -
4. Settings
Not much information
Was therapy conducted in probate practice offices? -

32. Threats to External Validity
Can we generalize observed relations across persons, settings and times
1. Person-Units
2. Treatments
3. Outcome Measures
4. Settings

33. Instruments Description of Measure Validity Estimates
Instrument name
Convergent/Discriminate Validity
Acronym
Validation Sample(s)
Authors
Key References
Reliability Estimates
Brief description of construct(s)
Chronbach’s alpha coefficient
Type of measure (e.g self-report)
Previous and Current studies
Number of items
Test/Re-Test
Example of items
Items response options
Factors or subscales
Scoring options and direction