MEASUREMENT: VALIDITY Lu Ann Aday, Ph.D. The University of Texas School of Public Health

VALIDITY: Definition Extent of systematic departure in answers to questions in relationship to the meaning of the concept itself (content), equivalent measures of the same concept (criterion), or hypothesized associations with measures of similar or different concepts (construct).

VALIDITY: Types Content validity Criterion validity Construct validity

VALIDITY: Computation Requires analyzing the association between survey measures and documented definitions of the underlying concept (content); an equivalent “gold standard” of the survey measures (criterion); or theoretical associations with other abstract concepts (construct).

VALIDITY: Content Definition: extent to which the survey measures adequately represent the concept of interest

VALIDITY: Content Factors affecting: Failure to thoroughly conduct background research on the definition & measurement of the study concept Failure to clearly define the study concept

VALIDITY: Content Computation: 1. Identify domain or universe of interest through a thorough review of the literature. 2. Select or construct questions based on the review. 3. Ask subject matter experts to review the items you have selected or constructed.

VALIDITY: Content Validity criteria: 1. Is there at least one item to represent the concept? 2. Are there enough items? 3. Are the items well accepted, based on the literature & expert opinion? 4. Are they reliable?

VALIDITY: Criterion Definition: extent to which the survey measure predicts (predictive) or agrees with (concurrent) a criterion indicator of the concept of interest

VALIDITY: Criterion Factors affecting: Underreporting due to problems with recalling non- salient or frequently occurring events Overreporting due to problems with telescoping salient events

VALIDITY: Criterion- Predictive: Correlation Analysis Computation: Compute correlation coefficient between answers to survey question and future criterion source value for same concept : RespondentSurvey QuestionMD Observation 1ADLs=3 ADLs=3 2ADLs=2 ADLs=2 3ADLs=1ADLs=1 4ADLs=0ADLs=3 5ADLs=0 ADLs=0

VALIDITY: Criterion- Concurrent: Correlation Analysis Computation: Compute correlation coefficient between answers to survey question and current criterion source value for same concept: RespondentSurvey QuestionClinic Record 1BP=140/90 BP=140/90 2BP=150/80 BP=150/80 3BP=145/95BP=145/95 4BP=145/95BP=120/80 5BP=140/90 BP=140/90

VALIDITY: Criterion- Concurrent: Sensitivity-Specificity Analysis CRITERION / SURVEY IS (+) Hypertensive IS NOT (-) Hypertensive IS (+) Hypertensive a = true + Sensitivity = a/a + c b = false + IS NOT (-) Hypertensive c = false -d = true – Specificity = d/b + d

VALIDITY: Criterion- Concurrent: Sensitivity-Specificity Analysis CRITERION / SURVEY IS (+) Hypertensive IS NOT (-) Hypertensive IS (+) Hypertensive a = 130 Sensitivity = 130/180= 72.2% b = 1,000 IS NOT (-) Hypertensive c = 50d = 64,000 Specificity = 64,000/65,000=98.5%

VALIDITY: Construct Definition: extent to which relationships between measures agree with relationships predicted by theories or hypotheses Convergent – Ho: Related Discriminant – Ho: Not Related

VALIDITY: Construct Factors affecting: Inadequate conceptualization or measurement of study constructs Inadequate theory to guide specification of hypothesized relationships

VALIDITY: Construct: Correlation Analysis Computation: Compute correlation coefficient between survey question or scales measuring construct and related (or unrelated) constructs : RespondentPhysicalMentalOverall Fair 1GoodFairGood Fair 2GoodFairGood Fair 3GoodFairGood 4Fair 4GoodFairGood Fair 5GoodFairGood

VALIDITY: Construct- Known Groups Validity Analysis ( McHorney, et al., 1993, Table 3, p. 255 ) COMPARISON GROUPS/ MOS-SF 36 SUMMARY SCALES MINOR MEDICAL SERIOUS MEDICAL PSYCHIATRIC ONLY PSYCHIATRIC & SERIOUS MEDICAL Physical Functioning Mental Health

VALIDITY: CONSTRUCT- Factor Analysis (McHorney, et al., 1993, Table 1, p. 251) FACTOR LOADINGS (HYPOTHESES)/ SUBSCALES PhysicalMental Physical functioning0.88 (+)0.04 Role-physical0.78 (+)0.30 Bodily pain0.77 (+)0.24 Mental health (+) Role-emotional (+) Social Functioning0.44 (~)0.71 (+) Vitality0.59 (~)0.57 (~) General health perceptions0.68 (~)0.32 (~)

VALIDITY: Construct— Multi-Trait, Multi-Method Trait X Method ABCABCABC A (.90) B (.85) C (.80) A.75(.80) B.55(.75) C.35(.80) A.80.85(.95) B.55 (.90) C.30 (.65)

VALIDITY: Effect of Reliability Theoretical Variable can be no more valid than it is reliable. Empirical Rt = theoretical correlation between study & criterion variables (unobservable) Rx = reliability of study variable (computed) Ry = reliability of criterion variable (computed) Rt * sqrt [ Rx * Ry] = Max. achievable correlation 1.00 * sqrt [.60 *.80] =.69

WHEN TO UNDERTAKE VALIDITY ANALYSIS VALIDITY/ DIMENSIONS CONTENTCRITERIONCONSTRUCT QUESTIONSAll questionsFactual questions for which “gold standard” is available Summary scales of attitudes or other abstract concepts STUDIESAll types of surveys Esp. used in behavioral surveys Esp. used in attitudinal surveys STAGESQuestion development Pretest or final study

REFERENCES  DeVellis, Robert F. (2003). Scale Development: Theory and Applications. Second Edition. Thousand Oaks, CA: Sage. Gandek, B., & Ware, J.E., Jr., for the IQOLA Project (1998). Methods for validating and norming translations of health status questionnaires: The IQOLA Project Approach. J. Clinical Epidemiology, 51 (11), McHorney, C.A., Ware, J.E., Jr., & Raczek, A.E. (1993). The MOS 36-item short-form health survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), [Note: Tables 1 and 3 adapted and reprinted with the permission of Lippincott, Williams & Wilkins, Publishers.]