Reliability & Validity the Bada & Bing of YOUR tailored survey design

This presentation has been influenced  not at all  a little bit  CONSIDERABLY by the work & wisdom of Dan Koretz. Thanks! Pre-Presentation credit

Core concept of validity You wish to measure a construct, but can never know true score for sure (e.g. 6th grade math proficiency, self esteem) You must draw an inference about the construct based on a sample or indicator of behavior-- something you can actually “touch” (e.g., 6th grade math test, self esteem survey) VALIDITY describes how well performance on your indicator justifies your inference about the construct

Error: Validity’s arch nemesis Sampling error: occurs from sampling units of observation (i.e, populations of humans) Measurement error (M.E.): occurs across instances of measurement from each unit (i.e. individual humans)

More on measurement error Outcome i = True score + M.E. Measurement error can be… systematic (does not “wash out” across repeated measurements) or random (does “wash out” across repeated measurements)

Systematic error Chris repeatedly takes a self esteem survey written in Latin Outcome repeatedly affected by factor not relevant to construct being measured “Construct irrelevant variance” **Note: the only Latin that Chris speaks involves inesway**

Systematic error cont’d “Construct Under-representation” If construct is poorly represented, repeated measurements will not converge on true score with regard to entire construct e.g. global self esteem survey asks only about Chris’ confidence on golf course

Random error Chris repeatedly takes self esteem survey Sometimes mood = Sometimes mood = Over time, outcomes will converge on Chris’ true score

Some details Validity is an attribute of your inference, not the instrument itself, which may be more valid for some inferences and/or populations than others (e.g. self-esteem survey in Latin). Validity is not an all or nothing phenomenon, but a matter of degree, and we must piece together evidence suggesting how valid our inferences may or may not be.

Types of validity/validity evidence note: other terms exist, but this will be the focus of S-015

Types of validity/validity evidence Construct Validity: How well does performance on our instrument justify inferences about the construct? “Validity” What we’re ultimately shooting for

Types of validity/validity evidence Content based evidence (a/k/a content validation study): Compare your instrument to your very thoroughly defined construct… Does the instrument adequately represent the construct? Harder than it seems (constructs can be messy)

Types of validity/validity evidence Convergent-discriminant evidence: Measures of similar constructs should converge. Measures of less similar constructs should diverge. (e.g. Two math tests should correlate more strongly than a math and reading test)

Multitrait-Multimethod Matrix (MTMM) A fun way to display convergent- discriminant validity (or not) Pass out hand outs: Now

But alas, complications abound… What constitutes “similar” What constitutes “less similar” What constitutes “convergence” What constitutes “divergence” ????

Plausible toy correlations Math 1 Math Math 2.82 Read 1.74 Read 2.70

Closing thoughts on validity We must piece together evidence that is often murky and incomplete to reach judgment. An instrument that is fairly valid for one use, inference, or population may not be valid for others.

Oh, BTW… The more reliable your instrument, the better your chance of drawing fairly valid inferences. (Old Faithful) 

Core concept of reliability Reliability is consistency of results across repeated measurements (e.g. assuming no interventions or natural attitudinal shifts in between, a subject taking a highly reliable survey would perform quite similarly each time s/he took it.)

Some details Reliability is also a matter of degree, often expressed as a coefficient ranging from A test or survey may be more reliable for some populations than others (e.g. surveys tend to be more reliable among older/more educated populations.)

POP QUIZ True or false… 1.) An instrument that allows us to draw a reasonably valid inference must be reasonably reliable 2.) A reasonably reliable instrument must allow us to draw a reasonably valid inference

POP QUIZ cont’d Regarding R & V, how might one describe…

A few (of many) ways to assess reliability Assess internal consistency Assuming a survey taps one and only one construct, the results from the first half should correlate highly with results from the second half; the odd items should correlate highly with the evens, etc. (split-half correlations)

Coefficient Alpha a/k/a Cronbach’s alpha The average of all possible split-half correlations in a given sample generally preferred to single split-half correlation

A few (of many) ways to assess reliability Test-retest Assuming no interventions or natural shifts in attitude, a reasonably reliable survey will yield similar results from the same person across repeated administrations

WARNING A test or survey with a high reliability coefficient does not guarantee that your results will be highly reliable. (e.g. Differences in administrative conditions can effect the consistency of your results across repeated administrations.

Questions???

References, etc Linn, R.L. & Gronlund, N. E. (2000). Measurement and Assessment in Teaching, 8 th Ed. New Jersey: Prentice-Hall, Inc. See diagram displayed on page 75 of the reference textbook. A very cool link that covers a TON of stuff on all forms of social research…