Reliability & Validity  Reliability  “dependability”  is the indicator consistent?  same result every time?  Does not necessary measure what you think it measures (ex. May consistently measure something other than the concept)  Validity  measurement validity - how well the conceptual and operational definitions mesh with each other  does measurement tool measure what we think ?

Relationship between Measurement Reliability and Validity  reliability necessary for validity but does not guarantee it ( “necessary but not sufficient”)  measure can be reliable but invalid Source: Neuman (2000: 171)

Examples of Types of Reliability n stability n over time n representative n across different subgroups of a population n (ex. of problems: young people may exaggerate their ages, older people may reduce theirs) n equivalence u Comparable results from multiple indicators u intercoder reliability

Improving Reliability n clearly conceptualize constructs n increase level of measurement n use pretests, pilot studies n e.g. use multiple indicators : Dependent Variable Measure Independent Variable Measure Empirical Association? a2a3a1b1b2 AB Specific Indicators Measurement Using Multiple Indicators Neuman (2000: 167)

Selected Types of Validity: Face Validity n judgement by the scientific community that indicator measures the concept Construct/concept Measure Scientific Community

Content Validity n measure represents all the aspects of conceptual definition of construct. n judgement concerning how adequately a test samples behavior representative of the universe of behavior the test was designed to sample. Love Beliefs & Values

n Criterion u The validity of indicator is verified by comparing it with another measure of the same construct n Predictive u Relies on occurrence of future event or behavior for verification of indicator n Concurrent u relies on pre-existing accepted measure to verify indicator n Construct Validity u A type of measurement validity that uses multiple indicators u convergent and discriminate Some types of Measurement Validity

Other Dimensions of Validity n Internal Validity n no error of logic internal to research design n External Validity n results can be generalized n Statistical validity n correct statistical methodology chosen ? n assumptions fully met

More terms & concepts for assessing validity (usually in experimental research) n Threats to Internal Validity u Selection bias u History effects (something in context changes) u Maturation u Testing effect u Instrumentation u Mortality u Statistical regression u Contamination u Compensatory behaviour u Experimenter expectanc n Threats to External Validity u Realism u Reactivity

Another Aspect of Measurement Design: Levels of Measurement n 1-Nominal (ex. Mother tongue) u different categories (names, labels, images) u not ranked n 2-Ordinal (county fair prizewinners ranked by first, second & third prize) u different categories u rank-ordered u attributes indicate relatively more or less of that variable u distance between the attributes of a variable is imprecise

Levels of Measurement (cont’d) n 3- Interval Measures (age measured by 5 year age groups) u different categories u ranked in order u Can tell amount of difference between categories u Usually no true zero n 4- Ratio Measures (age measured by date of birth)  different categories  ranked in order  amount of difference between categories  also possible to state proportion (have a true zero)  Relations between levels --can collapse from higher into lower, not vice versa

Babbie (1995: 101) The Research Process