Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 11 Measurement and Data Quality

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Measurement The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rules Advantages: –Removes guesswork –Provides precise information –Less vague than words

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Levels of Measurement There are four levels (classes) of measurement: –Nominal (assigning numbers to classify characteristics into categories) –Ordinal (ranking objects based on their relative standing on an attribute) –Interval (objects ordered on a scale that has equal distances between points on the scale) –Ratio (equal distances between score units; there is a rational, meaningful zero.) A variable’s level of measurement determines what mathematic operations can be performed in a statistical analysis.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Errors of Measurement Obtained score = true score ± error –Obtained score: an actual data value for a participant (e.g., anxiety scale score) –True score: the score that would be obtained with an infallible measure –Error: the error of measurement, caused by factors that distort measurement

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Factors That Contribute to Errors of Measurement Situational contaminants Transitory personal factors (e.g., fatigue) Response set biases Item sampling

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Question Is the following statement True or False? The true score is data obtained from the actual research study.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Answer False Rationale: The true score is the score that would be obtained with an infallible measure. The obtained score is an actual value (datum) for a participant.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability The consistency and accuracy with which an instrument measures the target attribute Reliability assessments involve computing a reliability coefficient. –Reliability coefficients can range from.00 to –Coefficients below.70 are considered unsatisfactory. –Coefficients of.80 or higher are desirable.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Three Aspects of Reliability Can Be Evaluated Stability Internal consistency Equivalence

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Stability The extent to which scores are similar on two separate administrations of an instrument Evaluated by test–retest reliability –Requires participants to complete the same instrument on two occasions –Appropriate for relatively enduring attributes (e.g., creativity)

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Internal Consistency The extent to which all the items on an instrument are measuring the same attribute Evaluated by administering instrument on one occasion Appropriate for most multi-item instruments The most widely used approach to assessing reliability Assessed by computing coefficient alpha (Cronbach’s alpha) Alphas ≥.80 are highly desirable.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Question When determining the reliability of a measurement tool, which value would indicate that the tool is most reliable? –A –B –C –D. 1.10

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Answer C Rationale: Reliability coefficients can range from 0.0 to Coefficients of 0.80 or higher are desirable. Thus, a coefficient of 0.90 would indicate that the tool is very reliable. A value greater than 1.00 for a coefficient would be an error.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Equivalence The degree of similarity between alternative forms of an instrument or between multiple raters/observers using an instrument Most relevant for structured observations Assessed by comparing agreement between observations or ratings of two or more observers (interobserver/interrater reliability)

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability Principles Low reliability can undermine adequate testing of hypotheses. Reliability estimates vary depending on procedure used to obtain them. Reliability is lower in homogeneous than in heterogeneous samples. Reliability is lower in shorter than in longer multi-item scales.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Validity The degree to which an instrument measures what it is supposed to measure Four aspects of validity: –Face validity –Content validity –Criterion-related validity –Construct validity

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Face Validity Refers to whether the instrument looks as though it is an appropriate measure of the construct Based on judgment; no objective criteria for assessment

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Content Validity The degree to which an instrument has an adequate sample of items for the construct being measured Evaluated by expert evaluation, often via a quantitative measure—the content validity index (CVI)

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Question Is the following statement True or False? Face validity of an instrument is based on judgment.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Answer True Rationale: Face validity refers to whether the instrument looks like it is an appropriate measure of the construct. There are no objective criteria for assessment; it is based on judgment.

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Criterion-Related Validity The degree to which the instrument is related to an external criterion Validity coefficient is calculated by analyzing the relationship between scores on the instrument and the criterion. Two types: –Predictive validity: the instrument’s ability to distinguish people whose performance differs on a future criterion –Concurrent validity: the instrument’s ability to distinguish individuals who differ on a present criterion

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Construct Validity Concerned with these questions: –What is this instrument really measuring? –Does it adequately measure the construct of interest?

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Some Methods of Assessing Construct Validity Known-groups technique Testing relationships based on theoretical predictions Factor analysis

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Sensitivity and Specificity Sensitivity: the instrument’s ability to correctly identify a “case”—i.e., to diagnose a condition Specificity: the instrument’s ability to correctly identify noncases, that is, to screen out those without the condition

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins End of Presentation