Measurement and Scaling Farzin Madjidi, Ed.D. Pepperdine University Graduate School of Education and Psychology

Variables Independent Dependent Extraneous Precedes, influences or predicts results Dependent Affected by or predicted by the Independent Variable Extraneous Affected by the D.V., but not controlled or measured. Causes error

Variables Confounding Intervening An extraneous variable that varies systematically (has a relationship) with the I.V. Intervening Unobservable trait that influences behavior (e.g., effect of new intervention on self-esteem may be affected by the motivation level of subjects)

Variables Control Organismic Used to eliminate the effect of extraneous variables Organismic Aka, measured, or assigned Characteristics of the subjects that cannot be manipulated

Levels of Measurements Four levels of Measurements Nominal Measures categories Ordinal Categories + rank and order Interval Equal distance between any two consecutive measures Ratio Intervals + meaningful zeros

Categories of Scales Categorical (ratings) Comparative (ranking) Score without comparison - 1 to 5 scales Comparative (ranking) Score by comparing - Smartest Preference Subjective - which do you prefer Non-preference Objective - which solution is less costly

Categories of Scales Unidimensional Multi-dimensional Involves only one aspect of the measurement Measurement by one construct Multi-dimensional Involves several aspects of a measurement Uses several dimensions to measure a single construct

Types of Scales Likert/Summated Rating Scales Semantic Differential Scales Magnitude Scaling Thruston Scales Guttman Scales

Likert Scales A very popular rating scale Measures the feelings/degree of agreement of the respondents Ideally, 4 to 7 points Examples of 5-point surveys Agreement SD D ND/NA A SA Satisfaction SD D ND/NS S SS Quality VP P Average G VG

Summative Ratings A number of items collectively measure one construct (Job Satisfaction) A number of items collectively measure a dimension of a construct and a collection of dimensions will measure the construct (Self-esteem)

Summative Likert Scales Must contain multiple items Each individual item must measure something that has an underlying, quantitative measurement continuum There can be no right/wrong answers as opposed to multiple-choice questions Items must be statements to which the respondent assigns a rating Cannot be used to measure knowledge or ability, but familiarity

Semantic Differential Scales Uses a set of scale anchored by their extreme responses using words of opposite meaning. Example: Dark ___ ___ ___ ___ ___ Light Short ___ ___ ___ ___ ___ Tall Evil ___ ___ ___ ___ ___ Good Four to seven categories are ideal

Magnitude Scaling Attempts to measure constructs along a numerical, ratio level scale Respondent is given an item with a pre-assigned numerical value attached to it to establish a “norm” The respondent is asked to rate other items with numerical values as a proportion of the “norm” Very powerful if reliability is established

Thurston Scales Thurston Scales Items are formed Panel of experts assigns values from 1 to 11 to each item Mean or median scores are calculated for each item Select statements evenly spread across the scale

Thurston Scales Example: Please check the item that best describes your level of willingness to try new tasks I seldom feel willing to take on new tasks (1.7) I will occasionally try new tasks (3.6) I look forward to new tasks (6.9) I am excited to try new tasks (9.8)

Guttman Scales Also known as Scalograms Both the respondents and items are ranked Cutting points are determined (Goodenough-Edwards technique) Coefficient of Reproducibility (CReg) - a measure of goodness of fit between the observed and predicted ideal response patterns Keep items with CReg of 0.90 or higher

Scale Construction Define Constructs Conceptual/theoretical basis from the literature Are their sub-scales (dimensions) to the scale Multiple item sub-scales Principle of Parsimony Simplest explanation among a number of equally valid explanations must be used

Item Construction Agreement items Write declarative statements Death penalty should be abolished I like to listen to classical music Frequency items (how often) I like to read Evaluation items How well did your team play How well does the police serve your community

Item Writing Mutually exclusive and collectively exhaustive items Use positively and negatively phrased questions Avoid colloquialism, expressions and jargon Avoid the use of negatives to reverse the wording of an item Don’t use: I am not satisfied with my job Use: I hate my job! Be brief, focused, and clear Use simple, unbiased questions

Sources of Error Social desirability Response sets Acquiescence Giving politically correct answers Response sets All yes, or all no responses Acquiescence Telling you what you want to hear Personal bias Wants to send a message

Sources of Error Response order Item order Recency - Respondent stops reading once s/he gets to the response s/he likes Primacy - Remember better the initial choices Fatigue Item order Answers to later items may be affected by earlier items (simple, factual items first) Respondent may not know how to answer earlier questions

Assessing Instruments Three issues to consider Validity: Does the instrument measure what its supposed to measure Reliability: Does it consistently repeat the same measurement Practicality: Is this a practical instrument

Types of Validity Face validity Content validity Does the instrument, on its face, appear to measure what it is supposed to measure Content validity Degree to which the content of the items adequately represent the universe of all relevant items under study Generally arrived at through a panel of experts

Types of Validity Criterion related Degree to which the predictor is adequate in capturing the relevant aspects of criterion Uses Correlation analysis Concurrent validity Criterion data is available at the same time as predictor score- requires high correlation between the two Predictive validity Criterion is measured after the passage of time Retrospective look at the validity of the measurement Known-groups

Types of Validity Construct Validity Measures what accounts for the variance Attempts to identify the underlying constructs Techniques used: Correlation of proposed test with other existing tests Factor analysis Multi-trait-multimethod analysis Convergent validity - Calls for high correlation between the different measures of the same construct Discriminant validity - Calls for low correlation between sub-scales within a construct

Types of Reliability Stability Test-retest: Same test is administered twice to the same subjects over a short interval (3 weeks to 6 months) Look for high correlation between the test and retest Situational factors must be minimized

Types of Reliability Equivalence Degree to which alternative forms of the same measure produce same or similar results Give parallel forms of the same test to the same group with a short delay to avoid fatigue Look for high correlation between the scores of the two forms of the test Inter-rater reliability

Types of Reliability Internal Consistency Degree to which instrument items are homogeneous and reflect the same underlying constructs Split-half testing where the test is split into two halves that contain the same types of questions Uses Cronbach’s alpha to determine internal consistency. Only one administration of the test is required Kuder-Richardson (KR20) for items with right and wrong answers

Practicality Is the survey economical Convenience Cost of producing and administering the survey Time requirement Common sense! Convenience Adequacy of instructions Easy to administer Can the measurement be interpreted by others Scoring keys Evidence of validity and reliability Established norms