6 Scales, Tests, & Indexes

6.1 Foundations of Scales, Tests, and Indexes Three approaches for measuring a construct: Scales Measure abstract concepts, like attitudes Tests Measure some ability or knowledge Indexes Combines several measures to estimate a more complex construct

6.2 Scales and Scaling Scaling Unidimensional scaling types Involves the construction of an instrument that associates qualitative constructs with quantitative metric units Unidimensional scaling types Thurstone or Equal-Appearing Interval Scaling Likert or Summative Scaling Guttman or Cumulative Scaling

6.2a General Issues in Scaling Figure 6.2 Scaling as the assignment of numbers according to a rule

6.2a Types of Scales Response scale Dichotomous response A sequential-numerical response format, such as a 1-to-5 rating format Dichotomous response Has two possible options (e.g., true/false or yes/no) Interval response scale Measured on an interval level, where the size of the interval is meaningful

6.2a Differences Between Scaling and Response Scales Table 6.1 Differences between scaling and response scales. Ask the class to identify an example of a construct being measured for each scale and response scale.

6.2b Purposes of Scaling To test a hypothesis To discover if a construct is unidimensional or multidimensional As part of exploratory research To represent a construct as a single score

6.2c Dimensionality Unidimensional Scales Two Dimensional Scales Figure 6.3 Unidimensional scales: For each construct in the figure (height, thirst, and self-esteem) the measurement is on a single dimension. Figure 6.4 A two dimensional scale: Here, a single construct is being measured in two dimensions (quantitative and verbal). Unidimensional Scales Two Dimensional Scales

6.2c Dimensionality Figure 6.5 A three dimensional scale. Semantic differential: A scaling method in which an object is assessed by the respondent on a set of bipolar adjective pairs.

6.2d Unidimensional or Multidimensional? Unidimensional scales are easier to use and understand Used when: What you are measuring is unidimensional in reality If the concept is not unidimensional, this scale will not measure the concept well, and you need to choose a multidimensional approach

6.2d Unidimensional or Multidimensional? Unidimensional types Thurstone scaling Likert scaling Guttman scaling Thurstone Scaling: A class of scaling methods (the method of equal appearing intervals, the method of successive intervals, and the method of paired comparisons) that were designed to yield unidimensional, intervallevel, multi-item scales. Likert or summative scaling: A method of scaling in which the items are assigned interval level scale values and the responses are gathered using an interval-level response format. Guttman or cumulative scaling: A method of scaling in which the items are assigned scale values that allow them to be placed in a cumulative ordering with respect to the construct being scaled.

Thurstone Scaling: The Method of Equal Appearing Intervals Develop the focus of the scaling project Generate potential scale items (statements); 80-100 separate items Rate the scale items Compute scale score values for each item Median, interquartile range Selecting the final scale items Administering the scale Median: The score found at the exact middle or 50th percentile of the set of values. One way to compute the median is to list all scores in numerical order and then locate the score in the center of the sample. interquartile range: The difference between the 75th (upper quartile) and 25th (lower quartile) percentile scores on the distribution of a variable. The interquartile range is an estimate of the spread or variability of the measure.

Displaying the Median and Interquartile Ranges of the Attitude Data Figure 6.7 Histogram displaying the median and interquartile ranges of the attitude data.

Likert Scaling Define the focus Generate the items on a 1-5 or 1-7 agree/disagree response scale Have a group of judges rate the items Select the items by computing the intercorrelations between all pairs of items Administer the scale Forced Choice Response Scale; Reversal Items Forced-choice response scale: A response scale that does not allow for a neutral or undecided value. By definition, a forced-choice response scale has an even number of response options. Reversal items: Items on a multi-item scale whose wording is in the opposite direction of the construct of interest. Reversal items must have their scores reversed prior to computing total scale scores.

The Employment Self-Esteem Likert Scale Table 6.2: The Rosenberg scale.

Guttman Scaling (Scalogram Analysis) Define the focus Develop the items (80-100) Have a group of judges rate the items Develop the cumulative scale Administer the scale Scalogram analysis: A method of analysis of a set of scale items used when constructing a Guttman or cumulative scale. In scalogram analysis, one attempts to determine the degree to which responses to the set of items allows the items to be ordered cumulatively in one dimension with respect to the construct of interest.

Developing a Cumulative Scale with Guttman Scaling Figure 6.9 Developing a cumulative scale with Guttman scaling.

6.3 Tests A test is a measure designed to measure a respondent’s knowledge, skill, or performance The history of testing goes back to 2200 BCE The key for a good test is reliability and validity

Example: A Driving Simulator Figure 6.9 This driving simulator was used as part of a driver-training study for adolescents with attention deficit problems (Fabiano et al., 2011). Courtesy of Kevin F. Hulme Tests: A measure designed to measure a respondent’s knowledge, skill, or performance.

6.3a Validity, Reliability, and Test Construction Factor analysis Consequential validity Test-retest reliability Item analysis Figure 6.10 What validity issue do you see here? Factor analysis: A multivariate statistical analysis that uses observed correlations (variability) as input and identifies a fewer number of unobservedvariables, known as factors, that describe the original data more efficiently. Consequential validity: The approximate truth or falsity of assertions regarding the intended or unintended consequences of test interpretation and use. Test-retest reliability: A method of estimating the reliability or consistency of a test or measure by assessing the degree of correlation between two successive administrations. Item Analysis: The correlation of a particular item with the total subtest or test score.

6.3b Standardized Tests A method of test construction that uses statistics and a large sample of previously taken tests to “standardize” the measurement Includes statistics such as: Mean, median, and mode Percentiles Variance and standard deviation Correlations with other, related tests

6.3c Test Fairness Many factors affect standardized test performance Consequential validity Many decision makers (e.g., universities) have begun to either do away with standardized test scores or to combine single test scores with other measures of ability Does your SAT score fairly reflect your intelligence?

6.3d How to Find A Good Test Test publishers Primary research literature Test reviews in academic journals Buros Center for Testing at the University of Iowa APA

6.4 Indexes Index A quantitative score that measures a construct of interest by applying a formula or a set of rules that combines relevant data

6.4a Some Common Indexes Consumer Price Index (CPI) Socioeconomic Status (SES) Duncan Socioeconomic Index (SEI)

6.4b Constructing an Index Conceptualize the index Operationalize and measure the components Develop the rules for calculating the index score Weighted Index Weighted Index: A quantitative score that measures a construct of interest by applying a formula or a set of rules that combines relevant data where the data components are weighted differently.

Discuss and Debate I Consider standardized tests Are they fair? What are some factors that may affect a person’s performance?

Discuss and Debate II Have you ever answered questions on a scale in a way that would make you look good? Some measurements may not be accurate when respondents do not answer truthfully – what can be done? In research, this is called the social-desirability bias. Allow the class to discuss the ways in which that bias can skew the results of a study.