Other Basic Considerations in Designing Measures “Not everything that counts can be measured, and not everything that can be measured counts”, Albert Einstein. “Not everything that counts can be measured, and not everything that can be measured counts”, Albert Einstein. n Levels of Measurement u Continuous vs. discrete variables (implications for analysis of findings) n Scales and Indices

Levels of Measurement n 1-Nominal (ex. Mother tongue) u different categories (names, labels, images) u not ranked n 2-Ordinal (county fair prizewinners ranked by first, second & third prize) u different categories u rank-ordered u attributes indicate relatively more or less of that variable u distance between the attributes of a variable is imprecise

Levels of Measurement (cont’d) n 3- Interval Measures (ex. age measured by 5 year age groups) u different categories u ranked in order u Can tell amount of difference between categories u Usually no true zero n 4- Ratio Measures (age measured by date of birth)  different categories  ranked in order  amount of difference between categories  also possible to state proportion (have a true zero)  Relations between levels --can collapse from higher into lower, not vice versa

Scales & Indices n Often used interchangeably u Scale F Intensity or degree of variable or construct often nominal or ordinaloften nominal or ordinal F Different types of common scaling techniques u Index F Combines multiple indicators into single score F Often interval or ratio level

Measurement principles for scales and indices n Mutually exclusive and exhaustive attributes n Unidimensionality (measure a single construct)

What are composite measures? n Composite measures are instruments that use several questions to measure a given variable (construct). n A composite measure can be either unidimensional or multidimensional. n Indexes and scales are two types of composite measures.

Logic of Index Construction actions combined in single measure, usually ordinal level of measurement

Logic of Scales actions ranked

Technical Issues in Index Construction n weighting u e.g. Quality of life index n treatment of missing data n standardization

Standardization example actually about weighting

Treatment of Missing Data n eliminate cases with missing data? n substitute average score ? n Guess ? n insert random value ?

Rates and Standardization n makes comparison possible u e.g. rate of mobile phone owners per class F # of owners/size of class n Issues: deciding what measure to use for reference populations u e.g. employment rates

Scales: ex. Likert Scale n ordinal level measure n Debates about inclusion of neutral category (!?) n switching directions of some questions (to avoid response sets) n use to create index (scoring issues) n Ex. has positively transformed communication between people (p. 179)

examples

Thurston Scale n uses judges to pick statements n tests agreement/disagreement n comparative measurement, not often used today

Semantic Differential Scale n mark spot on continuum between paired opposites n difficult to analyze results Fig. 1. Modern Japanese version of the Semantic Differential. The Kanji characters in background stand for "God" and "Wind" respectively, with the compound reading "Kamikaze".

Guttman Scaling n hierarchical relationship established by response to indicators n e.g. Attitudes to abortion, attitudes to alcohol, cigarettes and drugs n Link to on-line description u ods.net/kb/scalgutt.htm ods.net/kb/scalgutt.htm ods.net/kb/scalgutt.htm u See also Buddenbaum & Novak Applied Communication Research u 8&lpg=PA98&dq=guttman+scale+calculate+score&source=w eb&ots=dJfvTHPJDZ&sig=XIydBYJOmsxptjCmVbhV__Els b8&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PP A100,M1 8&lpg=PA98&dq=guttman+scale+calculate+score&source=w eb&ots=dJfvTHPJDZ&sig=XIydBYJOmsxptjCmVbhV__Els b8&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PP A100,M1 8&lpg=PA98&dq=guttman+scale+calculate+score&source=w eb&ots=dJfvTHPJDZ&sig=XIydBYJOmsxptjCmVbhV__Els b8&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PP A100,M1 Develop the Cumulative Scale. The key to Guttman scaling is in the analysis. We construct a matrix or table that shows the responses of all the respondents on all of the items. We then sort this matrix so that respondents who agree with more statements are listed at the top and those agreeing with fewer

Perfect Guttman Scale

Bogardus Social Distance Scale (type of Guttman scale) n ordered statements expressing closeness to respondent n must be refer to specific context n problems comparing behaviour and feelings

example

Indexes (Indices) n combine responses into single score u e.g. quality of life n Weighting Index Items u importance given to components of index u unweighted index

The Macleans Index n STUDENT BODY 5 items n CLASSES 3 items n FACULTY 4 items n FINANCES 3 items n LIBRARY 3 items n REPUTATION 2 items Based on twenty discrete rankings of six attributes of universities:

Operational Definition Student Body (21%) average grades of incoming students 12 proportion of incoming students with 75% or more 3 proportion of incoming students from out of province 1 graduation rates (% full-time undergrads in second year who graduate) national academic awards won by students 2 3

Operational Definition Classes (18%) 7.5 class sizes at 3rd and 4th year levels 7.5 class sizes at 1st and 2nd year levels (1-25, 26-50, , , , 501 plus) % 1st year courses taught by tenured or tenure-track professors 3

Operational Definition Faculty (17%) 3 success of eligible faculty in winning federal grants (arts and social sciences) 5.5 % with Ph.D’s success of eligible faculty in winning federal grants (medicine and sciences) 5.5 national academic awards won by profs 3

Weighting of Macleans Index n STUDENT BODY 21% n CLASSES18% n FACULTY 17% n FINANCES12% n LIBRARY12% n REPUTATION20%

Critique of Maclean’s Survey as Measurement tool n What are the operational measures? n Which measures (or parts of the “composite measure”) are more important? n What is the concept being measured?

Advantage of Scales & Indices n Easy to use & administer n Inexpensive n Time-efficient (fast for respondent & researcher) n Easy to calculate scores

Disadvantages of Scales n Assumption that we know what is important n Tend to be very broad n Predetermined questions & problems may not be suited or cover full meanings n Response set effect (answering the same way for all questions) u But there are techniques for counteracting response set bias

Specialized techniques for Index construction n factor analysis n q-sort n cluster analysis