Other Basic Considerations in Designing Measures “Not everything that counts can be measured, and not everything that can be measured counts”, Albert Einstein. “Not everything that counts can be measured, and not everything that can be measured counts”, Albert Einstein. n Levels of Measurement u Continuous vs. discrete variables (implications for analysis of findings) n Scales and Indices

Levels of Measurement n 1-Nominal (ex. Mother tongue) u different categories (names, labels, images) u not ranked n 2-Ordinal (county fair prizewinners ranked by first, second & third prize) u different categories u rank-ordered u attributes indicate relatively more or less of that variable u distance between the attributes of a variable is imprecise

Levels of Measurement (cont’d) n 3- Interval Measures (ex. age measured by 5 year age groups) u different categories u ranked in order u Can tell amount of difference between categories u Usually no true zero n 4- Ratio Measures (age measured by date of birth)  different categories  ranked in order  amount of difference between categories  also possible to state proportion (have a true zero)  Relations between levels --can collapse from higher into lower, not vice versa

Scales & Indices n Often used interchangeably u Scale F Intensity or degree of variable or construct often nominal or ordinaloften nominal or ordinal F Different types of common scaling techniques u Index F Combines multiple indicators into single score F Often interval or ratio level

Mearuement principles for scales and indices n Mutually exclusive and exhaustive attributes n Unidimensionality (measure a single construct)

What are composite measures? n Composite measures are instruments that use several questions to measure a given variable (construct). n A composite measure can be either unidimensional or multidimensional. n Indexes and scales are two types of composite measures.

Logic of Index Construction actions combined in single measure, usually ordinal level of measurement

Logic of Scales actions ranked

Technical Issues in Index Construction n weighting u e.g. Quality of life index n treatment of missing data n standardization

Standardization example actually about weighting

Treatment of Missing Data n eliminate cases with missing data? n substitute average score ? n Guess ? n insert random value ?

Rates and Standardization n makes comparison possible u e.g. rate of mobile phone owners per class F # of owners/size of class n Issues: deciding what measure to use for reference populations u e.g. employment rates

Scales: ex. Likert Scale n ordinal level measure n Debates about inclusion of neutral category (!?) n switching directions of some questions (to avoid response sets) n use to create index (scoring issues) n Ex. has positively transformed communication between people (p. 179)

examples

Thurston Scale n uses judges to pick statements n tests agreement/disagreement n comparative measurement, not often used today

Bogardus Social Distance Scale n ordered statements expressing closeness to respondant n must be done for specific context n problems comparing behaviour and feelings

example

Semantic Differential Scale n mark spot on continuum between paired opposites n difficult to analyze results

Guttman Scaling n hierarchical relationship established by response to indicators n e.g. Attitudes to abortion, attitudes to alcohol, cigarettes and drugs

Perfect Guttman Scale

Indexes (Indices) n combine responses into single score u e.g. quality of life n Weighting Index Items u importance given to components of index u unweighted index

The Macleans Index n STUDENT BODY 5 items n CLASSES 3 items n FACULTY 4 items n FINANCES 3 items n LIBRARY 3 items n REPUTATION 2 items Based on twenty discrete rankings of six attributes of universities:

Operational Definition Student Body (21%) average grades of incoming students 12 proportion of incoming students with 75% or more 3 proportion of incoming students from out of province 1 graduation rates (% full-time undergrads in second year who graduate) national academic awards won by students 2 3

Operational Definition Classes (18%) 7.5 class sizes at 3rd and 4th year levels 7.5 class sizes at 1st and 2nd year levels (1-25, 26-50, , , , 501 plus) % 1st year courses taught by tenured or tenure-track professors 3

Operational Definition Faculty (17%) 3 success of eligible faculty in winning federal grants (arts and social sciences) 5.5 % with Ph.D’s success of eligible faculty in winning federal grants (medicine and sciences) 5.5 national academic awards won by profs 3

Weighting of Macleans Index n STUDENT BODY 21% n CLASSES18% n FACULTY 17% n FINANCES12% n LIBRARY12% n REPUTATION20%

Critique of Maclean’s Survey as Measurement tool n What are the operational measures? n Which measures (or parts of the “composite measure”) are more important? n What is the concept being measured?

Specialized techniques for Index construction n factor analysis n q-sort n cluster analysis