1. Measurement and Scaling
Farzin Madjidi, Ed.D.
Pepperdine University
Graduate School of
Education and Psychology

2. Variables Independent Dependent Extraneous
Precedes, influences or predicts results
Dependent
Affected by or predicted by the Independent Variable
Extraneous
Affected by the D.V., but not controlled or measured. Causes error

3. Variables Confounding Intervening
An extraneous variable that varies systematically (has a relationship) with the I.V.
Intervening
Unobservable trait that influences behavior (e.g., effect of new intervention on self-esteem may be affected by the motivation level of subjects)

4. Variables Control Organismic
Used to eliminate the effect of extraneous variables
Organismic
Aka, measured, or assigned
Characteristics of the subjects that cannot be manipulated

5. Levels of Measurements
Four levels of Measurements
Nominal
Measures categories
Ordinal
Categories + rank and order
Interval
Equal distance between any two consecutive measures
Ratio
Intervals + meaningful zeros

6. Categories of Scales Categorical (ratings) Comparative (ranking)
Score without comparison - 1 to 5 scales
Comparative (ranking)
Score by comparing - Smartest
Preference
Subjective - which do you prefer
Non-preference
Objective - which solution is less costly

7. Categories of Scales Unidimensional Multi-dimensional
Involves only one aspect of the measurement
Measurement by one construct
Multi-dimensional
Involves several aspects of a measurement
Uses several dimensions to measure a single construct

8. Types of Scales Likert/Summated Rating Scales
Semantic Differential Scales
Magnitude Scaling
Thruston Scales
Guttman Scales

9. Likert Scales A very popular rating scale
Measures the feelings/degree of agreement of the respondents
Ideally, 4 to 7 points
Examples of 5-point surveys
Agreement SD D ND/NA A SA
Satisfaction SD D ND/NS S SS
Quality VP P Average G VG

10. Summative Ratings
A number of items collectively measure one construct (Job Satisfaction)
A number of items collectively measure a dimension of a construct and a collection of dimensions will measure the construct (Self-esteem)

11. Summative Likert Scales
Must contain multiple items
Each individual item must measure something that has an underlying, quantitative measurement continuum
There can be no right/wrong answers as opposed to multiple-choice questions
Items must be statements to which the respondent assigns a rating
Cannot be used to measure knowledge or ability, but familiarity

12. Semantic Differential Scales
Uses a set of scale anchored by their extreme responses using words of opposite meaning.
Example:
Dark ___ ___ ___ ___ ___ Light
Short ___ ___ ___ ___ ___ Tall
Evil ___ ___ ___ ___ ___ Good
Four to seven categories are ideal

13. Magnitude Scaling
Attempts to measure constructs along a numerical, ratio level scale
Respondent is given an item with a pre-assigned numerical value attached to it to establish a “norm”
The respondent is asked to rate other items with numerical values as a proportion of the “norm”
Very powerful if reliability is established

14. Thurston Scales Thurston Scales Items are formed
Panel of experts assigns values from 1 to 11 to each item
Mean or median scores are calculated for each item
Select statements evenly spread across the scale

15. Thurston Scales Example:
Please check the item that best describes your level of willingness to try new tasks
I seldom feel willing to take on new tasks (1.7)
I will occasionally try new tasks (3.6)
I look forward to new tasks (6.9)
I am excited to try new tasks (9.8)

16. Guttman Scales Also known as Scalograms
Both the respondents and items are ranked
Cutting points are determined (Goodenough-Edwards technique)
Coefficient of Reproducibility (CReg) - a measure of goodness of fit between the observed and predicted ideal response patterns
Keep items with CReg of 0.90 or higher

17. Scale Construction Define Constructs
Conceptual/theoretical basis from the literature
Are their sub-scales (dimensions) to the scale
Multiple item sub-scales
Principle of Parsimony
Simplest explanation among a number of equally valid explanations must be used

18. Item Construction Agreement items Write declarative statements
Death penalty should be abolished
I like to listen to classical music
Frequency items (how often)
I like to read
Evaluation items
How well did your team play
How well does the police serve your community

19. Item Writing Mutually exclusive and collectively exhaustive items
Use positively and negatively phrased questions
Avoid colloquialism, expressions and jargon
Avoid the use of negatives to reverse the wording of an item
Don’t use: I am not satisfied with my job
Use: I hate my job!
Be brief, focused, and clear
Use simple, unbiased questions

20. Sources of Error Social desirability Response sets Acquiescence
Giving politically correct answers
Response sets
All yes, or all no responses
Acquiescence
Telling you what you want to hear
Personal bias
Wants to send a message

21. Sources of Error Response order Item order
Recency - Respondent stops reading once s/he gets to the response s/he likes
Primacy - Remember better the initial choices
Fatigue
Item order
Answers to later items may be affected by earlier items (simple, factual items first)
Respondent may not know how to answer earlier questions

22. Assessing Instruments
Three issues to consider
Validity: Does the instrument measure what its supposed to measure
Reliability: Does it consistently repeat the same measurement
Practicality: Is this a practical instrument

23. Types of Validity Face validity Content validity
Does the instrument, on its face, appear to measure what it is supposed to measure
Content validity
Degree to which the content of the items adequately represent the universe of all relevant items under study
Generally arrived at through a panel of experts

24. Types of Validity Criterion related
Degree to which the predictor is adequate in capturing the relevant aspects of criterion
Uses Correlation analysis
Concurrent validity
Criterion data is available at the same time as predictor score- requires high correlation between the two
Predictive validity
Criterion is measured after the passage of time
Retrospective look at the validity of the measurement
Known-groups

25. Types of Validity Construct Validity
Measures what accounts for the variance
Attempts to identify the underlying constructs
Techniques used:
Correlation of proposed test with other existing tests
Factor analysis
Multi-trait-multimethod analysis
Convergent validity - Calls for high correlation between the different measures of the same construct
Discriminant validity - Calls for low correlation between sub-scales within a construct

26. Types of Reliability Stability
Test-retest: Same test is administered twice to the same subjects over a short interval (3 weeks to 6 months)
Look for high correlation between the test and retest
Situational factors must be minimized

27. Types of Reliability Equivalence
Degree to which alternative forms of the same measure produce same or similar results
Give parallel forms of the same test to the same group with a short delay to avoid fatigue
Look for high correlation between the scores of the two forms of the test
Inter-rater reliability

28. Types of Reliability Internal Consistency
Degree to which instrument items are homogeneous and reflect the same underlying constructs
Split-half testing where the test is split into two halves that contain the same types of questions
Uses Cronbach’s alpha to determine internal consistency. Only one administration of the test is required
Kuder-Richardson (KR20) for items with right and wrong answers

29. Practicality Is the survey economical Convenience
Cost of producing and administering the survey
Time requirement
Common sense!
Convenience
Adequacy of instructions
Easy to administer
Can the measurement be interpreted by others
Scoring keys
Evidence of validity and reliability
Established norms