2. Scaling, Reliability and Validity
Chapter 9
Scaling, Reliability and Validity

3. Chapter Objectives
Know how and when to use the different forms of rating scales and ranking scales
Explain stability and consistency, and how they are established
Explain the different forms of validity
Discuss what ‘goodness’ of measures means, and why it is necessary to establish it in research

4. Rating and Ranking Scales
Rating Scales
have several response categories and are used to elicit responses with regard to the object, event or person studied.
Ranking Scales
make comparisons between or among objects, events or persons, and elicit the preferred choices and ranking among them.

5. Rating Scales dichotomous scale category scale Likert scale
numerical scales
semantic differential scale
itemised rating scale
fixed or constant sum rating scale
Stapel scale
graphic rating scale
consensus scale

6. Dichotomous Scale Used to elicit a Yes or No answer, eg:
Do you own a car? Yes No

7. Category Scales

8. Likert Scale
Indicate the extent to which you agree or disagree with the following statements:
My work is very interesting
Life without my work would be dull

9. Semantic Differential Scale
Bad
Good
Unresponsive
Responsive
Timid
Courageous

10. Numerical Scale How pleased are you with your new car?
Extremely Extremely
pleased displeased

11. Itemised Rating Scale
This is an unbalanced rating scale which does not have a neutral point.

12. Fixed or Constant Sum Rating Scale
Respondents are asked to distribute a given number of points across various items, eg:
Fragrance —
Colour —
Shape —
Size —
Texture of lather —
Total points

13. Stapel Scale
Measures the direction and intensity of the attitude towards the items under study, eg

14. Graphic Rating Scale

15. Ranking Scales
paired comparison
forced choice
comparative scale

16. Paired Comparison
Used when, among a small number of objects, respondents are asked to choose between two objects at a time.
The paired choices for n objects will be ((n) (n-1)/2).

17. Forced Choice
Rank your preferences among the following magazines, 1 being your preferred choice and 5 being your least preferred:
Australian Financial Review __
Business Review Weekly __
Playboy __
The Economist __
Time __

18. Comparative Scale
In a volatile financial environment, compared with shares, how useful is it to invest in government bonds?
More useful About the same Less useful

19. Goodness of Measures Reliability measures Validity measures
How stable and consistent is the measuring instrument?
Validity measures
Are we measuring the right thing?

20. Reliability and Validity in Target Shooting

21. Forms of Reliability and Validity

22. Reliability Stability Internal consistency
refers to the ability of a measure to maintain stability over time, despite uncontrollable testing conditions or the state of the respondents themselves
Internal consistency
indicates how well the items ‘hang together as a set’ and can independently measure the same concept, so respondents attach the same overall meaning to each of the items

23. Stability of Measures Test-retest reliability
the reliability coefficient obtained with a repetition of the same measure on a second occasion
Parallel-form reliability
the correlation obtained from responses on two comparable sets of measures (changed for wording & question order) tapping the same construct

24. Internal Consistency of Measures
Inter-item consistency reliability
test of the consistency of respondents’ answers to all the items in a measure
usually tested by Cronbach’s coefficient alpha
Split-half reliability
reflects the correlations between two halves of an instrument

25. Types of Validity