1. Lecture 6 Structured Interviews and Instrument Design Part II:
Scales and Scaling
Lecture 6
Info 271B

2. Agenda Scales and Scaling
Brief Demonstration and Getting Started with STATA

3. Know your sample population
Regional language and terminology
Cultural differences
How you conduct survey can influence your valid sample
Door to door?
Registered telephone directory?
Internet-based survey?

4. Scales and Scaling
Scales are any device used to assign units of analysis to categories of a variable. Age is a scale, so is a categorical response like religion.

5. Single Indicator Scales
Assigns units of analysis to categories of a variable
Includes many common single-item questions (including nominal and metric variables)
“Do you feel happy in general?”
(answered on 0-10 point scale)
“What is your race?”
(answered on 6 categories)

6. Single-option questions online

7. Composite Measures Composite Measure Scale Cumulative index
(Q1 + Q2 + Q3 + Q4)
Cumulative scales

8. Other Types of Complex Scales
Criterion referenced index
Examples:
Grade on 100 point scale
Score on composite scale, regardless of distribution
Norm referenced
Grade on relative scale
Relative score (high/low) based on sample distribution
Why would you choose one or the other; under what conditions?

9. Common Scaling Techniques: Guttman Scaling
Guttman Scale (Q1  Q2  Q3)
Measurements for the item have a pattern indicating that the items measure a unidimensional variable.
Define a focal concept
Develop items for the scale
Rate the items (judges, small sub-sample)
Create cumulative scale
Randomize order and distribute scale

10. Common Scaling Techniques: Likert-style scales
Likert-style Scales [Q1, Q2, Q3]
Originally 5-point, but also includes other types
Procedure:
Decide on the concept of interest
Create list of indicator questions/statements
Decide on number of responses
Test items on small sample
Use a form of item analysis to check for a unidimensional scale

11. Item Analysis
Factor Analysis is most common approach, but also a fairly advanced procedure.
Basic principle of all item analysis is to see if the individual variables in your composite “hang together”
Score responses
Get the interitem correlation

12. Interitem Correlation

13. Cronbach’s Alpha Uses split-half reliability technique
Checking similarity of scores for various sets of questions

14. Finding items that do not discriminate
Simple way: look at the extremes and check to see which items they have in common.
Formal way: Item-total correlation

15. Relationship Structures questionnaire
It helps to turn to this person in times of need.
I usually discuss my problems and concerns with this person.
I talk things over with this person.
I find it easy to depend on this person.
I don't feel comfortable opening up to this person.
I prefer not to show this person how I feel deep down.
I often worry that this person doesn't really care for me.
I'm afraid that this person may abandon me.
I worry that this person won't care about me as much as I care about him or her.
I don't fully trust this person.
Strongly
disagree
Strongly
agree
–3 –2 –
Info 271B

16. Relationship Structures questionnaire
It helps to turn to this person in times of need.
I usually discuss my problems and concerns with this person.
I talk things over with this person.
I find it easy to depend on this person.
I don't feel comfortable opening up to this person.
I prefer not to show this person how I feel deep down.
I often worry that this person doesn't really care for me.
I'm afraid that this person may abandon me.
I worry that this person won't care about me as much as I care about him or her.
I don't fully trust this person.
Avoidance R R R R
Anxiety
Info 271B

17. Ipsatization “Ipse” (self) normalization
Accounts for individual use of a scale:
Transformation: subtract mean, divide by SD as computed for each individual
Strongly
disagree
agree
–3 –2 –
Person A
Person B
Person C
Info 271B

18. Semantic Differential Scales
Useful for understanding how people interpret various things, using feelings rather than items

19. Replication and Using Existing Survey Instruments
Find other surveys that are used in your area of interest.
Especially with existing surveys when reliability has been established.
Allows for comparisons between different samples if the question wording is the same.
If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.

20. Pre-Testing and Pilots
Pre-tests and Pilots are always necessary, unless the survey in its existing form has already been given before to the current population.
Pre-testing and Pilot studies should have a large enough response rate so that you can actually find problems!
Example: You want to survey 100 undergrads for a small study. You may need to at least pre-test on a 20% sample from your population of undergraduates. However, you cannot use these pre-test participants in your full sample of 100 students.
Pre-tests: Focus on individual questions or the entire survey instrument/questionnaire.
Pilots: Usually larger scale than pre-testing, involve testing the entire survey procedure.

21. Testing your questions during pre-testing
Behavior coding– interview some respondents as you give the survey questions and keep track of requests for clarification.
Ask pretest respondents to rephrase your questions in their own words.
Panels of ‘experts’: give your questions to groups of individuals for comments/suggestions.
Does the respondent’s comprehension of question meaning match that of the researcher?
Does the researcher put too much of an expectation of recall on the respondent?
Best case scenario: you get some or most of what you wanted to get– but often an uphill battle with justifying your operationalizations and wording choices.
Worst case scenario: you get wild differences in responses, respondents don’t understand key questions, large incompletion rate, money and time spent on conducting survey is wasted (except for your newfound appreciation for pre-testing and pilots)

22. Replication and Using Existing Survey Instruments
ALWAYS a good idea to find other surveys that are used in your area of interest.
Especially with large, funded surveys the questions may have been tested for reliability.
Allows for comparisons between different samples if the question wording is the same.
If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.