Lecture 6 Structured Interviews and Instrument Design Part II: Scales and Scaling Lecture 6 Info 271B

Agenda Scales and Scaling Brief Demonstration and Getting Started with STATA

Know your sample population Regional language and terminology Cultural differences How you conduct survey can influence your valid sample Door to door? Registered telephone directory? Internet-based survey?

Scales and Scaling Scales are any device used to assign units of analysis to categories of a variable. Age is a scale, so is a categorical response like religion.

Single Indicator Scales Assigns units of analysis to categories of a variable Includes many common single-item questions (including nominal and metric variables) “Do you feel happy in general?” (answered on 0-10 point scale) “What is your race?” (answered on 6 categories)

Single-option questions online

Composite Measures Composite Measure Scale Cumulative index (Q1 + Q2 + Q3 + Q4) Cumulative scales

Other Types of Complex Scales Criterion referenced index Examples: Grade on 100 point scale Score on composite scale, regardless of distribution Norm referenced Grade on relative scale Relative score (high/low) based on sample distribution Why would you choose one or the other; under what conditions?

Common Scaling Techniques: Guttman Scaling Guttman Scale (Q1  Q2  Q3) Measurements for the item have a pattern indicating that the items measure a unidimensional variable. Define a focal concept Develop items for the scale Rate the items (judges, small sub-sample) Create cumulative scale Randomize order and distribute scale

Common Scaling Techniques: Likert-style scales Likert-style Scales [Q1, Q2, Q3] Originally 5-point, but also includes other types Procedure: Decide on the concept of interest Create list of indicator questions/statements Decide on number of responses Test items on small sample Use a form of item analysis to check for a unidimensional scale

Item Analysis Factor Analysis is most common approach, but also a fairly advanced procedure. Basic principle of all item analysis is to see if the individual variables in your composite “hang together” Score responses Get the interitem correlation

Interitem Correlation

Cronbach’s Alpha Uses split-half reliability technique Checking similarity of scores for various sets of questions

Finding items that do not discriminate Simple way: look at the extremes and check to see which items they have in common. Formal way: Item-total correlation

Relationship Structures questionnaire It helps to turn to this person in times of need. I usually discuss my problems and concerns with this person. I talk things over with this person. I find it easy to depend on this person. I don't feel comfortable opening up to this person. I prefer not to show this person how I feel deep down. I often worry that this person doesn't really care for me. I'm afraid that this person may abandon me. I worry that this person won't care about me as much as I care about him or her. I don't fully trust this person. Strongly disagree Strongly agree –3 –2 –1 0 1 2 3 Info 271B

Relationship Structures questionnaire It helps to turn to this person in times of need. I usually discuss my problems and concerns with this person. I talk things over with this person. I find it easy to depend on this person. I don't feel comfortable opening up to this person. I prefer not to show this person how I feel deep down. I often worry that this person doesn't really care for me. I'm afraid that this person may abandon me. I worry that this person won't care about me as much as I care about him or her. I don't fully trust this person. Avoidance R R R R Anxiety Info 271B

Ipsatization “Ipse” (self) normalization Accounts for individual use of a scale: Transformation: subtract mean, divide by SD as computed for each individual Strongly disagree agree –3 –2 –1 0 1 2 3 Person A Person B Person C Info 271B

Semantic Differential Scales Useful for understanding how people interpret various things, using feelings rather than items

Replication and Using Existing Survey Instruments Find other surveys that are used in your area of interest. Especially with existing surveys when reliability has been established. Allows for comparisons between different samples if the question wording is the same. If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.

Pre-Testing and Pilots Pre-tests and Pilots are always necessary, unless the survey in its existing form has already been given before to the current population. Pre-testing and Pilot studies should have a large enough response rate so that you can actually find problems! Example: You want to survey 100 undergrads for a small study. You may need to at least pre-test on a 20% sample from your population of undergraduates. However, you cannot use these pre-test participants in your full sample of 100 students. Pre-tests: Focus on individual questions or the entire survey instrument/questionnaire. Pilots: Usually larger scale than pre-testing, involve testing the entire survey procedure.

Testing your questions during pre-testing Behavior coding– interview some respondents as you give the survey questions and keep track of requests for clarification. Ask pretest respondents to rephrase your questions in their own words. Panels of ‘experts’: give your questions to groups of individuals for comments/suggestions. Does the respondent’s comprehension of question meaning match that of the researcher? Does the researcher put too much of an expectation of recall on the respondent? Best case scenario: you get some or most of what you wanted to get– but often an uphill battle with justifying your operationalizations and wording choices. Worst case scenario: you get wild differences in responses, respondents don’t understand key questions, large incompletion rate, money and time spent on conducting survey is wasted (except for your newfound appreciation for pre-testing and pilots)

Replication and Using Existing Survey Instruments ALWAYS a good idea to find other surveys that are used in your area of interest. Especially with large, funded surveys the questions may have been tested for reliability. Allows for comparisons between different samples if the question wording is the same. If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.