1. Lecture 16

2. Office hour cancelled This Wednesday only!
I was asked to represent our department at SAMSI panel.

3. Survey validity
Thanks to Dr. Neupert (my wife) for slides

4. Questionnaire & Interview
2. Survey:
Questionnaire & Interview
Positive: People can be the best experts on their own behavior
Unobservable data (i.e. opinions & beliefs)
Negative: Social desirability, memory

5. Validity A. Construct: does the operational definition “fit”?
B. Internal: causal conclusions
C. External: generalizability
Balance between b and c

6. Reliability A. Consistency or stability of a measure B. Assessment
1. True score
2. Measurement error
Observed score = True score + Measurement error
B. Assessment
1. Not directly observed/tested
2. Pearson correlation (measure of similarity)
A Single administration should closely approximate someone’s true score. Use the most highly reliable test possible. (If repeated hopefully similar answer)
B2 – need at least 2 scores on the measure from many individuals. If reliable, the two scores should be similar = high pos. corr.

7. 2. Pearson correlation (cont)
a. test-retest reliability: 2 points in time
b. internal consistency: 1 point in time
i. split-half
ii. Cronbach’s alpha
c. interrater reliability
Bi. Corr of an indiv’s total score one ½ of the test with the total score on the other half. Problems with this?
Bii. Corr of each item with every other item. Alpha = average of all correlations. Benefit – items that do not corr well with others can be dropped.
c. Extent to which raters agree. Cohen’s Kappa.
You will need to create a scale for Project 1 and test its reliability.

8. Construct validity A. How well does the operational definition “fit”?
B. Face validity
C. Criterion-oriented validity
1. Predictive validity
2. Concurrent validity
3. Convergent validity (positive control)
4. Discriminant validity (negative control)
b. Content appears to reflect construct.
C – scores relate to an indicator of the construct (criterion)
Scores predict beh. SAT pred. College. GRE predict grad school.
groups differ in expected ways. AP Calculus exam – expect those who have taken Calculus to do better than those who did not.
Two measures of the same construct “converge” or agree. Your new measure should corr highly with another established measure. When is this method not useful?
Measure should not be related to variables with which it should not be related. What would the expected correlation coefficient value be? EXAMPLE: spelling and IQ.

9. Measurement scales A. Nominal: no numeric properties
B. Ordinal: rank order
C. Interval: difference is meaningful
D. Ratio: absolute zero
1 is “higher” than 2, but the distance between 1 and 2 could be very different than that between 2 and 3.
c. Generally have 5+ quantitative levels. EXAMPLE: clothing sizes, shoe sizes. 0 does not mean absence of size!
D – height.
Stats tests for c and d are the same.

10. Surveys A. Match questions to research objectives
B. Types of questions:
1. attitudes and beliefs
2. facts and demographics
3. behaviors
don’t go overboard with questions – keep them focused.
B2 – necessary to adequately describe your sample.
B3 – past, present or future behs.

11. C. Question wording 1. Avoid: a. technical terms b. vagueness
c. grammatical errors
d. memory overload
e. misleading information
f. double-barreled questions
g. loaded questions
h. negative wording
i. yea-saying and nay-saying
Think of why you don’t like some exams. Do you think the prof. is trying to “trick you” with misleading information or vagueness?
f. Ask more than one thing at once.
g. Emotionally charged words (waste, dangerous, unethical)
h. Avoid negative wording – social security should not be privatized. Someone who agrees with privitization would need to select “disagree” , which can be confusing
i. If you ask lots of questions on the same topic…. Agreeing or disagreeing with anything you ask based on a response set. If yes to all questions indicates the same thing (e.g., happiness), they may consciously or otherwise just keep on agreeing regardless of the question.

12. Example (better): Were you raised in the same household as your twin?
Example (bad): Did or did not you’re monozygotic, genetically identical sibling have the same environmental resources as you?
CRITIQUE (from Quaid)
1. Unfamiliar technical term : sibling . The following term may be unfamiliar to some respondents: "sibling" in the first sentence.
2. Complex syntax Sentence 1: The question is either ungrammatical or difficult to parse syntactically. (4 modifiers for a noun are counted.)
3. Unfamiliar technical term : environmental . The following term may be unfamiliar to some respondents: "environmental" in the first sentence.
4. Unfamiliar technical term : you’re . The following term may be unfamiliar to some respondents: "you’re" in the first sentence.
Example (better): Were you raised in the same household as your twin?

13. 2. Do: a. keep it simple b. reverse the direction of some items
c. make items emotionally neutral
B – without using negative wording, when possible

14. D. Responses to questions
1. Closed-ended
2. Open-ended
3. Number of response alternatives
4. Rating scales: labeled extremes
a. graphic
b. semantic differential
c. non-verbal
3. Dichotomy (yes/no, disagree/agree), or 5-7 point scales.
4a. Making a mark along a line.
4b. Good/bad, strong/weak, active/passive. Evaluation, potency, activity
4c. Nonverbal. Good for research with kids , but also of abstract topics. Power within a couple.

15. 5. Labeling response alternatives (cont)
a. vague quantifiers
b. odd vs. even numbers of responses
c. quantitative differences between intervals
5. Vague quantifiers. Strongly disagree. What is the difference between disagree and strongly disagree? Will it be the same difference for all participants?
c. Less than twice per week (low end) to at least once each day (high end) vs. less than once per month (low) to more than once per week (high)

16. E. Formatting 1. PROOFREAD!!! 2. consider the sequence
3. clearly demarcate questions/items and responses
4. keep response formats consistent within a scale

17. Examples
FAMILISM SCALE (Reliability .84; Mean 47.77; S.D. 8.89) 1. Generally we don't all have meals at the same time in our family. R 2. Our family discuss a lot of things together. 3. We usually go out together at least once a week in our family. 4. I care about my family above everything else. 5. Our family has always been a close-knit family. 6. I am very proud of my family. 7. Members of our family have tended to grow away from one another. R 8. We're not as close as we used to be in our family. R 9. Our family does a lot of things together We always co-operate with one another in our family Family life can be terribly dull and boring. R 12. I sometimes feel like a stranger in my family. R 13. Family life these days is not what it used to be.
1-7 scale, 1 = strongly disagree, 7 = strongly agree (except where noted by R).
Australia
Social conservatism in Australia

18. Familism scale
1. If someone has the chance to help a person get a job, it is always better to choose a relative rather than a friend.
2. When someone has a serious problem, only relatives can help.
3. When looking for a job a person should find a job near his/her parents even if it means losing a better job somewhere else.
Scale – 1-4, with 1 = agree a lot, 2 = agree a little, 3 = disagree a little, 4 = disagree a lot.
Used for latino samples.

19. ECONOMIC CONSERVATISM SCALE (Perhaps better referred to as a "Socialism" scale) (Reliability .82; Mean 41.14; S.D. 9.84) "Am satisfied with" 1. Existing levels of foreign ownership of Australian industry. 2. The housing and land situation. "Am in favour of": 3. Moves to control poverty in Australia. R 4. Increased government spending on pensions and other social service payments. R 5. Moves towards government control of the health insurance schemes. R 6. Greater control of the unions by the government. 7. Power for the Federal government to control prices. R 8. Power for the Federal government to control incomes. R 9. A free dental service provided by the government. R 10. Government control of the big industries such as steel. R 11. Higher rate of taxation on the large income earner. R 12. Government power to control big businesses and monopolies. R
Eysenck's (1958) Neuroticism scale correlated with economic conservatism, with social conservatism, .041 with intended vote in a Federal election,
GO OVER REQUIREMENTS FOR PROJECT 1 IN TERMS OF QUESTIONS TO ASK FOR REQUIRED STATS (CORR, T-TESTS).

20. Final Project Design your own survey!
Find an interesting question and population
Design your sampling plan
Collect Data
Analyze using R
Write 5 page paper on your results
Due December 1

21. Final presentation All are required to give a short presentation
Last two classes (December 1 and 6)
Select one of the three projects
Make a powerpoint presentation (no more than 3-5 slides)
Present your results to the class