Download presentation
Presentation is loading. Please wait.
Published byAldous Pearson Modified over 9 years ago
1
Experimental thinkaloud protocols: a new method for evaluating the validity of survey questions Patrick Sturgis National Centre for Research Methods (NCRM) and University of Southampton Paper presented at the New Measurement Issues in Survey Research meeting of the Survey Resources Network, 21 September 2010
2
Do different questions measure the same thing? Many important concepts are measured by different ‘standard’ questions in surveys: Social/political trust General health Life happiness/satisfaction Fear of crime/confidence in police How to tell if they are ‘equivalent’? How to tell which is the ‘best’ measure?
3
Validity assessment strategies Face/process validity Correlation with criterion variables Multi-trait-multi-method (MTMM) Expert panels Behaviour coding Interviewer debrief Thinkaloud protocols/cognitive interview
4
Experimental thinkalouds Randomly assign respondents to receive one or other version of the ‘same’ question Follow-up with verbatim probe ‘what came to mind when answering last question?’ Examine marginal distribution of cognitive frames by question type Are people thinking of things they should be? Use thinkaloud variables in regression model to predict earlier response Which cognitive frames are most relevant in forming answers to the questions?
5
Example 1 - Trust
6
Conceptions of Trust Trust is a ‘good thing’ Trusting citizens are good citizens (voting, volunteering, civic engagement) Trusting societies are good societies (more democratic, egalitarian, > economic performance) Trust ‘lubricates’ social and economic transactions Reduces ‘monitoring costs’ and the need for contracts etc.
7
The standard trust question Generally speaking, would you say that most people can be trusted, or that you can't be too careful in dealing with people? Most people can be trusted Can’t be too careful Usually credited to Rosenberg (1959), the ‘Rosenberg Generalized Trust’ (RGT) item
8
The Local Area Trust item How much do you trust people in your local area? a lot a fair amount not very much not at all Reflects Putnam’s emphasis on trust being a property of local areas
9
Trust by Question type These items are both used more or less interchangeably as measures of generalized trust Yet, they yield very different estimates of trust at the national level. e.g.: Social Capital Community Benchmark survey: 47% most people can be trusted; 83% trust people in local area ‘some’ or ‘a lot’ UK Taking Part survey: 44% most people can be trusted; 74% trust ‘many’ or ‘some’ of the people in their local area Why such a large discrepancy in generalized trust (trust in strangers)?
10
Research Design Ipsos-MORI general population omnibus survey Random selection of small areas, quota controlled selection of individuals n=989 (fieldwork, November 2007) Respondents randomly assigned to RGT or TLA item In answering the last question, who came to mind when you were thinking about ‘most people’/ ‘people in your local area’?
11
Distributions for trust questions
13
Who comes to mind by RGT
14
Who comes to mind by TLA
15
Who came to mind – both questions
16
Explanatory Models 1
17
Explanatory Models 2
18
The science of well-being “Now is the time for every government to collect data on a uniform basis on the happiness of its population…every survey of individuals should automatically measure their well-being, so that in time we can really say what matters to people and by how much. When we do, it will produce very different priorities for our society. ” Layard 2010, Science.
19
Survey measures of subjective well-being Tend to ask about ‘happiness’ or ‘satisfaction’ with life And treat these as if they are measuring the same concept
20
Happiness = Satisfaction? Yes – time-series models show same pattern of effects (Blanchlower and Oswald, 2002) No – happiness and satisfaction correlated but not equivalent in European Values Survey (Gundelach and Kreiner 2004)
21
Mode effects Widely different estimates of well-being across different surveys Could mode be an explanatory factor? Being unhappy with your life is not socially desirable (people may over-state happiness to an interviewer) Conti and Pudney (2008) find higher ratings of satisfaction in interviewer relative to self- administered questions
22
Design Ipsos-MORI face-to-face omnibus survey (quota sample), April 2010 n=2033 Respondents randomly allocated to: 1. interviewer administered life satisfaction 2. Self-administered life satisfaction 3. Interviewer administered happiness 4. Self-administered happiness
23
Questions (from European Social Survey) All things considered, how happy would you say you are? Please answer using the scale on the card where 1 means ‘extremely unhappy’ and 10 means ‘extremely happy’. 1. Extremely unhappy. 10. Extremely happy All things considered, how satisfied are you with your life as a whole nowadays? Please answer using the scale on the card where 1 means ‘extremely dissatisfied’ and 10 means ‘extremely satisfied 1. Extremely dissatisfied. 10. Extremely satisfied
24
Verbatims Now, thinking about your answer to the last question, please tell me what came to mind when thinking about your answer. There are no right or wrong answers; I just want you to tell me everything that came to mind in thinking about how happy you are. What else? PROBE FULLY
25
Results 1 satisfaction = happiness?
26
Raw distributions for happiness and satisfaction Mean=7.38 Mean=7.39
27
Satisfaction v Happiness - distributions Pearson’s Chi Square, p=0.041
28
Satisfaction v Happiness by sex p=0.394p=0.047 Means Male = 7.43 Female = 7.34
29
Results 2 mode effects
30
Mode effect by question - means
31
Mode effect by question - distributions p=0.015p=0.209
32
Question*mode*sex - means
33
Question*mode*sex - distributions p=0.018 p=0.145p=0.037 p=0.053
34
Prediction model
35
Verbatim responses
36
Verbatim responses coded to a descriptive frame with 111 codes These were then allocated to one of 14 thematic codes
37
Thematic Codes
38
Significant differences in thematic codes across questions
39
Conclusions great deal of heterogeneity in the frames of reference people use in answering trust questions Acquaintances more trusted than strangers Problematic to assume these questions measure generalized trust Local area question should not be used interchangeably with standard trust item
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.