It Takes All Sorts to Make a (Survey) World: Differential Effects of Survey Mode for the Factorial Vignettes and a Self-Report Personality Inventory Andrey Bykov and Inna F. Deviatko National Research University Higher School of Economics Moscow ESA RN21 / EQMC Conference, Mannheim, October 24-25, 2014

Methods of social science research: online vs. “traditional” (offline) Rapid increasing of Global Internet access – potentially representative samples Convenient for both participants and researchers Administering research online could be relatively cheap and fast However: the problem of equivalence!

Equivalence of online and offline modes of administration Numerous studies concerning surveys and psychological self-report inventories (Worthy and Mayclin, 2013; Wang et al., 2013; Gravlee et al., 2013; Kleck and Roberts, 2012, and others) Generally, the modes are found to be equivalent, although, usually with certain peculiarities To our knowledge, there are very few published studies (Nekrasov, 2011) on the modes’ equivalence of factorial vignettes (Rossi and Anderson, 1982) and no such studies concerning the Self-Report Altruism Scale (Rushton et al.,1981) In our experiment, we examined the mode equivalence of the altruism-related factorial vignettes and the SRA scale

Factorial Vignettes Vignettes are short, coherent, usually fictional, descriptions of situations or persons which are presented to the subjects as a kind of experimental stimuli and followed by questions or rating scales Factorial vignettes assume a model consisting of several factors, each with several levels, with vignettes being unique combinations of these levels phenomena (Rossi and Anderson, 1982) There can be different experimental designs (random, quota, full-factorial) FV are very convenient for administering online

The Self-Report Altruism Scale Introduced by Rushton et al. (1981) as a tool for measuring individual proneness to altruistic behavior A relatively simple Likert-type scale which consists of 20 items describing different altruistic actions, such as helping strangers, donating to charity, et cet. The participants are instructed to check the frequency with which they have engaged in these actions on a 5-point scale ranging from ‘never’ to ‘very often’

Method: participants and recruiting Two samples: offline (PP – paper and pencil) (73 students of HSE) and online (WB – Web based) (93 students of HSE and other Russian universities) PP participants were recruited in classroom. No reward was given for participation WB participants were recruited partly in classroom and partly online (via invitation after completing another study or an advertisement in a popular social network). No reward for participation.

Method: research design The questionnaire we used in our experiment (which was a part of a larger study of altruistic actions) consisted of factorial vignettes and the SRA scale, also including a block of questions about the participants’ evaluation of the questionnaire’s subjective difficulty The vignettes described two persons, X and Y, in a situation where X was in need (actually, we used different personal names instead of X and Y). After reading each vignette, participants were asked to assess whether, in their opinion, Y should give X the required donation using 11-point scale ranging from 0 (absolutely should not) to 100 (absolutely should) Full-factorial within-subject design. The experimental plan included 4 factors with 2 levels each: 1) relatedness between the two persons (relative vs. non-relative), 2) history of their relationships (X often helped Y in the past vs. X seldom helped Y in the past), 3) the possibility to meet in the future (moderate vs. high), and 4) the amount of required help (low vs. high). Therefore, every participant had to rate all the 16 (2*2*2*2) vignettes The vignettes and the SRA scale were presented to the subjects in a counterbalanced order WB and PP questionnaires were maximally identical in terms of design (2 vignettes per page, 10 SRA scale items per page).

Analysis and results Completion time (in seconds): Questionnaire typeMeanNStd. Deviation Std. Error of Mean MinimumMaximum WB873, ,130790, ,06701,0 PP530, ,073412, ,0840,0 ANOVA, F=11.17; p≤0.001, eta squared =

Analysis and results No significant differences between the modes for the FV

Analysis and results However, our analysis also showed a very slight, but significant difference between the modes for the total SRA scores. In the PP mode, mean SRA score was (σ=8.6), while (σ=8.9) in the WB mode. ANOVA confirmed statistical significance at p=0.018 with F=5.75 (eta squared =0.034). The SRA scale’s internal consistency (reliability) was almost equally high in both conditions (Cronbach’s alpha was 0.81 for offline and 0.80 for online).

Analysis and results (the SRA scale’s factorization) Eigenvalues of the factors in principal component analysis of the SRA scale (varimax rotation) WB modePP mode

Analysis and results (the SRA scale’s factorization) According to both Kaiser Eigenvalue and the Cattell’s scree plot criteria, we defined the number of factors for further CFA investigation: 6 factors for PP mode and 5 factors for WB mode. These factors respectively explained 60.6% of total variance for offline mode SRA data and 51.6 % - for online mode Individual items loadings in PP and WB modes were found to be partially coinciding for the three first factors and rather different for the rest.

Analysis and results (subjective comfort, FV) (Chi Squared=18, df=4, p<0.001).

Analysis and results (subjective comfort, SRA scale)

Discussion 1. First, our study showed that participants spent more time completing the web-based questionnaire compared to the paper-and-pencil one Possible explanation: other activities while filling in the questionnaire (drinking coffee, surfing the Web…) 2. Second, we found no significant differences in the assessments of factorial vignettes’ between WB and PP modes Possible explanation: there is no survey mode effect for the factorial vignettes 3. Third, our analysis showed a slight but significant effect the survey mode has on the Self-Report Altruism Scale total scores: these scores were a little higher for the participants who completed the online questionnaire Possible explanation: self-selection of the WB mode participants

Discussion (continued) 4. Fourth, we conducted a factor analysis of the Self-Report Altruism Scale separately for both PP and WB modes of administration: the factor structure was found rather different but partially coinciding Possible explanation: different recruiting procedures and unequal group sizes 5. Finally, participants in our WB mode study reported that working with factorial vignettes was for them rather difficult, while those who used a PP questionnaire did not think so. We did not find such differences for the SRA scale Possible explanation: rating of a number of complex factorial vignettes contradicts people’s general expectations of the Internet activities

Thank you for your attention!

Factor 1 (22.5% of variance explained)Description of items I have offered to help a handicapped or elderly stranger across the street I have helped carry a stranger’s belongings (books, parcels, etc.) I have given money to a stranger who needed it (or asked me for it) I have given directions to a stranger Factor 2 (10.7% of variance explained) I have given money to a charity I have donated blood I have done volunteer work for a charity I have donated goods or clothes to a charity I have bought ‘charity” Christmas cards deliberately because I knew it was a good cause Factor 3 (8% of variance explained) I have helped a classmate who I did not know that well with a homework assignment when my knowledge was greater than his or hers I have delayed an elevator and held the door open for a stranger I have let a neighbor whom I didn’t know too well borrow an item of some value to me (e.g., a dish, tools, etc.) I have pointed out a clerk’s error (in a bank, at the supermarket) in undercharging me for an item Factor 4 (7.3% of variance explained) I have helped an acquaintance to move households I have made change for a stranger I have before being asked, voluntarily looked after a neighbor’s pets or children without being paid for it have allowed someone to go ahead of me in a lineup (at photocopy machine, in the supermarket) Factor 5 (6.4% of variance explained) I have offered my seat on a bus or train to a stranger who was standing I have helped push a stranger’s car out of the snow Factor 6 (5.7% of variance explained) I have given a stranger a lift in my car

Factor 1 (22.1% of variance explained)Description of item I have offered my seat on a bus or train to a stranger who was standing 0, I have delayed an elevator and held the door open for a stranger I have made change for a stranger I have given money to a stranger who needed it (or asked me for it) I have offered to help a handicapped or elderly stranger across a street I have allowed someone to go ahead of me in a lineup (at photocopy machine, in the supermarket) Factor 2 (9% of variance explained) I have before being asked, voluntarily looked after a neighbor’s pets or children without being paid for it I have let a neighbor whom I didn’t know too well borrow an item of some value to me (e.g., a dish, tools, etc.) I have helped a classmate who I did not know that well with a homework assignment when my knowledge was greater than his or hers I have pointed out a clerk’s error (in a bank, at the supermarket) in undercharging me for an item Factor 3 (7.4% of variance explained) I have bought ‘charity” Christmas cards deliberately because I knew it was a good cause I have donated goods or clothes to a charity I have given directions to a stranger I have helped carry a stranger’s belongings (books, parcels, etc.) I have given money to a charity Factor 4 (7.3% of variance explained) I have donated blood I have helped push a stranger’s car out of the snow I have done volunteer work for a charity Factor 5 (5.8% of variance explained) I have given a stranger a lift in my car I have helped an acquaintance to move households