1 Internet-based research Steve Janssen University of Amsterdam

2 Internet-based research Advantages: DNMT and GC Limitations Trustworthiness Reliability Validity: Self-selection bias and drop- out confounding Conclusions

3 Advantages (Reips, 2000) Diversity of population Potentially high number of participants Fewer subject-experimenter effects Up-to-date stimuli Low costs Longevity experiment

4 Characteristics Internet users (Gosling et al., 2004) Internet samples are more divers than traditional samples (i.e., first-year psychology students), although they are not completely representative of the entire population. Internet users do not differ from non- users on markers of adjustment and depression. Online participants are usually highly motivated.

5 High number of participants People can participate at any time from any place.

6 When do people take tests?

7

8 Advantages Daily News Memory Test (Meeter et al., in press) Galton-Crovitz Test (Janssen et al., in press-a; in press-b)

9 Daily News Memory Test (DNMT) 10 open-end and 20 multiple-choice questions Q: Welk land stemde op 29 mei 2005 als eerste tegen de Europese grondwet? Constantly 400 questions available Stratified selection of questions

10

11

12

13

14 Daily News Memory Test November 2000 – June 2005 N = Male 47.47%; Female 52.53% Lag. School5.2%;LBO 4.8%; VMBO 9.4%; HAVO 9.8%; VWO 12.1%; MBO 11.8%; HBO 24.7%; WO 22.2% M age = yrs

15

16 Galton-Crovitz Test Participants are presented 10 cue words They have to describe the first specific personal, which is associated to the cue word, that comes to mind Then, they have to date these 10 personal events and 10 public events. Example: Wanneer stemde de Nederlandse bevolking tegen de Europese grondwet?

17

18 Galton-Crovitz Test N = 8291 June 2002 – June 2005 Male 39.78%; Female 60.21% N au = 92; N be = 305; N ca = 119; N uk = 227; N nl = 6596; N us = 952 M age = yrs

19 Reminiscence Bump Period around early adulthood with relatively more memories than the period before or after.

20 Rubin, Wetzler & Nebes (1986)

21 Removing recency effect

22 Removing recency effect

23 Absence culture effect?

24 Current focus Galton-Crovitz Test Different cultures: Portuguese, Italian, and Japanese versions First times Emotionality and valence Semantic knowledge Preference in movies, records, and books

25 Limitations No psychophysical measurements Response-time experiments need plug- ins Refresh rate computer screens Tests should not take more than 20 to 30 minutes Participants browse through instructions

26 Drop-out

27 Drop-out Giving financial reward Indicating in advance how long the test takes Indicating participants progression

28 Usability test / Pilot Do the links between pages work? Does the test work under different browsers and computers? Do error messages appear? Are the instructions clear? Are there any grammatical errors? Is the time to complete the test not too long?

29 Seriousness experiment Short URL Sober lay-out, no flash Name and logo university or institute Emphasize goal and importance of experiment Emphasize that the site is not commercial Mention approval Internal Review Board Debriefing Provide opportunity to give feedback or ask questions

30 Build it and they will come? New media: Search engines, mailing lists, and websites Traditional media: Newspapers, magazines, and radio programmes Word of mouth

31 Algemeen Dagblad

32 Het Financieel Dagblad

33 Metro

34 De Volkskrant

35 Chessa & Murre (2004)

36 Problem publications Participants know what the experiment is measuring

37 Action letter Memory & Cognition …you need to show that you have better awareness of who these people were. That is, how can you be sure that you were not testing some bored 14 year olds?

38 Trustworthiness Trustworthiness = subject fraud Participant is not who he says he is Cheating (e.g., using other websites to look up the answer) Participant takes the test multiple times, each time under a different name Multiple participants take the test under the same name

39 Trustworthiness Password technique: send only one password to the provided address Dynamic test: test contains each time different questions Long registration – Short log-in procedure: only analyze the first test Record IP address Measure time needed to complete the test: omit tests, which took too little or too much time to complete Filter questions

40 Reliability The extent to which a test is free from random error components or non- systematical errors The circumstances in which the test is taken

41 Reliability DNMT Test-retest correlation; N = 1750, r =.617, p <.001 Split-half reliability; N = 4797, r =.684 Cronbachs alpha; N = 4797, α =.681 KR 21 ; N = 7192, KR 21 =.635

42 Validity The extent to which a test reflects only the desired construct without contamination from other systematically varying constructs Does the test measure computer skills as well? Mirroring, self-selection bias, and drop- out confounding

43 Mirroring Half the participants take the test in the laboratory, whereas the other half is send away and asked to take the test somewhere else. The results of those two groups are compared to each other and to the results of a third group of participants with similar characteristics.

44 Mirroring Buchanan & Smith, 1999 Gosling et al., 2004 McGraw et al., 2000 Smith & Leigh, 1997

45 Mirroring Compare results to results of experiments by other people Compare results before publication in media with results recorded directly after publication

46 Self-selection bias The effect that voluntary participants perform better on a test or that they are more biased in a questionnaire, because they have a greater interest in the specific topic of the test (Smith & Leigh, 1997) DNMT: Record peoples interest in the news (e.g., how frequently does the participant read a newspaper)

47 Self-selection bias Participants pool technique: Subjects are selected from a large database. Therefore, one knows who took the test and who did not take the test. Multiple site entry technique: The test has more than one home pages or it has links from different websites.

48 Drop-out confounding (Reips, 2002) Participants, who perform badly on a test, are less likely to complete a test than participants, who perform well on a test. Therefore, one should record who does not complete the test as well as who completes the test.

49 Conclusions High number of participants with relatively diverse backgrounds However, no psychophysical measurements Problems: Trustworthiness, self- selection bias, and drop-out confounding