1. Today's program The Utz et al. Paper Our follow up survey
Related matters
from research question to research question
selection effects (and what to do about it)
Multi-level analysis on such kinds of data

2. The Utz et al. paper (and follow up questions)

3. [show Utz summary and follow-up questions]

4. New questions should really follow-up on the scientific discussion, not just “be related”.
For instance: Do the results still hold: (1) when the weaknesses are taken into account? (2) under slightly different circumstances? (3) given different kinds of respondents? But you have to have a reason why you think this might be so! An this potential difference should be interesting.
In Utz et al, for instance: come up with a reason why we do not replicate the Kim finding.

5. Our follow-up survey Take our follow-up survey (now)…
Concentrate, don’t interact with others
… also because these will be the data that you will work with at the exam
… and because you will need to invite other people to participate

6. For your own run, use your own StudentID and p=000
For your own run, use your own StudentID and p=000
For your invitations use your own StudentID and different p-values per invitation!

7. <go through the survey here>

8. Selection effects My sample might not be representative for the population That is: for which group of people are my results supposed to hold?

9. Selection effects Kiezen van de doelgroep vs non-response.
eBay vs non-eBay users
computer literate vs not
...
those who participate and those who don't
carefully consider what the difference might be
compare the early with the late responders
ask something to the non-participants

10. Selection effects (cont’d)
And ... what was so good about experiments again? ...
... non-response might screw that up!

11. To Do We're going to run the survey you just completed
Come up with 15 persons to invite. Vary the characteristics. When you did not submit follow-up questions you have to come up with 25
(use the Excel template online)
Create an invitation list. Do not invite anyone yet. Submit this list to CS on Thursday latest. Start inviting this Friday.
We will keep track of who indeed participates, and who does not
These data can also be part of the exam data