1. Debriefing in Online Experiment Harryadin Mahardika, PhD

2. Harryadin Mahardika Pop Economist FEUI & laporsuap.com Research objective: – “to liberate and empower consumer...” Current research: – Consumer empowerment – Consumer intervention/engineering – Mobile advertising Contact: – harryadin.mahardika@ui.ac.id / harryadin@gmail.com harryadin.mahardika@ui.ac.idharryadin@gmail.com – @HarrySastro 2

3. Discussion Agenda Informed consent Debriefing Randomization Participant incentives

4. Risk of experiment on human subject Deception, manipulation, priming, scenario. – May harm participants’ mind. – Mental fatigue. Examples?

5. Informed consent Information about the experiment and its risk. – As a basis for participants to decide after they understand what the research involves (risks and benefits ). Written consent vs unwritten consent. – Written: Investigators must typically obtain and document voluntary informed consent from research subjects. – Unwritten: button on an online form to indicate they have read and understood the consent form.

6. Informed consent in online experiment Limited interaction with participants – investigator often cannot tell whether a subject understood the informed consent statement. Online form: – Researchers can increase the likelihood that subjects are granting truly informed consent by requiring feedback from subjects about their level of understanding, Example: – by requiring a “click to accept” for each element in an informed consent statement or even administering short quizzes to establish that a subject understood. Reduce response rate: – Increase nonresponse to sensitive items (Singer, 1978) – Possibly produce biased data (Trice, 1987).

7. Risk in online experiment It exposes subjects to innocuous questions and benign or transient experiences with little lasting impact. In general, online experiments is no more risky than any of their offline counterparts. In some respects, they may be less risky: – The reduced social pressure (Sproull & Kiesler, 1991) in online surveys or experiments makes it easier for subjects to quit whenever they feel discomfort. – This freedom to withdraw is no trivial benefit, given the strong pressures to continue in face-to-face studies (e.g., Milgram, 1963) and even telephone calls.

8. Risk in online experiment Although risk in online settings is typically low, the actual risk depends on the specifics of the study. For example: – Some questions in a survey or feedback from an experiment may cause subjects to reflect on unpleasant experiences or to learn something unpleasant about themselves – e.g., Nosek et al.’s, 2002b, research on automatic stereotyping.

9. Risk in online experiment Experiments that deliberately manipulate a: – subject’s sense of self-worth, – reveal a lack of cognitive ability, – challenge deeply held beliefs or attitudes, or – disclose some other real or perceived characteristic..... may result in mental or emotional harm to some subjects.

10. Debriefing American Psychological Association (2002) ethical guidelines call for debriefing subjects: – “Providing an explanation of the nature, results, and conclusions of the research—as soon after their participation as practical.“ If deception was involved: – Researcher needs to explain the value of the research results and why deception was necessary.

11. Debriefing in online experiment When conducting research online: – Researchers can post debriefing materials at a Web site, – Provide debriefing materials to those who leave before completing the research (Nosek, Banaji, & Greenwald, 2002a). – For example, researchers can deliver debriefing material through a link to a “leave the study” button or through a pop-up window, which executes when a subject leaves a defined Web.

12. Debriefing in online experiment Appropriate debriefing in online research may be difficult: – The absence of a researcher in the online setting makes it difficult to assess a subject’s state. – Difficult to determine whether an individual has been upset by an experimental procedure or understands feedback received.

13. Extraneous Variables cont. Selection Bias  Improper assignment of test units to treatment conditions [sampling error]

14. Extraneous Variables cont. Mortality  Loss of test units while the experiment is in progress [respondents selected no longer wish to participate]

15. Controlling Extraneous Variables Randomisation  Randomly assigning test units to experimental groups by using random numbers Matching  Comparing test units on a set of key background variables before assigning them to the treatment Statistical Control  Measuring the extraneous variables and adjusting for their effects through statistical analysis Design Control  Use of experiments designed to control specific extraneous variables

16. Case: Beauty is in the eye of the beholder  Scientists have found a link between drinking alcohol and perceptions of beauty  80 students were shown colour photographs of 120 male and female students and were asked to rate the aesthetic properties on a 7-point scale from high unattractive to highly attractive  Half the students had drunk up to four units of alcohol, the other half had no alcohol.  The students who had consumed alcohol rated the people in the photographs as more attractive than the student who did not consume alcohol. Source: http://www.theage.com.au/articles/2002/09/091031115991721.html