Preregistration Challenge The $1 Million Preregistration Challenge The preregistration challenge is an education campaign. Its purpose is to initiate new habits in researchers by giving $1,000 prizes for specifying key details before seeing the data. This step adds transparency in a research culture whose values don’t encourage it. But in order to demonstrate why preregistration is powerful, and why it requires prizes to encourage, let me firt What is problem- lower Goal- initiate a habit, gain experience and education, can affect them and their colleagues, Problem is shown by K and I How does pre-reg improve the problem?

Problem Researchers face strong incentives to present the cleanest possible results.

Problem Researchers face strong incentives to present the cleanest possible results. There is little incentive for presenting the messy, unclear reality.

Problem Researchers face strong incentives to present the cleanest possible results. There is little incentive for presenting the messy, unclear reality. In biomedical research, 96% of published studies showed a p value of .05 or lower. (Chavalarias et al., 2016) This conflict is demonstrated by the fact that most published research findings show support for a hypothesis. So in this survey, 96% of published studies that contained a p value, showed one that was .05 or lower- the traditional level of statistical significance. If the published literature represented the true body of evidence, you would expect that proportion to be much lower. So how do these incentives affect individual researchers and lead to this bias?

A Garden of Forking Paths To explain how those incentives leads to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the researcher can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision that is easy to justify and which is very biased once you see the collected data, leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. Jorge Luis Borges; Gelman and Loken

A Garden of Forking Paths “Does X affect Y?” To explain how those incentives leads to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the research can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision that is easy to justify and which is very biased once you see the collected data, leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. Jorge Luis Borges; Gelman and Loken

A Garden of Forking Paths Median or mean? “Does X affect Y?” To explain how those incentives leads to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the research can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision that is easy to justify and which is very biased once you see the collected data, leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. Jorge Luis Borges; Gelman and Loken

A Garden of Forking Paths Exclude outliers? Median or mean? “Does X affect Y?” To explain how those incentives leads to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the research can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision that is easy to justify and which is very biased once you see the collected data, leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. Jorge Luis Borges; Gelman and Loken

A Garden of Forking Paths Control for time? Exclude outliers? Median or mean? “Does X affect Y?” To explain how those incentives leads to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the research can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision that is easy to justify and which is very biased once you see the collected data, leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. Jorge Luis Borges; Gelman and Loken

A Garden of Forking Paths Control for time? Exclude outliers? Median or mean? “Does X affect Y?” To explain how those incentives lead to that set of unrepresentative p values, let me describe a research project as a garden of forking paths. Each fork in the path represents a decision that the researcher can make. At the beginning, there is a seemingly simple question: Does X affect Y. However, along the way there are actually many possible decisions that researchers can make. 1, 2, 3. Each seemingly minor decision is easy to justify HOWEVER, the decision is biased once you see the collected data, and that bias reasoning leads you to the small subset of results that tell the most compelling story, even though the vast body of evidence in the dataset and all the other tests, which are equally valid, go unreported. A preregistration is one specific path in that garden, made before seeing the data that can bias those decision forks. Jorge Luis Borges; Gelman and Loken

Exploratory research: Finds unexpected trends Pushes knowledge into new areas Results in a testable hypothesis

Confirmatory research: Puts a hypothesis to the test Does not allow data to influence the hypothesis Results are held to the highest standard of rigor What we usually mean to do is confirmatory research- testing a specific, prespecified hypothesis.

A preregistration is a read only copy of a detailed research plan in an online archive. Think of it as one path in the garden, specified before seeing the data. Key details are required: exactly what is your hypothesis? How are you going to measure your variables? What statistical test will you use and how will you remove your weird outliers? By preserving a read-only copy of your plans- preregistration preserves the value of your hypothesis. RQ and Analyis plan

Data collection methods Research questions Data collection methods Variables Statistical tests Outliers A preregistration is a read only copy of a detailed research plan in an online archive. Think of it as one path in the garden, specified before seeing the data. Key details are required: exactly what is your hypothesis? How are you going to measure your variables? What statistical test will you use and how will you remove your weird outliers? By preserving a read-only copy of your plans- preregistration preserves the value of your hypothesis. RQ and Analyis plan

FAQ Q: Are research plans reviewed? Q: What if there are changes to my study? Q: Is it private? Q: Can I preregister without entering the competition? Q: I have more questions! A: Yes, research plans are reviewed for completeness and adherence to the competition’s rules. This is guaranteed to take less than 10 days and typically takes 2 or 3 days. A: Preregistration makes clear what was specified ahead of time versus what happened after the study began. Rule of thumb: be transparent and report deviations and justifications. Email prereg@cos.io for advice. A: Roughly 40% of entrants so far have made their research plans private under an embargo that may be up to 4 years long. A: Yes A: Contact: prereg@cos.io or @OSFramework

FAQ Q: Are research plans reviewed? Q: What if there are changes to my study? Q: Is it private? Q: Can I preregister without entering the competition? Q: I have more questions! A: Yes, research plans are reviewed for completeness and adherence to the competition’s rules. This is guaranteed to take less than 10 days and typically takes 2 or 3 days. A: Preregistration makes clear what was specified ahead of time versus what happened after the study began. Rule of thumb: be transparent and report deviations and justifications. Email prereg@cos.io for advice. A: Roughly 40% of entrants so far have made their research plans private under an embargo that may be up to 4 years long. A: Yes A: Contact: prereg@cos.io or @OSFramework

FAQ Q: Are research plans reviewed? Q: What if there are changes to my study? Q: Is it private? Q: Can I preregister without entering the competition? Q: I have more questions! A: Yes, research plans are reviewed for completeness and adherence to the competition’s rules. This is guaranteed to take less than 10 days and typically takes 2 or 3 days. A: Preregistration makes clear what was specified ahead of time versus what happened after the study began. Rule of thumb: be transparent and report deviations and justifications. Email prereg@cos.io for advice. A: Roughly 40% of entrants so far have made their research plans private under an embargo that may be up to 4 years long. A: Yes A: Contact: prereg@cos.io or @OSFramework

FAQ Q: Are research plans reviewed? Q: What if there are changes to my study? Q: Is it private? Q: Can I preregister without entering the competition? Q: I have more questions! A: Yes, research plans are reviewed for completeness and adherence to the competition’s rules. This is guaranteed to take less than 10 days and typically takes 2 or 3 days. A: Preregistration makes clear what was specified ahead of time versus what happened after the study began. Rule of thumb: be transparent and report deviations and justifications. Email prereg@cos.io for advice. A: Roughly 40% of entrants so far have made their research plans private under an embargo that may be up to 4 years long. A: Yes A: Contact: prereg@cos.io or @OSFramework

FAQ Q: Are research plans reviewed? Q: What if there are changes to my study? Q: Is it private? Q: Can I preregister without entering the competition? Q: I have more questions! A: Yes, research plans are reviewed for completeness and adherence to the competition’s rules. This is guaranteed to take less than 10 days and typically takes 2 or 3 days. A: Preregistration makes clear what was specified ahead of time versus what happened after the study began. Rule of thumb: be transparent and report deviations and justifications. Email prereg@cos.io for advice. A: Roughly 40% of entrants so far have made their research plans private under an embargo that may be up to 4 years long. A: Yes A: Contact: prereg@cos.io or @OSFramework

Our typical strategy at the Center for Open Science is to meet people where they are and to encourage better practices from there. Preregistration, on the other hand, requires a change in workflow. Therefore, the $1,000 prizes are necessary in order to provide more stimulus to take that bigger step and to initiate a new habit. The goal of the competition is to initiate a new habit. If a researcher tries it out once and sees the benefit to their workflow, they will come back again without the need for a prize. They will pass it on to their students, to their colleagues, and across disciplines. We are learning what researchers need and will use that for surfacing existing education content and creating new content. Email: prereg@cos.io