Presentation is loading. Please wait.

Presentation is loading. Please wait.

An Open Science Framework for Managing and Sharing Research Workflows

Similar presentations


Presentation on theme: "An Open Science Framework for Managing and Sharing Research Workflows"— Presentation transcript:

1 An Open Science Framework for Managing and Sharing Research Workflows
David Mellor Journal and Funder Initiatives | | @OSFramework

2 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

3 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

4 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

5 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

6 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

7 Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Merton, 1942

8 Anderson, Martinson, & DeVries, 2007

9 Anderson, Martinson, & DeVries, 2007

10 Anderson, Martinson, & DeVries, 2007

11 Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

12 Flexibility in analysis
Problems Flexibility in analysis Selective reporting Ignoring nulls Lack of replication Examples from: Button et al – Neuroscience Ioannidis – why most results are false (Medicine) GWAS Biology Two possibilities are that the percentage of positive results is inflated because negative results are much less likely to be published, and that we are pursuing our analysis freedoms to produce positive results that are not really there. These would lead to an inflation of false-positive results in the published literature. Some evidence from bio-medical research suggests that this is occurring. Two different industrial laboratories attempted to replicate 40 or 50 basic science studies that showed positive evidence for markers for new cancer treatments or other issues in medicine. They did not select at random. Instead, they picked studies considered landmark findings. The success rates for replication were about 25% in one study and about 10% in the other. Further, some of the findings they could not replicate had spurred large literatures of hundreds of articles following up on the finding and its implications, but never having tested whether the evidence for the original finding was solid. This is a massive waste of resources. Across the sciences, evidence like this has spurred lots of discussion and proposed actions to improve research efficiency and avoid the massive waste of resources linked to erroneous results getting in and staying in the literature, and about the culture of scientific practices that is rewarding publishing, perhaps at the expense of knowledge building. There have been a variety of suggestions for what to do. For example, the Nature article on the right suggests that publishing standards should be increased for basic science research. [It is not in my interest to replicate – myself or others – to evaluate validity and improve precision in effect estimates (redundant). Replication is worth next to zero (Makel data on published replications; motivated to not call it replication; novelty is supreme – zero “error checking”; not in my interest to check my work, and not in your interest to check my work (let’s just each do our own thing and get rewarded for that) Irreproducible results will get in and stay in the literature (examples from bio-med). Prinz and Begley articles (make sure to summarize accurately) The Nature article by folks in bio-medicine is great. The solution they offer is a popular one in commentators from the other sciences -- raise publishing standards. Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

13 Figure by FiveThirtyEight.com Silberzahn et al., 2015

14 A Garden of Forking Paths
“Does X affect Y?” Exclude outliers? Control for year? Median or mean? A series of perfectly reasonable decisions coupled with motivated reasoning can quickly lead us to a subset of statistically significant results. In effect, our hypothesis changed without us even realizing it. Our confirmatory, hypothesis testing became exploratory, hypothesis generating without our permission! Jorge Luis Borges; Gelman and Loken

15 Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan and Irvin, 2015

16 Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Positive result rate dropped from 57% to 8% after preregistration became required for clinical trials. Kaplan and Irvin, 2015

17 Metascience: Estimates Reproducibility

18

19

20

21 Reproducibility in other fields
Developmental Psychology Michael Frank Ecology Emilio Bruna Health Sciences Computer Science Leslie McIntosh Cynthia Hudson-Vitale Christian Collberg Todd Proebsting

22 Solutions? Appeal to intentions, values, and goals.
“Hey You! Behave by your values! Be objective!”

23 Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

24 Rewards What is published? What is not? Novel results Positive results
Clean results Replications Negative results Mixed evidence

25 Flexibility in analysis
Problems Flexibility in analysis Selective reporting Ignoring nulls Lack of replication Examples from: Button et al – Neuroscience Ioannidis – why most results are false (Medicine) GWAS Biology Two possibilities are that the percentage of positive results is inflated because negative results are much less likely to be published, and that we are pursuing our analysis freedoms to produce positive results that are not really there. These would lead to an inflation of false-positive results in the published literature. Some evidence from bio-medical research suggests that this is occurring. Two different industrial laboratories attempted to replicate 40 or 50 basic science studies that showed positive evidence for markers for new cancer treatments or other issues in medicine. They did not select at random. Instead, they picked studies considered landmark findings. The success rates for replication were about 25% in one study and about 10% in the other. Further, some of the findings they could not replicate had spurred large literatures of hundreds of articles following up on the finding and its implications, but never having tested whether the evidence for the original finding was solid. This is a massive waste of resources. Across the sciences, evidence like this has spurred lots of discussion and proposed actions to improve research efficiency and avoid the massive waste of resources linked to erroneous results getting in and staying in the literature, and about the culture of scientific practices that is rewarding publishing, perhaps at the expense of knowledge building. There have been a variety of suggestions for what to do. For example, the Nature article on the right suggests that publishing standards should be increased for basic science research. [It is not in my interest to replicate – myself or others – to evaluate validity and improve precision in effect estimates (redundant). Replication is worth next to zero (Makel data on published replications; motivated to not call it replication; novelty is supreme – zero “error checking”; not in my interest to check my work, and not in your interest to check my work (let’s just each do our own thing and get rewarded for that) Irreproducible results will get in and stay in the literature (examples from bio-med). Prinz and Begley articles (make sure to summarize accurately) The Nature article by folks in bio-medicine is great. The solution they offer is a popular one in commentators from the other sciences -- raise publishing standards. Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

26

27 Mission: Increase the openness, integrity, and reproducibility of scientific research.

28 Our Strategy: Evidence to encourage change.
Incentives to embrace change. Technology to enable change. Improving scientific ecosystem

29 Incentives to embrace change

30 Agnostic to discipline
TOP Guidelines Transparency and Openness Promotion Agnostic to discipline Low barrier to entry Modular

31 Transparency & Openness Promotion Guidelines
Eight Standards Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication Three Tiers Disclose Require Verify Signatories 539 Journals 59 Organization Learn more at

32 Transparency & Openness Promotion Guidelines
Eight Standards Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication Three Tiers Disclose Require Verify Signatories 539 Journals 59 Organization Learn more at

33 Transparency & Openness Promotion Guidelines
Eight Standards Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication Three Tiers Disclose Require Verify Signatories 757Journals 64 Organization Learn more at

34 Signals: Making Behaviors Visible Promotes Adoption
Badges Open Data Open Materials Preregistration Psychological Science (Jan 2014)

35

36 40% 30% % Articles reporting that data was available 20% 10% 0%

37 100% 75% 50% 25% 0% % of Articles reporting that data was available
On the y axis, we have % of articles reporting data available in an independent, open access location On the x axis, we have five categories: reportedly available, accessible, correct, usable, and complete data 25% 0% Reportedly Available Accessible Correct Data Usable Data Complete Data

38 100% 75% 50% 25% 0% % of Articles reporting that data was available
In an ideal world, we’d see straight lines for articles published in all journals, and in psychological science before and after badges This would mean that all articles that reported available data had data that was accessible, correct, usable, and complete 25% 0% Reportedly Available Accessible Correct Data Usable Data Complete Data

39 100% 75% 50% 25% 0% % of Articles reporting that data was available
In Psychological Science prior to badges and in other comparison conditions, only 39% or less (20%, 16%) of articles with reportedly available data in the comparison conditions had data that was accessible, correct, usable, and complete 25% 0% Reportedly Available Accessible Correct Data Usable Data Complete Data

40 100% 75% 50% 25% 0% % of Articles reporting that data was available
Of articles in Psychological Science that earned badges, over 75% of articles with data reportedly available had data that was accessible, correct, usable, and complete While not perfect, reportedly available data were more likely to be persistent when badges were earned -- perhaps due to the accountability of receiving a badge on the publication. 25% 0% Reportedly Available Accessible Correct Data Usable Data Complete Data

41 The $1,000,000 Preregistration Challenge
Another incentive for researchers to try out preregistration.

42 Exploratory research:
Finds unexpected trends Pushes knowledge into new areas Results in a testable hypothesis

43 Confirmatory research:
Puts a hypothesis to the test Does not allow data to influence the hypothesis Results are held to the highest standard of rigor

44 https://cos.io/prereg

45

46 Data collection methods
Research questions Data collection methods Variables Statistical tests Outliers

47 Technology to enable change

48

49 http://osf.io/ free, open source
Share data, share materials, show the research process – confirmatory result make it clear, exploratory discovery make it clear; demonstrate the ingenuity, perspiration, and learning across false starts, errant procedures, and early hints – doesn’t have to be written in painstaking detail in the final report, just make it available. free, open source

50 Collaboration Documentation Archiving
And it does things that many content management systems do. But it always balances things researchers care about while either automatically or making it very easy to align with values. Again, scholars should focus on scholarship--and we respect that they have a workflow—this has to be frictionless.

51 So here, it’s very easy to curate a project, add contributors, share, but we also keep an activity log. To the users that’s either meaningless or an easy way to keep up-to-date, but to us, this is provenance—this is important information about how science happens—science is a process, this is great time series metadata for metascience

52 Put data, materials, and code on the OSF

53 Manage access and permissions

54 Version Control

55

56 Registration Registrations: Freezing the current state of the project giving it a unique identifier.

57 We also collect a bit of metadata as well.
People traditionally use these like clinical trials: state hypotheses, plan, and then see if you confirm those results, conduct that plan

58 Merges Public-Private Workflows
We don’t mandate openness—you can use the site publically or privately.

59 But being open is only 2 clicks away.
You’ll also note that components can have independent authorship lists as well as permissions and privacy settings.

60 Incentives to embrace change

61 Persistent Citable Identifiers

62

63 Incentives for Openness File downloads
We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited.

64 Visits File downloads Forks
We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited. Forks

65 Connecting the workflow is critical to enabling change

66 OpenSesame

67 OpenSesame

68 https://api.osf.io/v2/docs/
All of this is now possible with the new OSF API. API Docs

69

70

71 Substantive experts shouldn’t have to deal with…
Workflow integration Authentication Permissions File storage File rendering Database Metadata/Annotations/Commenting Persistence External service integrations Search SHARE Data

72 Substantive experts should focus on…
Defining community standards Changing community values and incentives Determining specific community requirements Doing scholarship Just let experts be experts!

73 Society-branded Preprint Servers

74 OSF for Institutions

75 Thank you! David Mellor, Find me online: Find this presentation:


Download ppt "An Open Science Framework for Managing and Sharing Research Workflows"

Similar presentations


Ads by Google