Make the World a Better Place through Reproducible Research Roger D. Peng Department of Biostatistics Johns Hopkins Bloomberg School of Public Health Wall of Wonder

Trends in Scientific Research Signal-to-noise in many investigations is getting smaller Smaller relative risks –e.g. relative risk of mortality is per 10 ppb of ozone High-throughput measurement technologies Powerful computers

Trends in Computing: Then...

...And Now

The Result? Large databases for investigating subtle associations Interactive computing with advanced statistical algorithms Sophisticated searches across models and variables to identify important risks Bigger and better studies

Replication: The Standard Scientific evidence is strengthened when important findings are replicated by independent investigators, data, methods, laboratories, instruments, etc. Replication is often not possible because of time, funding constraints Policy decisions must often be made with evidence at hand

Reproducible Research: A Minimum Standard Published research where the following are made available: Analytic data Computer code implementing methods Documentation about code/data All are distributed using standard means

Benefits of Reproducible Research Published findings can be verified Alternative analyses conducted Challenge uninformed criticisms (“put up or shut up”) Expedite exchange of ideas among investigators

Challenges to RR “If I give away my data, others will publish results and scoop me” “I own my data and ideas, other people don’t necessarily have any rights to them” Why should I just give away my intellectual property?

Ya see, it’s what I call the “ownership society”

Property [Automatt][JRodrigues] [james.thompson] [nervsappy]

“Intellectual Property” “the intangible value created by human creativity and invention” –from JHSPH Office of Technology Transfer (emphasis added) How can something that is intangible be property?

There’s No Such Thing as “Intellectual Property” If I copy your book, you still have your book If I use your idea, you still have your idea If I copy your data, you still have your data If I use your statistical model, you still have your statistical model If I implement your algorithm, you still have your algorithm etc.

Research done by you regardless of sharing What are the Potential Gains and Losses from Sharing Data? Research done by you regardless of sharing Data Research done by others Research you would have done if you hadn’t shared data Don’t share Share = Y(1) = Y(0) (a)D = 0 (b)D < 0 (c)D > 0 D = Y(1) - Y(0)

What is a Dataset? Represents already published findings and ideas Contains potential findings and ideas yet to be discovered and exploited Datasets do not fit well into the framework of copyrights and patents

What Do We Need? Infrastructure –Tools for researchers, developers –Repositories for datasets –Rights framework for datasets Privacy preservation Handle computer language Babel Structured research modularity

“WWKD”

“WWKD” What Would Karl Do?

Models for Reproducibility

An Example

Partial Rights for Data? A First Cut Full access: the data can be used for any purpose Attribution: the data can be use for any purpose with a specific citation Share-alike: the data can be used for any purpose but any “improvements” must be made available under the same license Reproduction-only: the data can only be used for reproducing published results and commenting via a letter to the editor

Thank you!