Reproducible Science Gordon Watts (University of Washington/Seattle) @SeattleGordon 2017-09-22 Center for modeling complex interactions G. Watts (UW/Seattle)

Outline Introduction and Motivation Work at UW eScience Center Work from DASPOS Work from CERN Funding Agencies Conclusions & Links G. Watts (UW/Seattle)

The Royal Society’s motto Nullius in Verba “take no one’s word for it” The Royal Society’s motto (founded 1660) G. Watts (UW/Seattle)

Gatherings where you would see the experiments reproduced The Royal Society Gatherings where you would see the experiments reproduced Robert Boyle Documentation standards so you could imagine being in the room Provides evidence the claim is true Check for fraud and error A “springboard for progress” by enabling replication The knowledge and the techniques used to gain that knowledge G. Watts (UW/Seattle)

Is this science today? G. Watts (UW/Seattle)

For a search at the world’s largest experiment 19 pages of text and plots 1 title page 4 pages of references 13 pages of author list For a search at the world’s largest experiment G. Watts (UW/Seattle)

G. Watts (UW/Seattle)

Complexity of Modern analysis software Big Science Big Datasets Complexity of Modern analysis software Page limits on articles G. Watts (UW/Seattle)

Can we do better? We might not be able to build a second Large Hadron Collider Perhaps some of the same tools that enable Big Science/Big Datasets can be used to address this G. Watts (UW/Seattle)

This is a many year effort! We have not yet codified best practices as Boyle did for this new environment G. Watts (UW/Seattle)

UW eScience Center G. Watts (UW/Seattle)

https://gordonwatts.github.io/ros-roadshow An adaptation of the standard eScience Road Show (Workshops for Students) G. Watts (UW/Seattle)

Badges https://github.com/uwescience-open-badges A small icon to be displayed on your Poster or Talk or Paper Open Data Badge Your data and your analysis code is open and available Open Materials Badge Your analysis code is open and available (e.g. medical data) Badges are unique for each of your submissions (DOI’s) “Signaling that you value openness will help to shift the expected norms of behavior in science.” This is a grassroots effort G. Watts (UW/Seattle)

Badges Submit Request Review Result Include disclosure statement (where to find materials, etc.) Request badge via Email, pull request, etc. Include the PDF of your talk, paper, poster. Review Two reviewers COS disclosure process Confirm that the links lead to data and code Will not attempt to run Will not attempt to insure that the data is the relevant data Loop to address reviewer comments Result Reviews and result published anonymously in a github repro for this submission DOI assigned to this repro (which includes a copy of your submission) Badge image which includes DOI linked to the repro is created G. Watts (UW/Seattle)

The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2018) https://www.practicereproducibleresearch.org/ A massive book being written by members of the eScience Center Part I: Practicing Reproducibility Part II: High-Level Case Studies Part III: Low-Level Case Studies (sampling of topics from last two sections) G. Watts (UW/Seattle)

DASPOS G. Watts (UW/Seattle)

Data And Software Preservation for Open Science Notre Dame, Chicago, UIUC, Washington, Nebraska, NYU, (Fermilab, BNL) Initial Goal: How to preserve knowledge from particle physics experiments This $3M grant started from the premise that backing up the data was easy. (re) Using it is hard Physicists, digital librarians, and computer scientists High Energy Physics linked to Biology, Astrophysics, Digital Curation, etc. G. Watts (UW/Seattle)

Data + Software + Knowledge Processed Data Selection Simplification Reduced Data Selection Analysis Simplification Twice-reduced Data Workflow at an LHC experiment is more than a simple linear sequence. How to preserve that – and make it reproducible? Data & Metadata Instructions on how to process data Lab notebooks, web pages, WebAPI’s From LHC More Inputs Twice-reduced Data Simulated Data Further Analysis Higgs Discovery! All must be preserved! G. Watts (UW/Seattle)

Meta Data https://github.com/Vocamp/ComputationalActivity https://github.com/Vocamp/DetectorFinalState Meta Data Use the Web Ontology Language (OWL) to describe meta data Part of the semantic web Searchable, discoverable, common tools to manipulate, etc. Encodes OS my.exe v4.6.31 v21.6A Input files 1 N … libAA v3.12.6 libCC v1.1.9 Config v0.1 Databases libBB v6.1.3 external Output files M G. Watts (UW/Seattle)

Capturing a Computation Step (as an example) Containers have replaced Virtual Machines as the preferred way Light weight, low resource usage Very fast to start Composable Must capture along with container Software, build environment Environment Instructions to run (metadata) But current containerization tools require expert usage What should be captured? What not? Permissions? Building the container? Smart containers can be linked in a knowledge graph to complete a full analysis G. Watts (UW/Seattle)

Work at CERN Putting this together G. Watts (UW/Seattle)

Workflow for An Analysis https://recast.perimeterinstitute.ca/ Workflow for An Analysis “Analysis” Data Workflow New Models Can we easily re-run an analysis when a new model becomes available? Repeat the data/theory comparison forever Preserve the analysis for reuse

Wouldn’t it be nice if you had a tool that: Automatically captured provenance and processing details for everything you did could manage the bookkeeping for thousands of analysis jobs provided a way of saving snapshots of work could tell you “what did I do last week?” G. Watts (UW/Seattle)

Outreach & Research: open to public http://opendata.cern.ch/ Internal Envisioned as a tool for the analysts to preserve, share and re-use analysis techniques, executables, and code External Outreach & Research: open to public CMS has released AOD-level data, VMs for analysis code for 2010, 2011, and analysis examples *(2 external papers)* Starting to look for partners outside of HEP G. Watts (UW/Seattle)

Link existing pieces of infrastructure CERN Gitlab connects to a computational backend G. Watts (UW/Seattle)

Workflow Capture Each computation’s environment and software is specified in a container JSON to chain the steps into a workflow, provide required metadata Complete flexibility. CERN IT Provided backend G. Watts (UW/Seattle)

Funding Agencies G. Watts (UW/Seattle)

Policy Questions mpsopendata.crc.nd.edu What data should be saved? What else should be saved besides the data to enable re-use? Where should it be stored? Who pays for the storage? Federal grants typically run 3 years after this the money is spent Who pays to allow the public to access the data? network & storage infrastructure isn’t free Funding agencies developing policies, need guidance NSF (and DOE) want to update their Data Management Plan guidelines Looking for input from the communities they serve

Policy Responses Community exercise to provide feedback to NSF on knowledge preservation questions from Physical Sciences (input from APS, ACS, surveys, workshops, etc.) Final Report just issued, some conclusions: Data and other digital artifacts upon which publications are based should be made publicly available in a digital, machine-readable format, and persistently linked to those relevant publications. Different disciplines have a wide variety of current practices and expectations for appropriate levels of the sharing of data and other scientific results. A discipline-specific policy discussion will be required to determine an appropriate level of preservation and re-use. The provision of public access to data entails costs in infrastructure and human effort, and that some types of data may be impractical to archive, annotate, and share. Cost-benefit analyses should be conducted in order to set the level of expectations for the researcher, his or her institution, and the funding agency. Creating incentives toward the sharing of data is the primary way to accomplish broad adoption of open access to data as the norm Exploring and understanding these ingredients are the next steps for the science disciplines

Conclusions Tools Next Big Task Improve the tools! Lots of personal and career benefits to using Open Science and Reproducible techniques in our daily work. Common Techniques Favor composable techniques Command line – all things scriptable and controllable from a common script As many non-proprietary tools as possible Good tools are already there Git/GitHub Containers and VM’s Notebooks Web Tools Get Started! Do not hesitate to publish and share new tools This is where the frontier of this effort currently exists Tools are all almost certainly going to be domain specific G. Watts (UW/Seattle)

References Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2018). The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. Oakland, CA: University of California Press. Mike Hildreth (Notre Dame), “Data, Software, and Knowledge Preservation of Scientific Results”, ACAT 2017 (Seattle). https://indico.cern.ch/event/567550/contributions/2656689/ UW eScience Center - http://escience.washington.edu/ - The reproducibility working group: http://uwescience.github.io/reproducible/ DASPOS - http://daspos.org/ eScience slides contain direct links as well G. Watts (UW/Seattle)