1. Reproducible Research
Sergey Fomel
The University of Texas at Austin

2. Outline Personal experience Why do reproducible research?
How to do reproducible research?
Reproducible computational experiments

3. Personal Experience Jon Claerbout and RR at Stanford
Madagascar open-source software
CiSE special issue (Jan-Feb 2009)
David Donoho et al
Randall LeVeque
Roger Ping and Sandrah Eckel
Victoria Stodden

4. Jon Claerbout’s Story 1987: Sunview experience 1992: LaTeX + cake
Interactive programs are slavery
1992: LaTeX + cake
Rebuilding books by a single command

5. Reproducible Research at SEP
Stanford Exploration Project
Founded in 1973
2 Ph.D. students per year
Reproducible research
From CD-ROMs to WWW
From cake to GNU make
2001 CiSE paper
The principal beneficiary is the author

6. The Madagascar Project
Multidimensional data analysis
Started in 2006
Open community
Open source (GPL)
Three levels
Building blocks in C
Recipes in Python/SCons
Papers in LaTeX + SCons

7. Personal Experience Jon Claerbout and RR at Stanford
Madagascar open-source software
CiSE special issue (Jan-Feb 2009)
David Donoho et al
Randall LeVeque
Roger Ping and Sandrah Eckel
Victoria Stodden

8. CiSE Reproducible Research
David Donoho, Arian Maleki, Inam Rahman, Morteza Shahram, Victoria Stodden
15 years of reproducible research in computational harmonic analysis
MATLAB
WaveLab: 690 citations
“Striving for reproducibility imposes a discipline that leads to better work.”

9. CiSE Reproducible Research
Randall J. LeVeque
Python tools for reproducible research on hyperbolic problems
Fortran + Python
Clawpack: 7,000 registered users
“Scientific and mathematical journals are filled with pretty pictures of computational experiments that the reader has no hope of repeating.”

10. CiSE Reproducible Research
Roger D. Peng and Sandrah P. Eckel
Distributed reproducible research using cached computations
R language
Cacher package
“We propose that a modular research approach lends itself more naturally to reproducible results.”

11. CiSE Reproducible Research
Victoria Stodden
The legal framework for reproducible research in the sciences
Licensing and copyright
ORL (Open Research License)
“We need a license designed with the needs of computational researchers in mind.”

12. Why Reproducible? open exchange of data, procedures and materials.
Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories. The success and credibility of science are anchored in the willingness of scientists to independent testing and replication by other scientists. This requires the complete and
open exchange of data,
procedures and materials.

13. Open-Source Software the long-awaited maturation of
“Abandoning the habit of secrecy in favor of process transparency and peer review was the crucial step by which alchemy became chemistry. In the same way, it is beginning to appear that open-source development may signal
the long-awaited maturation of
software development as a
discipline.” Eric S. Raymond

14. How to Do Reproducible Computational Experiments?
Code attached to published results
Continuous maintenance
Previous results used for testing
Test-driven development
Lessons from open-source
Intellectual property
Community