Madagascar Software Package Technology for Reproducible Computational Experiments Sergey Fomel Bureau of Economic Geology Jackson School of Geosciences The University of Texas at Austin Scientific Software Day 2007

Outline Past  Seismic Imaging MIT, Industry, Stanford, … Present  Test-driven development Reproducible computational experiments Future  Asking for help Automatic testing, parallel processing, graphics, …

Seismic Imaging Computations 45 clusters on the top 500 list  many missing  Europe’s most powerful supercomputer (CGG/IBM) Data size  Gb (2-D; 1980x)  Tb (3-D; 1990x)  ?? (4-D, 4-C, multi-azimuth, …; 200x)

History of Seismic Imaging Norbert Wiener (© RLE/MIT) Enders Robinson Jon Claerbout

SEPlib Software Innovations Universal data file format (RSF)  Data separated from text headers  Conceptual regularly gridded N-dimensional hypercubes If you feel an urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes. -- Eric S. Raymond, TAUP Universal graphics format (Vplot) Reproducible research 30+ years legacy  R. Clayton, J. Claerbout, D. Hale, S. Levin, J. Dellinger, S. Cole, D. Nichols, M. Karrenbach, B. Biondi, R. Clapp, … Data n1=1000 n2=500 n3=100 d1=0.001 d2=0.1 o2=1

“Madagascar” Innovations Open source (GPL) Open community (SourceForge) Open science  Command-line modules  Data processing scripts  Documentation

MondayGet an idea TuesdayImplement it WednesdayTest it ThursdayCommunicate it FridayApply it Technology Transfer I23I23

Three Levels of Technology Implement it (Tuesday)  Command-line modules C (C++, Fortran-77, Fortran-90, Matlab, Python) 400 modules Test it (Wednesday)  Data processing flows Python/SCons 4,000 tests Communicate it (Thursday)  Papers and reports LaTeX + Python/SCons 60 papers

Reproducible Computations Test Driven Development  Document computational experiments  Old experiments become regression tests Within the world of science, computation is now rightly seen as a third vertex of a triangle complementing experiment and theory. However, as it is now often practiced, one can make a good case that computing is the last refuge of the scientific scoundrel. –R. LeVeque

Asking for Help Open problems  Automatic reproducibility testing  Parallelization  Interfaces  Graphics  … Open opportunities  Summer students  Consultants

Information  Wiki, blog, lists, forums RSF/Madagascar School in Austin  April 20, 2007  Bureau of Economic Geology