Notebook Development and Testing 1Hans J. Johnson, 2,4Bradley C. Lowekamp, 2,3Ziv Yaniv 1The University of Iowa 2National Institutes of Health 3TAJ Technologies Inc. 4MSC LLC

Maintaining Your Notebooks Notebooks are code and prose. Use a version control system (git, hg, svn). Test them, at least execution (pytest, nose2, unittest). Notebooks are not pure code, they may also contain execution results. Commit clean notebooks to avoid messy history and an unnecessarily large repository. Use a continuous integration service to run your tests (CircleCI, Travis CI, Appveyor).

SimpleITK Notebook Development and Testing One authoritative repository: github.com/InsightSoftwareConsortium/SimpleITK-Notebooks No direct commits to this repository, only pull requests. Code review and test (pytest) pull requests: Static: (1) ensure notebooks contain no output; (2) spellcheck all markdown/prose cells and comments in code. Dynamic: (1) Run the notebook using the jupyter nbconvert tool and analyze result (notebook in JSON format). Mark code cell meta-data: simpleitk_error_allowed (may or may not happen), simpleitk_error_expected; (2) regression testing – in the works. Deal with memory constraints on build machines.

SimpleITK Notebook Development and Testing Data sharing: Storage: (1) images on MIDAS data servers or on accessible web pages (raw url); (2) code repository contains data manifest file in JSON format with md5 to validate download integrity. Retrieval: Cache images locally. (1) without internet access, bulk download script; (2) with internet access, lazy download. For additional details see: “SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research”, J Digit Imaging 2017 .

Join the Community We welcome questions/requests/contributions from the community: Ask questions (itk discourse) Request features / examples. Report bugs. Contribute code or new notebooks. github.com/SimpleITK/SimpleITK and github.com/InsightSoftwareConsortium/SimpleITK-Notebooks