1. Notebook Development and Testing
Ziv Yaniv1,2 , Bradley C. Lowekamp1,2
1National Institutes of Health 2Medical Science and Computing LLC

2. Jupyter Notebooks – no silver bullet but a respectable experimentation environment
They aren’t as bad as some claim: Joel Grus’ JupyterCon 2018 talk “I Don’t Like Notebooks” (meme heavy, summary of sorts and comments below):
Notebook state is hidden (cells not run in linear order) – interactive vs. batch environment. Results are only valid when the whole notebook is run from scratch.
Encourage bad software practices (hacking) – interactive environments tend to do this, not specific to notebooks.
Require installation of extensions to enable additional functionality – this is the modern plugin mindset.
Developers don’t ship them with a requirements.txt file – this course just did.
No testing or linting built-in – true but is readily addressed (see next slides).
Useless code completion – move to jupyter lab. …
Talk is

3. Maintaining Your Notebooks
Notebooks are code and prose.
Use a version control system (git, hg, svn).
Test them, at least execution (pytest, nose2, unittest).
Notebooks are not pure code, they may also contain execution results.
Commit clean notebooks to avoid messy history and an unnecessarily large repository.
Use a continuous integration service to run your tests (CircleCI, Travis CI, Appveyor).

4. SimpleITK Notebook Development and Testing
One authoritative repository: github.com/InsightSoftwareConsortium/SimpleITK-Notebooks
No direct commits to this repository, only pull requests.
Code review and test (pytest) pull requests:
Static: (1) ensure notebooks contain no output; (2) spellcheck all markdown/prose cells and comments in code.
Dynamic: (1) Run the notebook using the jupyter nbconvert tool and analyze result (notebook in JSON format). Mark code cell meta-data: simpleitk_error_allowed (may or may not happen), simpleitk_error_expected; (2) regression testing – To Do.
Deal with memory constraints on build machines.

5. SimpleITK Notebook Development and Testing
Data sharing:
Storage: (1) images on Girder data servers or on accessible web pages (raw url); (2) code repository contains data manifest file in JSON format with sha512 to validate download integrity.
Retrieval: Cache images locally. (1) without internet access, bulk download script; (2) with internet access, lazy download.
For additional details see: “SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research”, 31(3): , 2018.

6. Join the Community
We welcome questions/requests/contributions from the community:
Ask questions (itk discourse)
Request features / examples.
Report bugs.
Contribute code or new notebooks.
github.com/SimpleITK/SimpleITK
and
github.com/InsightSoftwareConsortium/SimpleITK-Notebooks