1. Ontology and V&V Ursula Martin, Bill Rider, Carol Woodward, Kate Evans, Sarah Michalak, Sergey Formel, Blake Barker, John Bprwien, Ron Boisvert, David Bayley Nick Hengartner

2. We require to foster credibility in our application of software We have a habit in writing articles published in scientific journals to make the work as finished as possible, to cover up all the tracks, to not worry about the blind alleys or describe how you had the wrong idea first.... So there isn't any place to publish, in a dignified manner, what you actually did in order to get to do the work. — Richard Feynman (Nobel acceptance 1966) Extraordinary claims require extraordinary evidence. --- C. Sagan, Billions and Billions, 1998, p 60.

3. Taxonomy of types of reproducibility What does reproducibility mean, and what are its deisrata Science that can be independently replicated and validated by the community. Given a code, you turn a crank, but are you independently reproducing the result? Levels of reproducibility. There is a cultural environment on how we report results. Publish or perish put pressure on publishing positive (statistical significant) results. Sometimes negative results are useful. Example: James Quirk paper on Riemann solver.

4. Hierarchy of levels of reproducibility Hierarchy of levels of claims Referees and editors can expect authors have done due diligence in verifying claims. Levels of reproducibility. Algorithm without details; use package (with version) Replication of the result from executable, plus examples of claims + full description of algorithms and all special cases Executable assessed on standard test cases. Full source code (for relevant sections) Scripts, input files

5. Outcome Field is beyond reproach, “High Integrity by Design”. Examples: – Japanese research on stem cells – Performance claims of particular algorithms on parallel computers Failure to do so will impede progress in a field Establish and deserve trust of public Develop chain of trust

6. V&V: A Methodology to help provide credibility for computational science V&V can help convince reviewers that the calculations and claimed results are valid. Goes towards increasing credibility. Verification: Provide evidence that the implementation of the mathematical equations. Validation: Assessment of the degree to which your model is simulating/computing what you intended.

7. Verification… Verification: Provide evidence that the implementation of the mathematical equations. – Verification is a prerequisite to reproducibility. – Any code user should cite or provide independent verification – Verification is never complete: the level of verification should be commensurate with intended use of the codeRegression tests (to test future modification) may be required – Post tests to verify installation, updates, and patches to ensure that the claims remain valid through software revisions. any person using a piece of software should attempts to verify that code. Special care is needed when using commercial software/insist that commercial software provide verification.

8. Validation/UQ Identify the sources of uncertainties.