1. Automatic Testing in Madagascar Jim Jennings Houston, Texas July, 2010

2. Test Everything! 2

3. Outline Madagascar testing purposes, goals, methods, and components Give it a try! 3

4. Madagascar Testing Purposes Regression testing for developers Regression testing for users Reproducibility testing Data flow testing Computational example comparisons 4

5. Madagascar Testing Goals Make it convenient Make it automated Make it flexible Make it usable 5

6. Madagascar Testing Goals Make it convenient Make it automated Make it flexible Make it usable Make it fun! 6

7. Madagascar Built-in Test Base Over 300 computational examples in $RSFSRC/book Over 3,000 figures in a separate repository, https://rsf.svn.sourceforge.net/svnroot/rsf/figs https://rsf.svn.sourceforge.net/svnroot/rsf/figs The basic Madagascar testing method is to run the examples and compare the resulting figures to those downloaded from the repository. 7

8. … But it’s not as simple as it sounds Some of the examples require external data. Some of the examples require components that might not be installed. Some of the examples take a long time to run. Some of the examples use a lot of disk space. Round off differences make some examples not exactly reproducible. 8

9. … But it’s not as simple as it sounds Some of the examples require external data. Some of the examples require components that might not be installed. Some of the examples take a long time to run. Some of the examples use a lot of disk space. Round off differences make some examples not exactly reproducible. We need an easy user-adjustable way to select examples for routine testing. 9

10. Madagascar Examples Metadata Metadata stored for each example in an.rsfproj file: Madagascar tools used External data used Total disk space used Metadata for $RSFSRC/book examples stored in repository. Metadata generated automatically as part of “scons lock” operation. You can store metadata for your own examples. 10

11. Madagascar Testing Components sfbooklist Make an inventory of examples Select examples according to optional filters Run an optional command for each selected example sffiglist Make an inventory of figures in one example List and display figures that are different or missing sftestlist Make a summary of test results $RSFSRC/admin/quick_test.csh Do everything for a pre-selected collection of examples 11

12. sfbooklist Try these in $RSFSRC: > sfbooklist book Make an inventory of $RSFSRC/book > sfbooklist levels=2 book/geostats Make an inventory of $RSFSRC/book/geostats > sfbooklist levels=2 command=ls book/geostats Run a simple command in each $RSFSRC/book/geostats/*/* > sfbooklist levels=2 command=scons book/geostats Build each $RSFSRC/book/geostats/*/* example 12

13. sffiglist Try these in $RSFSRC/book/geostats/simulate/rfield: > sffiglist list=all Inventory of all figures > sffiglist list=miss List only missing or different figures --- Delete a figure and try it again --- Change something in SConstruct and try it again > sffiglist list=miss show=miss Show the different figures > sffiglist list=miss rsftest=y Save test results in an.rsftest file 13

14. sftestlist Try these in $RSFSRC: > sfbooklist levels=2 command="sffiglist rsftest=y" \ book/geostats Compare figures in each $RSFSRC/book/geostats/*/* example > sftestlist book Make summary of tests in $RSFSRC/book/*/*/* > sftestlist levels=2 book/geostats Make summary of tests in $RSFSRC/book/geostats/*/* 14

15. $RSFSRC/admin/quick_test.csh Try this in $RSFSRC: > admin/quick_test.csh 15

16. Test Everything! 16