1. October 10-11, 2002 Houston, Texas Erik DeBenedictis William McLendon Mike Carifio Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000. SciDAC SSS and Testing

2. Welcome to Texas

3. Testing Vision Create a machine-independent system for testing supercomputers –Infrastructure (QMTest selected) –Test ™ (from all sources) Value –Each time somebody creates a supercomputer today, they just test a few things –Improved method would be to execute the “SciDAC SSS Standard Test Body”

4. Recent Activity Hierarchical Test Structure QMTest on Sandia SciDAC Cluster Test “Package” Definition –Proposal Tomorrow Inventory of Tests –Public, ASCI Red, Cplant, …

5. Test Architecture QMTest Interpret RAW Test Output Report Results Expected Results... Test packages... Make Scripting

6. Test Architecture QMTest Interpret RAW Test Output Report Results Expected Results... Test Packages... QMTest Make

7. QMTest Review Scriptable test driver (Python) Test metadata defined with XML HTTP based interface (Zope) HTML Contents are customizable Open Source Active development community URL: –http://www.codesourcery.com/qm/qmtest

8. Running QMTest Must have Python >= 2.1 installed! Using HTTP Gui On Sandia SciDAC cluster: –SSH Port forwarding ssh -L4000:scidac-service.sandia.gov:4000 scidac- service.sandia.gov –Command Line qmtest gui --address scidac-service --port 4000 --no-browser

9. Reporting results to QMTest QMTest requires an exact match on ALL of the following: –Exit code Matching –STDOUT matching –STDERR matching QMTest calls a script which runs the test and then processes the output –Returns PASS if the results fit correct pattern –Otherwise returns the raw output of the test

10. Example QMTest XML File Tests saved as XML Easy to make a generator... Adding Tests is easy –Add test files to directory and QMTest will find it.

11. QMTest Screenshot

14. Interpreting the Raw Results The raw results need to be interpreted to determine PASS or FAIL Make calls scripted routines to process the output and determine if a test run passed or failed. –Retains a history of results –reports this in a tabular format

15. Expected Results Expected Results are Test specific! Different Kinds of Results –Correctness of execution –Performance comparison Versus historical execution? Discover hardware faults? Bottlenecks in a system?

16. Publicly Available Tests MPI Tests BLACS –Scalapack –PBLAS Parkbench? Netlib.org Stream memory benchmark NAS Family –Single node NAS kernel –NAS Parallel Livermore Loops –single node tests Many more available!

17. Transition to Mike

18. Terms package: a “source code unit” –an archive for sources (usually a tarball) –a makefile describing useful actions like compiling and linking executables framework lite: –a directory layout of packages –a convention for makefiles, supporting scripts treebuilding: –using makefiles recursively –using gmake to propagate actions into each package

19. Framework Directory Layout Tests/# all tests packages/# all packages nas-parallel/# wraps NAS para bnchmrks. templates/# all templates in the framework new-template/ # initial files for a new package

20. Package File Layout nas-parallell# every package has its own directory |-- GNUmakefile# walks the tree, specifically package/GNUmakefile |-- Makefile -- package# package directory “wraps” src directory |-- GNUmakefile |-- Makefile |-- bin# package specific supporting scripts, framework glue | |-- cluster.sh | |-- conf.sh |-- defaults.mk# shared by package/GNUmakefile and package/src/GNUmakefile |-- doc# package specific notes |-- made# where output (.o,.a,.so, etc) is placed | |-- current | | | `-- Linux | | | `-- built | | | `-- GNUmakefile | | `-- latest | | `-- Linux | | `-- built | | `-- GNUmakefile |-- src# sources always in here | |-- GNUmakefile | |-- HelloWorld.c | |-- HelloWorld.dep | |-- HelloWorld.o | |-- README.txt

21. Use GNU Make venerable GNU make drives the process: –leverage existing assets (makefiles) –leverage existing developer skills –all actions described in a single file (almost) –automated dependency processing –recursive makefiles well understood

22. Adding a Package $ cd Tests $ cp –r new-template nas-parallel-new $ tar –C nas-parallel-new/package/src src-archive.tar $ vi new-package/package/src/makefile # optional $ make build # now includes nas-parallel-new package Adding new packages to the framework is easy. Package internals left up to author.

23. Framework “Interface” The framework expects: –a specific package directory layout (to know where to find things) –a GNUmakefile that implements the “methods” Consistency matters: –need to put files in the right place, esp. for recursion –each GNUmakefile should implement all targets (but defaults make it easier)

24. Package Interface Follow the gmake conventions –targets: all, install, uninstall, check, … –variables: prefix, exec_prefix, bindir, … Writing/adapting a package’s makefile means learning the conventions and adding new ones when needed. (http://www.fsf.org/manual/make- 3.79.1/html_chapter/make_14.html#SEC116)

25. Added to GNU Conventions targets: –build synonym for ‘all’ –runsynonym for ‘check’ –resultsrun tests, generate a report variables: –SZnumber of nodes to use in test run –MAXSZtotal nodes available in cluster –OSoperating system –ARCHmachine architecture –MODELmachine model –CLUSTER cluster’s name –RTE_VERSION runtime environment version