1. Erik P. DeBenedictis Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000. Testing Integration and Validation Group

2. Agenda Hardware Inventory –Oak Ridge –Sandia –Berkeley –Argonne Intellectual Property –Process Documentation Standards –TEX –HTML –Embedded Bug tracking –Bugzilla Test Framework –Deja Gnu –QMTest

3. Hardware Berkeley –Hardware 32 nodes P II 400 –Software Redhat 6.2 Argonne –Hardware On demand 256 nodes P III 500s myrinet –Software Redhat 7.1 Oak Ridge (posted in notebook) –Hardware 4 dual 450 with 1 gig ram X Torque 64 nodes 2 gig –Fast ethernet + gig ethernet –Software Oscar install 1.2.1 Sandia –Hardware 8 Alpha + admin node –Software Cplant

4. Servers Ames –CVS Directory E-mail Brett Argonne –Bitkeeper E-mail Narayan –Bugzilla E-mail Narayan Sandia –CVS E-mail Erik

5. Intellectual Property Process –Fred (DOE) requests Open Source –Labs “Assert Copyright” on Open Source License What license Needs to be same –Contribute Source –Publish Source

6. Documentation Standards Objectives –Set standards for English Language contributions to group’s product HTML –(Doesn’t paginate well -- Hargrove) Rusty’s TeX Macros –(Not browsable? -- Hargrove) –(Paginates well Jackson) Microsoft Word –(Violent objection to this one) ASCII What types of documentation? –Design documents –User guide –Man pages –XML Schema specs –Installation guide –Administration guide –Reference guide (optional) Format requirements –Source format Html, tex –Distribution format Pdf, html

7. Bugtracking Bugzilla –Argonne has bugzilla installed and can create a branch for SciDAC

8. Test Harness The T-I-V-X group investigated test harness products –Selected Deja GNU (but did not review QMTest) Sandia investigated test harness products for Cplant –Selected QMTest The SciDAC SSS group finds QMTest interesting and support for Deja GNU has withered

9. What is QMTest? Scriptable (Python) test driver Test metadata (results) defined with XML Web-based interface Extensible webserver built in (Zope) HTML contents customizable New test types through inheritance More: http://www.codesourcery.com/qm/qmtest

10. Why QMTest? Opensource Scriptable (you can extend it) Object-oriented Declarative results in XML Web-based UI supports decentralized dev. Active and growing development commun. Good “separation of concerns” design

11. How Does Cplant Use It? Mostly as intended Subclass ExecTest Declare test results using UI right now Drive cases sequentially. PBS integration? We’re coming up the learning curve…

12. How QMTest Looks

13. Example Test Report

14. Step 1: Define a New Test

15. Step 2: Describe It

16. Who’s Doing It Lead: Sue Goudy spg@sandia.govspg@sandia.gov Grunt: Mike Carifio carifio@usys.comcarifio@usys.com Carifio is ½ time…

17. When Did We Do This? 12/01 Carifio invest. test frameworks 1/02 Goudy, Carifio select QMTest Q1/02 Goudy,Carifio learn Python, QMT. Q2/02 Carifio gathers test “packages”, implements “build framework” Q3/02 Drive packages with QMTest Q4/02 UI mods? PBS integration?

18. Conclusions QMTest is scriptable driver Tests are defined in XML UI is web-based, content generated New test kinds derived from base classes Best price Active development Seems to be “best in class” today