Erik P. DeBenedictis Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL Testing Integration and Validation Group

Agenda Hardware Inventory –Oak Ridge –Sandia –Berkeley –Argonne Intellectual Property –Process Documentation Standards –TEX –HTML –Embedded Bug tracking –Bugzilla Test Framework –Deja Gnu –QMTest

Hardware Berkeley –Hardware 32 nodes P II 400 –Software Redhat 6.2 Argonne –Hardware On demand 256 nodes P III 500s myrinet –Software Redhat 7.1 Oak Ridge (posted in notebook) –Hardware 4 dual 450 with 1 gig ram X Torque 64 nodes 2 gig –Fast ethernet + gig ethernet –Software Oscar install Sandia –Hardware 8 Alpha + admin node –Software Cplant

Servers Ames –CVS Directory Brett Argonne –Bitkeeper Narayan –Bugzilla Narayan Sandia –CVS Erik

Intellectual Property Process –Fred (DOE) requests Open Source –Labs “Assert Copyright” on Open Source License What license Needs to be same –Contribute Source –Publish Source

Documentation Standards Objectives –Set standards for English Language contributions to group’s product HTML –(Doesn’t paginate well -- Hargrove) Rusty’s TeX Macros –(Not browsable? -- Hargrove) –(Paginates well Jackson) Microsoft Word –(Violent objection to this one) ASCII What types of documentation? –Design documents –User guide –Man pages –XML Schema specs –Installation guide –Administration guide –Reference guide (optional) Format requirements –Source format Html, tex –Distribution format Pdf, html

Bugtracking Bugzilla –Argonne has bugzilla installed and can create a branch for SciDAC

Test Harness The T-I-V-X group investigated test harness products –Selected Deja GNU (but did not review QMTest) Sandia investigated test harness products for Cplant –Selected QMTest The SciDAC SSS group finds QMTest interesting and support for Deja GNU has withered

What is QMTest? Scriptable (Python) test driver Test metadata (results) defined with XML Web-based interface Extensible webserver built in (Zope) HTML contents customizable New test types through inheritance More:

Why QMTest? Opensource Scriptable (you can extend it) Object-oriented Declarative results in XML Web-based UI supports decentralized dev. Active and growing development commun. Good “separation of concerns” design

How Does Cplant Use It? Mostly as intended Subclass ExecTest Declare test results using UI right now Drive cases sequentially. PBS integration? We’re coming up the learning curve…

How QMTest Looks

Example Test Report

Step 1: Define a New Test

Step 2: Describe It

Who’s Doing It Lead: Sue Goudy Grunt: Mike Carifio Carifio is ½ time…

When Did We Do This? 12/01 Carifio invest. test frameworks 1/02 Goudy, Carifio select QMTest Q1/02 Goudy,Carifio learn Python, QMT. Q2/02 Carifio gathers test “packages”, implements “build framework” Q3/02 Drive packages with QMTest Q4/02 UI mods? PBS integration?

Conclusions QMTest is scriptable driver Tests are defined in XML UI is web-based, content generated New test kinds derived from base classes Best price Active development Seems to be “best in class” today