Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL Trilinos SQA Training Robert Heaphy October 16, 2003

Slide 2 ASCI Program Quality Policy ASCI Program Plan, DOE/DP DOE Notice 203.1, Software Quality Assurance DOE Order 414.1A Quality Criteria, QC-1, NNSA/AL ASCI Software Quality Engineering: Goals, Principles, and Guidelines DOE/DP/ACI-SQE-2000-FDRFT-VERS2 (commonly called the Trilab GP&G) Sandia National Laboratories ASCI Applications Software Quality Engineering Practices (A version has appeared for each assessment year)

Slide 3 Sandia Corporate Quality Policy DOE Order 414.1A 10 CRF 820 Quality Criteria, QC-1, NNSA/AL (Sandia) Corporate Quality Assurance Program CPR (Sandia) Corporate Software Quality Assurance CPR QC-2 (which specifically allows the use of QC-1 instead)

Slide 4 Sandia Quality Home Page Pubic Law (Federal agencies should now use non government standards) DOE order accepting the use of non DOE standards ISO 9000 (Standard of choice)

Slide 5 Other Quality Standards at Sandia CMM (DOD) replacing MilSpec 2167A + TSP, PSP, P-CMM… CMMI (to replace obsolete CMM) Baldrige

Slide 6 Last Audit Results Final report being written 3 weeks ago and is not yet available. Trilinos and Zoltan did well! Primary problem was missing processes (such as training) Next audit may have changes, but no indications of future directions yet Management is concerned about next ASCI external audit (FY 2005?)

Slide 7 Trilinos SQE Program Assumption: There is a true and significant component of research in Trilinos. Assumption: Professionally, you already know how to perform your job well and are doing it. –You don’t need 400+ practices in CMM or 618+ pages of CMMI! –You need to get credit for what you already do. This is a generic quality program based on QC-1. GOAL: To continue a successful quality program which is minimally intrusive to developers and provides tangible benefits the development team. KEY: Document our existing work practices using lightweight “process checklists”, then follow the checklists.

Slide 8 Changes for this next audit cycle Improve, complete and use “process checklists”. Implement checklist tracking (team leaders). Suggest the use of laboratory notebooks & project notebooks. Write new version of Trilinos (& Zoltan) SQE/SQA documentation (Heaphy). Each package team will perform a Trilinos self assessment. Future changes (TBD): – traceability from requirements to maintenance (QFD, perhaps). – continuous process improvement (PDCA/PDSA).

Slide 9 Suggestion Avoid negative vocabulary in s, documents, CVS comments, etc. Specifically, think how ASCI management (who may not understand programmer jargon) might react to terms like: hack, trick, and work around.

Slide 10 Process Checklists Include necessary boilerplate from Trilinos SQA examples Divide each checklist into 5-10 key activities. Provide a simple explanation of the activity and what is to be accomplished. Incorporate creating and saving appropriate artifacts Incorporate Peer Reviews and double checks Incorporate standard tools (Bugzilla, Doxygen, Bonzai, Purify, etc) Incorporate a few, simple metrics as appropriate Incorporate error proofing (future) Maintain master checklist under CVS control Maintain specific instantiation under CVS control Don't commit yourself unnecessarily! (Conduct code review vs conduct Fagan Review. Is a team code review invalid if a single team member is missing? Do you really want require a "Coding Style Guide"?)

Slide 11 Process Checklists - 2 Follow the checklist every time. If not –Change the checklist through a formal process. Do this to add improvements, remove ambiguity, modify processes, etc. –Obtain a formal waiver (please don't really do this!) –Document procedure changes in lab notebook for R&D activities per QC-1! (A changed procedure must be signed and dated) –Otherwise, failing to follow the checklist is a significant problem and a potential audit finding. NEW Track using spreadsheet/database (saved in CSV (ASCII) text for CVS)

Slide 12 New Training Checklist -1 New Developers (within 1 month of start) Read "An Overview of Trilinos" Read "Trilinos Users Guide" Read "Trilinos Developers Guide" Read "Trilinos Developers Guide, Part 2, ASCI SQA" Read Package Developers kit Mentor Assignment with initial 2 hours mentoring (provide dates/times with mentor name):

Slide 13 New Training Checklist -2 All Developers (three year continuous training cycle) Over each three year cycle, each developer must receive and document the indicated training hours. One hour credit is given for an hour of formal class attendance, conference attendance, team/project training, mentoring (either as mentor or student). One hour credit is given for participation in the monthly Trilinos phone conferences. Three hours credit will be given for a published paper (author or coauthor), conference presentation, or for each hour of instructing a relevant training course. Each developer will list below at least the required hours of training and maintain supporting documentation. –Math and Algorithms training (60 hours) –SQE, SQA training (15 hours) –Development tools training (5 hours)