1. Writing Quality Specifications July 9, 2004 Mark Skall Acting Director, Information Technology Laboratory National Institute of Standards and Technology mark.skall@nist.gov TGDC Meeting

2. Background NIST works with industry and Federal agencies to develop standards and tests to improve the quality of software and achieve interoperable solutions NIST has many years experience with –Formal standards organizations and Consortia (W3C, OASIS, ANSI, ISO, IEEE, HL7) –Helping to develop good specifications –Developing conformance test suites, tools, reference implementations –Developing validation and certification testing programs

3. Conformance Requirements Testing activities Specification Implementation Moving Towards Trust and Confidence

4. Good Specifications are the Key Goal is correct, reliable software Requirements are captured in a specification Spec must use appropriate language to designate requirements Spec must be precise, unambiguous, and testable Spec must contain a conformance clause Ideal spec would be defined in a formal language – not English

5. Specs Must Use Appropriate Language to Designate Requirements “MUST”, “MUST NOT”, “SHALL”, “SHALL NOT” are examples of mandatory requirements “SHOULD”, “SHOULD NOT”, “MAY”, “RECOMMEND” are not mandatory, but are often used. If used, they have no bearing on conformance “Good enough”, “Do the best you can” is not appropriate language. They are not precise, and can’t be tested

6. Specifications Must Be Precise A requirement is precise if one can determine if the requirement has been fulfilled An imprecise requirement: Systems shall guard against viruses, trojan horses, and worms. A precise requirement: An anti-virus program (from an approved list) shall be installed on one’s system and anti-virus software checks and live updates shall be run daily.

7. Specifications Must Be Unambiguous “The girl touched the cat with a feather” –Meaning: (a) The girl touched (the cat with a feather) or (b) (The girl touched the cat) with a feather Does the girl or the cat have a feather?

8. Specs Must Be Testable If a requirement cannot be tested (i.e., a method cannot be devised to determine whether a particular requirement is met) one cannot verify that it has been satisfied –Reformulate it so it can be tested or –Eliminate the requirement from the standard Testable assertions will be derived from normative statements in the specification (“must”, “must not”, “shall”, “shall not”...) Unspecified, ambiguous, or imprecise requirements cannot be tested

9. Conformance Clause Conformance is the fulfillment of a product, process, or service of specified requirements A conformance clause is a high-level description of what is required of implementers and application developers. Its a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance to the specification. –Who (what types of entities) may conform –How - What must they do to conform? –It may, for conformance purposes, refer to functional subsets, such as profiles, levels, or other structures. –Additionally it should specify the permissibility of extensions, options, and alternative approaches and how they are to be handled.

10. What Makes a Good Spec NIST is working with W3C to define how a spec should be written –Addresses what a spec should contain with respect to conformance and testability Addresses: –How to define conformance –How to sub-divide a spec –Discretionary items –Extensions –Test assertions

11. Relevant NIST Efforts Guidance on writing better specifications –W3C Quality Assurance effort –OASIS Conformance Requirements for Specs documentConformance Requirements for Specs –Conformance advisory Test methods and techniques –Automatic Test Generation from formal specification –Automatic Test Generation using XML Technologies –Software Component Integration Testing –Adaptable Conformance Test Method –Develop XML conformance tests National Software Reference Library (NSRL) to determine whether software has been altered

12. Contact For more information, please contact: Mark Skall mark.skall@nist.gov 301-975-3262 http://www.itl.nist.gov/div897/