1 Systems Validation & Verification, Quality and Standards (CSE4431) Dr Sita Ramakrishnan School CSSE Monash University

© S Ramakrishnan2 xTopics : x Relationship of testing to other activities: x Testing vs debugging x testing within SQA; within Cleanroom x Testing & certification xhttp:// Systems Validation & Verification, Quality and Standards (CSE4431)

© S Ramakrishnan3 xCertification - formal recognition of the level of proficiency in the IT quality assurance industry xRecipient deemed to have the grasp of the discipline & skills in a BOK for a respective discipline – whether it be PM, QA, Testing … xQuality Assurance Institute (QAI) established as a professional association in 1980s to represent the QA profession. xFirst Software certification in Formal exam process was launched in 1990s. xQAI certification program is global ~6000 certified Software Certification

© S Ramakrishnan4 x Certified Software Quality Analyst (CSQA) x Certified Software Test Engineer (CSTE) x Certified Software Project Manager (CSPM) xhttp:// x Others offered by other organisations around the world xSoftware Quality Engineer Certification (CSQE) Body of Knowledge xIn Europe, CSTE becoming mandatory x Recently (in 2002), Software Engineering Australia launched a similar program in OZ Certification supported by QAI

© S Ramakrishnan5 xTaking the Art out of Software Development – An In-Depth Review of Cleanroom Software Engineering x xAlso refer to text book – Sofwtare Engineering – J F Peters & W Pedrycz xError removal Software Development method x CSE focuses on theory-based engineering practices of software development and certification, while using CMM as its framework (Oshana, 1998). x CSE has SD life cycle – 80% time for design & 20% for writing code. In traditional SDLC, 40% design, 20% code, 20% test. In CSE, no unit testing as part of the CSE process (Head, 1994). The untested software product is expected to work. x If errors found, defects analysed to see how to improve the process. Cleanroom Software Engineering

© S Ramakrishnan6 x Principal characteristics of CSE incremental development independent assessment through verification-based inspection of increments statistical testing after delivery of verified increments to certification team x 4 components to CSE Process incremental development process model, formal methods for specification and design, correctness verification of developed code, and statistically based, independent testing x Where it differs from traditional approaches No testing activity by the programmer, the programmer must do formal (measurable) verification. Lastly, all testing must be statistical in nature. Cleanroom Software Engineering

© S Ramakrishnan7 x The testing principle focuses on certifying the reliability of software product as a measure of the quality. x This technique uses statistical testing based on expected system usage rather than trying to remove defects. 1) all possible software uses are generated into statistical subsets Performance is measured on these subsets based on the system usage model developed 2) the expected operational use is represented in a usage model 3) test cases are randomly generated and executed in the operational environment. Failures interpreted as per statistical models x (refer text for details) Cleanroom Software Engineering

© S Ramakrishnan8 x This testing is viewed as a sampling problem in CSE versus coverage testing for debugging in the traditional approach x The independent testing team does not assist the development group in getting the software product to an acceptable level of quality, but rather certifies its reliability Cleanroom Software Engineering

© S Ramakrishnan9 y Disadvantages w.r.t using CSE: y A Perception that y CSE is too theoretical y too mathematical y too radical for software development. y Even the correctness proof portion of CSE can be difficult and time consuming especially for non-safety-critical programs Cleanroom Software Engineering

© S Ramakrishnan10 y Although CSE is not widely accepted, industries such as aerospace, telecommunications, graphical, and CASE tools have implemented CSE successfully (Deck, 1997, May 27-30). y How CSE could be “phased-in” to develop better COTS software (Wolak 2001) Typically, COTS software relies on updates and patches for errors to be fixed; CSE process would benefit COTS vendors COT vendors could replace beta testers with CSE process of teams reviewing program code (i.e.verification of correctness) Followed by statistical testing on the marketable version Introducing reliability measurements can be the next step after statistical testing -proactively find errors instead of relying on consumer usage After errors are fixed, vendor can continue to provide software updates & patches Cleanroom Software Engineering