1. 1 Systems Validation & Verification, Quality and Standards (CSE4431) Dr Sita Ramakrishnan School CSSE Monash University

2. © S Ramakrishnan2 zDiscussion y Overview of the unit - CSE4431 y Week-by-Week Outline y Assessment Details y Basic Testing Concepts, IEEE Definitions zCSE4431 material online at http://www.csse.monash.edu.au/courseware/CSE4431 http://www.csse.monash.edu.au/courseware/CSE4431 z Assignments handouts in class Systems Validation & Verification, Quality and Standards (CSE4431)

3. © S Ramakrishnan3 yTopic areas: xIEEE Definition - Verification & Validation xSoftware Quality Attributes xStandards xIndependent V & V xRole of V & V method in SDLC xOverview - Test Models, Test Design Patterns, Testing tools, Testing xChallenges of Testing, Purpose of Inspection & Testing Systems Validation & Verification, Quality and Standards (CSE4431)

4. © S Ramakrishnan4 xIEEE Standard of Software Engineering Terminology (1990) defines software verification as the “process of evaluating a system or component to determine whether the products of a given development phase satisfy conditions imposed at the start of the phase”. That is to say, software verification is concerned with the degree of acceptability of a product for a particular phase. xFormal verification rarely used throughout a design or for tracking defects early because of time & expertise required IEEE Definition - Verification & Validation

5. © S Ramakrishnan5 xVerification -> the process of determining whether or not the products of a given phase of software development meet all the constraints established in the previous phase (Standard Glossary of SE Terminology) xVerification is often equated to software testing in the context of commercial applications in the industry xValidation is the process of evaluating software at the end of the development process to ensure compliance with the initial requirements IEEE Definition - Verification & Validation

6. © S Ramakrishnan6 xValidation of software refers to a set of activities, which ensure that the software built is traceable to customer requirements, and meets expected results xValidation method also appropriate where verification may be difficult or not feasible (proofs may require prohibitive memory & time, providing a complet set of spec. & identifying all relevant properties may not be feasible, and not always reachable) xValidation testing is conducted to ensure that the software meets the functional, behavioural & performance requirements of the system IEEE Definition - Verification & Validation

7. © S Ramakrishnan7 xSoftware testing is one element of a generic process referred to as Verification & Validation in engineering software systems xV & V process is used to ensure that the given software conforms to its spec. & meets the needs of the customer xV & V are different activities as stated by Boehm (1984): Verification: “Are we building the product right?” Validation: “Are we building the right product?” IEEE Definition - Verification & Validation

8. © S Ramakrishnan8 xA V & V process is a product a property to be controlled on the product a technique for evaluating the property xAn Example of V & V process a product: the whole software system a property : a functional property given by –a set of input data –a set of expected results a technique: testing –comparing expected results and the actual results from the system A Verification & Validation Process & An Example of V & V Process

9. © S Ramakrishnan9 xIn theoretical Comp Sc, verification is roughly synonymous to theorem proving xResearchers in Formal methods employ formal techniques based on model checking or theorem proving in the verification of software systems xTheorem proving is a mathematical process of proving that the functions computed by the program matches its specification xVerification of correctness is treated here (in CSE4431) as the set of activities carried out to ensure that software conforms to its specification, and to correctly implement a specific function - this definition is close to Software validation testing Verification & Validation Verification & Validation

10. © S Ramakrishnan10 xSoftware quality attributes assessment (McCall, Richards & Walters 1977) 11 quality attributes in terms of 3 categories: - product operation, product revision and product transitionsoftware quality model (1977) characterised 11 quality attributes in terms of 3 categories: - product operation, product revision and product transition Product operation Product operations consist of usability, correctness, reliability, integrity and efficiency attributesProduct operation - captures how effective the software operations are in performing a set of tasks such as ease with which data can be input and the ease and reliability of output data. Product operations consist of usability, correctness, reliability, integrity and efficiency attributes Product revision - Product revision category includes maintainability, flexibility & testability as its 3 sub characteristics.Product revision - measures the ease of update, change & maintain the performance of the software. Important w.r.t testing & cost of fixing errors. Product revision category includes maintainability, flexibility & testability as its 3 sub characteristics. Quality issues in software systems

11. © S Ramakrishnan11 Product transition - 3 sub characteristics of the product transition category are: interoperability, reusability and portabilityProduct transition - emphasis is to consider the ease with which software can migrate to other platforms, be reusable & interoperate. 3 sub characteristics of the product transition category are: interoperability, reusability and portability Boehm introduced some more quality attributes in 1978 As software demands grow, quality attributes were looked at again ISO adopted ISO 9126 in 1991 as the standard for software quality ISO 9126 standard moves from McCall’s 3 main attributes to 6 and includes 21 sub attributes Quality issues in software systems

12. © S Ramakrishnan12 xInternational organisation for standardisation (ISO) adopted ISO9126 as the standard for software quality(ISO, 1991). It has 6 main quality attributes and subattributes: –functionality (suitability, accurateness, interoperability, compliance, security) –reliability (fault tolerance, recoverability, maturity) – usability (learnability, operability, understandability) –efficiency ( w.r.t. time & resource) –maintainability (changeability, stability, testability, analyzability) –portability (adaptability, installability, conformance, replaceability) Quality issues in software systems

13. © S Ramakrishnan13 xThe ISO’s ISO 9126 standard is now widely adopted. xOther industry standards such as IEEE have adjusted their standards to comply with the ISO standards Quality issues in software systems

14. © S Ramakrishnan14 xIEEE publ. (1988, 1996) have metrics that could be used for testing quality attributes. These metrics include fault density requirements compliance test coverage & mean time to failure xHowever, no one metric is able to unambiguously measure a specific quality attribute - various metrics may give different rank orderings for the same quality and making comparative judgements across products difficult Quality issues in software systems

15. © S Ramakrishnan15 xV & V is a life cycle process of a software development effort and is used to assess the software quality and the V&V activites are performed at the unit, module & integration levels verify: Latin for versus = true –verify objectively that the output from the development activities fulfill the requirements imposed by the previous phase of the SDLC and is internally complete, consistent & correct for supporting the next phase –was the product built correctly? validate: Value –validate objectively that the products built satisfy the customer requirements – was the correct product built? Role of V & V and Independent V & V in SDLC

16. © S Ramakrishnan16 xIndependent V & V is a V & V process Where an iterative set of activities is carried out by a contractor or an organisation other than the developer of a system xAim of IV&V is to improve the quality & reliability of the system, assure that product meets the client’s operational requirements xIndependent group doing V&V provides an objective second opinion and is used to evaluate processes such as testing & configuration management as well as software & documentation Role of V & V and Independent V & V in SDLC

17. © S Ramakrishnan17 x I V & V activities done by outsourcing or with contractors or with a separate QA team xAdvantages: Independence – as V & V performed by someone other than the developer IV&V done concurrently with development, defects/deficiencies detected and corrected early Adopts a well-planned, well-defined V&V process and uses established quality standards, best practices such as formal inspection techniques and so on Role of V & V and Independent V & V in SDLC

18. © S Ramakrishnan18 x As reported by Sandia national lab, http://www.sandia.gov/surety/IV&V/ivv.htm http://www.sandia.gov/surety/IV&V/ivv.htm xRisk analysis & mitigation planning xRisk analysis & mitigation planning – work with develoeprs during risk analysis phase (which should start early in SDLC) and reduce identified risks by specific actions xEvaluation xEvaluation through analysis & testing for performance, robustness Early detection of deficiencies x Early detection of deficiencies/defects by independent analysis and separate testing – Arianne launch failure disaster example improve safety, security & reliabilityimproved performance xIV&V can assure that software will work as expected and a well planned IV&V strives to improve safety, security & reliability of the system. Provides improved performance by recommending improvements in design, development & testing of the system A Well-Planned I ndependent V & V returns benefits

19. © S Ramakrishnan19 x V& V aimed to be a systematic & technical evaluation of products associated with the development & maintenance process xRefer to Quality Manual – ISO-9001 xTwo major V & V activities are: reviews and testing xReviews – inspection & walkthroughs xFormal reviews - at the end of each phase of LC, to check if requirements, design & specs. have been met xInspection or walkthroughs – detailed examination of artifacts with a formal review panel xTesting V & V Activities

20. © S Ramakrishnan20 The major V & V activities during xSoftware system initiation phase – how system to be reviewed & tested. Clients may be involved in generating test cases or may use test data generators Resulting in better cost & schedule estimates due to adequate plans & test capabilities xSoftware Requirements phase (SRS) – V &V to check if software requirements consistent with system requirements Assure that requirements are testable Preliminary version of acceptance test plan, include a verification matrix which relates requirements to tests – to demonstrate that requirements are satisfied Start on test data generators if required Phase ending SR Review V & V Activities SDLC phases

21. © S Ramakrishnan21 x Software Architectural (Preliminary) Design Phase - V & V here should include Update to preliminary version of acceptance test plan & verification matrix Conduct informal reviews & walkthroughs or inspections of the prelim. DB or software design Phase ending preliminary design review (PDR) where allocations of requirements to software architecture are reviewed & approved V & V Activities SDLC phases

22. © S Ramakrishnan22 x Software Detailed Design Phase - V & V here should include Completing the acceptance test plan & verification matrix including test specifications & unit test plans Conduct informal reviews & walkthroughs or inspections of the detailed DB or software designs Phase ending critical design review (CDR) which completes the software design phase V & V Activities SDLC phases

23. © S Ramakrishnan23 x Software Implementation Phase - V & V activities here should include Code inspections and/or walkthroughs Unit testing software Locate, fix and retest errors Develop detailed test plans for next 2 phases of SDLC V & V Activities SDLC phases

24. © S Ramakrishnan24 x Software Integration and Test Phase - V & V activities here should include Tested units from previous phase integrated into subsystems and then into final system Activities include –Do Tests for each test procedure –Document test completion, expected vs actual results, test performance –Locate, record, fix & retest errors/nonconformance reports –Test readiness review (TRR) to confirm readiness for acceptance testing V & V Activities SDLC phases

25. © S Ramakrishnan25 x Software Acceptance & Delivery Phase - V & V activities here should include By testing, analysis & inspection, show that the system developed meets its functional, performance & interface requirements Locate, fix & retest any non conformances Phase ending acceptance review (AR) xSoftware Operation Phase - V & V activities here should include V & V activities of the previous 7 phases done during this phase for any updates of software V & V Activities SDLC phases

26. © S Ramakrishnan26 x Testing tools used in V&V of software especially for testing complex and/or large software systems xFormal inspection is a useful technique for finding errors. Developed by Michael Fagan, IBM Inspection involves line by line auditing of the product being reviewed Structured review - applied to any product of SDLC process (design, code, documentation) - direct feedback to the author - Efficient in locating & fixing errors Inspections done by a team, each member has a specific role; a moderator trained in the inspection process leads the team; a reader leads the team through the item; one or 2 reviewers who look for faults in the item; a recorder who records the faults; and the author, who has to explain the work being inspected. V & V Activities Techniques and Tools

27. © S Ramakrishnan27 xWhat is Testing? The term “Test” is derived from the Latin expression testum, an earthenware pot used by the Romans in the quality evaluation process of materials such as precious metal ores (Oxford English Dictionary). xSoftware Testing is done to: check that a program conforms to its specification uncover software defects ensure it functions correctly & does not produce unexpected results check robustness - how far can it be pushed before it fails badly understand the risks involved in software releases to customers An Overview Challenges of Testing An Overview Challenges of Testing

28. © S Ramakrishnan28 x“Testing is an activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results” (Hetzel, 1988). this testing approach known as positive testing x“Testing is the process of executing a program or system with the intent of finding errors” (Myers, 1979). this approach is known as negative testing as it looks for defects outside the scope of requirements to cover incomplete requirements & inaccurate specification xIn practice, testing combines both positive & negative testing elements xDefining testing in terms of risk has become more popular. xRisk relates to AUT (Appl. under test) An Overview Testing (More Definitions) An Overview Testing (More Definitions)

29. © S Ramakrishnan29 x How do we plan for software testing? x When do we perform software testing? x How do we determine the test criteria and perform software testing? x How do we determine that the test outputs are correct (test oracle)? x When do we stop testing (test adequacy)? Fundamental Testing Questions