Verification and Validation: A Quick Introduction Authors Massood Towhidnejad Massood Towhidnejad Mike Rowe Mike Rowe David Dampier David Dampier Sponsored In Part by NSF Grant EEC And published on

Learning Objectives 1.The student will be able to define both verification and validation, and to know the difference between them. 2.The student will be able to identify the appropriate activities and their classification as either verification or validation. 3.The student will be able to distinguish between verification and validation, given a set of activities. 4.Given a set of software artifacts, the student will understand the appropriate verification or validation activity to that artifact. 5.Given a set of projects in different categories, the student will be able to determine which category of projects would require more V&V activity, and which would require less. 6.Given a set of defects and the phases of development in which they were found, the student will be able to estimate the relative cost of correcting errors early or late in the lifecycle.

Why Invest in Quality? Cost effective Cost effective Provides competitive edge Provides competitive edge Essential for business survival Essential for business survival Essential for international marketing Essential for international marketing Helps to retain customers, and increase profits Helps to retain customers, and increase profits Hallmark of world class business Hallmark of world class business

Ways to Improve Quality Prevention of Defects Prevention of Defects Process ImprovementProcess Improvement Complexity ReductionComplexity Reduction Risk ManagementRisk Management Causal AnalysisCausal Analysis Detection and Correction of Defects Detection and Correction of Defects VerificationVerification ValidationValidation ReworkRework Causal AnalysisCausal Analysis

Verification Verification – Are we building the product right? Verification – Are we building the product right? Verification is any checking process conducted on software artifacts in an attempt to determine if they work as specified by the designers of the system. Verification is any checking process conducted on software artifacts in an attempt to determine if they work as specified by the designers of the system. Includes reviews, inspections, walkthroughs, unit testing and integration testing. Includes reviews, inspections, walkthroughs, unit testing and integration testing.

Validation Validation – Are we building the right product? Validation – Are we building the right product? Validation is the process of evaluating software artifacts during the software development process in an attempt to determine if the system works as required by the customers. Any evaluation activity that involves the customer can be used for validation purposes. Validation is the process of evaluating software artifacts during the software development process in an attempt to determine if the system works as required by the customers. Any evaluation activity that involves the customer can be used for validation purposes. Includes program reviews, system testing, customer acceptance testing. Includes program reviews, system testing, customer acceptance testing.

Verification vs. Validation Verification Verification Main purpose is to detect defects in the artifacts of the system under development.Main purpose is to detect defects in the artifacts of the system under development. Validation Validation Main purpose is to show that the system under development meets user needs, requirements, and expectations.Main purpose is to show that the system under development meets user needs, requirements, and expectations.

Verification & Validation Techniques Static Methods Static Methods Techniques applied to artifacts without execution.Techniques applied to artifacts without execution. Dynamic Methods Dynamic Methods Techniques applied to artifacts through execution.Techniques applied to artifacts through execution. Mathematically Based Methods Mathematically Based Methods

Static: Reviews Walkthroughs Walkthroughs CodeverificationCodeverification DocumentDocument ConOps, SRSvalidation ConOps, SRSvalidation STEP, SAD, SDDverification STEP, SAD, SDDverification Inspections Inspections CodeverificationCodeverification Document AuditsverificationDocument Auditsverification Program Reviews Program Reviews Customer involvedvalidationCustomer involvedvalidation No customerverificationNo customerverification

Effectiveness of Static Verification More than 60% of program defects can be detected by program inspections. More than 60% of program defects can be detected by program inspections. More than 90% of program defects may be detectable using more rigorous mathematical program verification. More than 90% of program defects may be detectable using more rigorous mathematical program verification. The defect detection process is not confused by the existence of previous defects. The defect detection process is not confused by the existence of previous defects.

Dynamic: Testing (Verification) Unit Test (Detailed Design): Testing the individual software modules, components, or units. Integration Testing (Architectural Design): After unit test, the system is put together in increments. Integration testing focuses on the interfaces between software components (OO thread-based, cluster-based testing) System Testing (Requirements Spec): One goal of system testing is to ensure that the system functions as specified in the specification.

Dynamic: Testing (Validation) System Testing (Requirements Spec): Another goal of system testing is to ensure that the system functions as the client expected in a controlled environment. User Acceptance Test (ConOps): A set of formal tests run for the client, and specified by the client. When the system passes these tests, the software has been accepted by the client as meeting the requirements.

Verification and Validation in the Development Lifecycle Code Execute Unit Tests White-Box & Black-Box Testing Black-Box Testing Requirements Analysis Execute System Tests Execute Integration Tests Design Validate the System Verify Design Verify Implementation Reviews

Mathematics-Based Verification Verification is based on mathematical arguments which demonstrate that a program is consistent with its specification. Verification is based on mathematical arguments which demonstrate that a program is consistent with its specification. Programming language semantics must be formally defined. Programming language semantics must be formally defined. The program must be formally specified. The program must be formally specified.

Why Inspection is important? Relative cost to fix a defect Relative cost to fix a defect Phase in which found Cost ratio (hours) Phase in which found Cost ratio (hours) – Requirement 1 – Design 3-6 – Coding 10 – Testing – Operation Data derived from Capers Jones.

Costs of Finding and Fixing Late Delays in identifying and fixing defects gets geometrically more expensive as the lifecycle progresses! $1 $10 $100 $1000 Cost of Fixing a Defect Defects Time in (Phase of) Development

Cost of Quality Includes all costs of quality-related activities. Includes all costs of quality-related activities. Quality costs = Quality costs = +Prevention costs +Detection and Appraisal costs +Failure costs Internal failure costs Internal failure costs External failure costs External failure costs Cost of Quality (COQ) – refers to the cost of correcting defects once found. Cost of Quality (COQ) – refers to the cost of correcting defects once found.

Quality Cost Components Direct Cost Direct Cost – Reviews/inspections – Unit testing – System testing – Acceptance testing – Test planning and design design – Computer time – Resources (terminals, staffs, etc.) staffs, etc.) Indirect Cost Indirect Cost – Rework – Recovery – Corrective action cost – Failures – Analysis meeting – Debugging – Retesting – Legal fees

Verification or… Validation? Reviews Reviews Unit Testing Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing

Verification or… Validation? ReviewsEither ReviewsEither Unit Testing Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing

Verification or… Validation? ReviewsEither ReviewsEither Unit Testing Verification Unit Testing Verification Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing

Verification or… Validation? ReviewsEither ReviewsEither Unit Testing Verification Unit Testing Verification Integration Testing Verification Integration Testing Verification System Testing System Testing Acceptance Testing Acceptance Testing

Verification or… Validation? ReviewsEither ReviewsEither Unit Testing Verification Unit Testing Verification Integration Testing Verification Integration Testing Verification System Testing Validation System Testing Validation Acceptance Testing Acceptance Testing

Verification or… Validation? ReviewsEither ReviewsEither Unit Testing Verification Unit Testing Verification Integration Testing Verification Integration Testing Verification System Testing Validation System Testing Validation Acceptance Testing Validation Acceptance Testing Validation

Balancing Trade-Offs What is the real requirement? What is the real requirement? What are the expected benefits of V & V vs. What are the expected benefits of V & V vs. – cost of increasing V&V activity – additional time required – technical feasibility – risk of not finding defects » delivery » operation “When is it good enough?” (Bach, 1997)

Summary Verification and Validation are both important to the quality of software. Verification and Validation are both important to the quality of software. Each is different: Each is different: Verification determines that what we are doing is done correctly.Verification determines that what we are doing is done correctly. Validation determines whether we are doing the right thing.Validation determines whether we are doing the right thing. Sometimes, less quality is okay. Some software systems require less V&V, as the cost is not supportable. Some require more no matter what it costs. Sometimes, less quality is okay. Some software systems require less V&V, as the cost is not supportable. Some require more no matter what it costs. The earlier defects are discovered, the less it will cost to correct them. The earlier defects are discovered, the less it will cost to correct them.

Backup Slides

Design Coding System Requirements Contractor Management Evaluate Requirements Evaluate Design Monitor Tests Integration Tested Products Discrepancies Subs Project Manager IV&V Testing Independent Verification and Validation (IV&V)

Static and Dynamic Verification & Validation Sommerville, p. 421

Test Case Development in the V Model Code Execute Unit Tests Test cases designed and executed by developers Test cases designed and executed by independent testers Requirements Analysis Execute System Tests Execute Integration Tests Design Validate Requirements Verify Design Verify Implementation