1 / 28 CS 425/625 Software Engineering Verification and Validation Based on Chapter 19 of the textbook [SE-6] Ian Sommerville, Software Engineering, 6 th Ed., Addison-Wesley, 2000 and on the Ch19 PowerPoint presentation available at the book’s web-site: November 10, 2003

2 / 28 Outline n n Introduction n n Testing and Debugging n n Verification and Validation Planning n n Software Inspections n n Program Inspections (forms of Software Inspections) n n Automatic Static Analysis

3 / 28 Introduction….. n n Verification and validation (V&V): the checking and analysis processes that ensure the software satisfies its specification and meets the needs of the clients who are paying for it n n Validation: “Are we building the right product?” n n Verification: “Are we building the product right?” n n Verification involves checking the software conforms with its specification while the more general process of validation ensures the software meets the needs of the clients n n V&V is a whole life-cycle process, encompassing requirements reviews, design reviews, code inspections, and program testing

4 / 28.Introduction.… n V & V techniques: u Software inspections u Software testing n n Software inspections u Are static V&V techniques as they do not require the software to be executed u Consist of inspections, automated static analyses, and formal verifications of source code or system models u u Can only check the correspondence between the software and its specification u u Cannot demonstrate the system is operationally useful u u Cannot check non-functional requirements of the software

5 / 28..Introduction... n n Software testing u u It is a dynamic V&V technique as it needs an executable version of the software system u u The system is executed with test data and its operational behaviour is assessed u u Can reveal the presence of errors, not their absence u u A successful test is a test which discovers one or more errors u u The V&V technique for checking non-functional requirements and for verifying system integration u u Should be used in conjunction with static techniques to provide full V&V coverage

6 / 28 …Introduction.. n n Types of software testing u u Defect testing   Intended to find inconsistencies between a program and its specification   Tests designed to discover program faults and defects   A successful defect test is one which reveals the presence of defects in a system u u Statistical testing   Designed for software’s performance and reliability estimation   By running tests that reflect actual user inputs and their frequency, an estimate of operational reliability can be made

7 / 28 ….Introduction. n n Static and dynamic V&V [Fig 19.1, Somm00]

8 / 28 …..Introduction n n Verification and validation should establish confidence that the software is fit for purpose n n This does not mean the software is completely free of defects; rather, it must be good enough for its intended use n n The required level of confidence depends on: u u Software’s purpose: the level of confidence depends on how critical the software is to an organisation u u User expectations: users may have lower expectations of certain types of software u u Marketing environment: getting a product to market early may have higher priority than finding its defects

9 / 28 Testing and debugging. n n Defect testing and debugging are distinct processes n n V&V is concerned with establishing the existence of defects in a program n n Debugging is concerned with locating and repairing these errors n n Debugging involves formulating hypotheses about program behaviour then testing these hypotheses to find the errors

10 / 28.Testing and debugging The debugging process [Fig. 19.2, Somm00]

11 / 28 n n The planning of V&V should start early in the development process n n The plan should balance static verification and testing, specify testing standards and procedures, establish checklists for inspections, and define the software test plan n n Test planning breaks down V&V into a number of stages, often organized according to the V-model (shown on next page) n n The focus is on setting standards and procedures for inspections and testing, not on describing product tests V & V Planning..

12 / 28.V&V Planning. The V-model of development [Fig. 19.3, Somm00]

13 / 28..V&V Planning..V&V Planning n n The structure of a software test plan u u The testing process u u Requirements traceability u u Tested items u u Testing schedule u u Test recording procedures u u Hardware and software requirements u u Constraints

14 / 28 Software Inspections. n n Involve people examining software code or models (representations) with the aim of discovering defects n n Do not require execution of a system thus can be used throughout the development process n n May be applied to any representation of the system: requirements, design, test data, etc. n n Very effective technique for discovering errors

15 / 28.Software Inspections n n In software inspections many different defects can be discovered in a single review of the source code or software model n n In testing, one defect may mask another hence several executions are required n n Software inspections reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly occur n n Software inspections and software testing are complementary, not competing techniques (see also slides 4 and 5)

16 / 28 Program Inspections……. n n Program inspections are a type of software inspections n n Consist of formal reviews conducted by teams and intended for program defect detection n n Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g., an un-initialized variable), or non-compliances with standards

17 / 28.Program Inspections…… n n Inspection pre-conditions: u u A precise specification must be available u u Team members must be familiar with the organisation standards u u Syntactically correct code must be available u u An error checklist should be prepared u u Management must accept that inspection will increase costs early in the software process u u Management must not use inspections for staff appraisal

18 / 28..Program Inspections….. The inspection process [Fig. 19.6, Somm00]

19 / 28 …Program Inspections…. n n Program inspection procedure: u u The inspection is planned by the moderator (or chairman) u u The system overview is presented to the inspection team by the program author (or owner) u u The code and associated documents are distributed to the inspectors (inspection team) in advance for individual preparation u u The inspection meeting takes place and errors are noted (the inspection also involves a reader and, possibly, a scribe) u u During re-work modifications are made by the author to repair discovered errors u u Re-inspection may or may not be required, based on moderator’s decision

20 / 28 ….Program Inspections… n Inspection teams are n Inspection teams are made up of at least 4 members: u u Author of the code being inspected u u Inspector who finds errors, omissions and inconsistencies u u Reader who reads the code to the team u u Moderator who chairs the meeting and notes discovered errors u u Other roles are chief moderator and scribe

21 / 28 …..Program Inspections.. n n Inspection checklist u u A checklist of common errors should be used during the inspection u u This error checklist is programming language dependent u u The weaker the type-checking of the language, the larger the checklist u u Examples of possible errors: initialisation, constant naming, loop termination, array bounds, etc.

22 / 28 Inspection checks (fault classes) [Fig. 19.7, Somm00]

23 / 28 …….Program Inspections n n Inspection rates: u u 500 statements/hour during overview u u 125 source statement/hour during individual preparation u u statements/hour can be inspected u u Inspection is therefore an expensive process u u Inspecting 500 lines costs about 40 man/hours effort = £2800

24 / 28 Automated Static Analysis…. n n Static analyzers are software tools for source code processing n n They parse the program text to discover erroneous conditions n n Very effective as an aid to inspections; supplement but cannot replace inspections n n Particularly valuable for languages such as C that have weak typing (many errors can remain undetected by the compiler) n n Less cost-effective for languages such as Java that have strong type checking (many errors can be detected during compilation)

25 / 28.Automated Static Analysis… Automatic static analysis checks [Fig. 19.8, Somm00]

26 / 28..Automated Static Analysis.. n n Stages of static analysis: u u Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc. u u Data use analysis. Detects un-initialised variables, variables written twice without an intervening assignment, variables which are declared but never used, etc. u u Interface analysis. Checks the consistency of routine and procedure declarations and their use

27 / 28 …Automated Static Analysis. n n Stages of static analysis [cont’d]: u u Information flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code review (inspection) u u Path analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review process

28 / 28 LINT static analysis [Fig 19.9, Somm00] 138% more lint_ex.c #include printarray (Anarray) int Anarray; { printf(“%d”,Anarray); } main () { int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ; } 139% cc lint_ex.c 140% lint lint_ex.c lint_ex.c(10): warning: c may be used before set lint_ex.c(10): warning: i may be used before set printarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10) printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(10) printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(11) printf returns value which is always ignored