1. PVK-HT04bella@cs.umu.se1 Contents Introduction Requirements Engineering Project Management Software Design Detailed Design and Coding Quality Assurance Maintenance Analysis Design Testing Coding Operation and Maintenance Installation Require- ments Requirements Specification Planning

2. PVK-HT04bella@cs.umu.se2 Quality Assurance Introduction Testing oTesting Phases and Approaches oBlack-box Testing oWhite-box Testing oSystem Testing

3. PVK-HT04bella@cs.umu.se3 What is Quality Assurance? Constructive vs analytic approaches to QA Qualitative vs quantitative quality standards Measurement oDerive qualitative factors from measurable quantitative factors Software Metrics QA is the combination of planned and unplanned activities to ensure the fulfillment of predefined quality standards.

4. PVK-HT04bella@cs.umu.se4 Approaches to QA Constructive Approaches oSyntax-directed editors oType systems oTransformational programming oCoding guidelines o... Analytic approaches oInspections oStatic analysis tools (e.g. lint) oTesting o... Usage of methods, languages, and tools that ensure the fulfillment of some quality factors. Usage of methods, languages, and tools to observe the current quality level.

5. PVK-HT04bella@cs.umu.se5 Fault vs Failure ?! human errorfaultfailure can lead to Different types of faults Different identification techniques Different testing techniques Fault prevention and -detection strategies should be based on expected fault profile

6. bella@cs.umu.se6 Specification/ requirements Environment/ support Documen- tation OtherDesignCode Fault origin: WHERE? Missing Unclear Wrong Changed Better way Fault mode: WHY? Fault type: WHAT? Requirements or specifications Functionality HW interface SW interface User interface Functional description Test HW Test SW Integration SW Development tools Logic Computation Data handling Module interface/ implementation Standards (Inter-)Process communications Data definition Module design Logic description Error checking Standards HP´s Fault Classification

7. PVK-HT04bella@cs.umu.se7 Fault Profile of a HP Division See [Pfleeger 98].

8. PVK-HT048 Testing Phases Unit test Unit test Unit test Integration test Function test Performance test Acceptance ( ,  ) test Installation test Component code...... Tested component Integrated modules Functioning system Verified software Accepted, validated system SYSTEM IN USE! Design specifications System functional requirements Other software requirements Customer requirements specification User environment System test Pre-implementation test

9. 9 Pre-Implementation Testing Inspections oevaluation of documents and code prior to technical review or testing Walkthrough oIn teams oExamine source code/detailed design Reviews oMore informal oOften done by document owners Advantages oEffective oHigh learning effect oDistributing system knowledge Disadvantages o Expensive

10. PVK-HT04bella@cs.umu.se10 Testing vs “Proofing” Correctness Verification oCheck the design/code against the requirements Are we building the product right? Validation oCheck the product against the expectations of the customer Are we building the right product? Testing Testing can neither proof that a program is error free, nor that it is correct [Myers 76] Testing can only show the presence of errors, never their absence. [Dijkstra 69] Testing is the process in which a (probably unfinished) program is executed with the goal to find errors.

11. 11 Testing Principles Construction of test suites oPlan tests under the assumption to find errors oTry typical and untypical inputs oBuild classes of inputs and choose representatives of each class Carrying out tests oTesters  implementers oDefine the expected results before running a test oCheck for superfluous computation oCheck test results thoroughly oDocument test thoroughly Simplify test oDivide programs in separately testable units oDevelop programs test friendly Each test must be reproducible

12. PVK-HT04bella@cs.umu.se12 Test Methods Structural testing (white-box, glass-box) oUses code/detailed design is to develop test cases oTypically used in unit testing oApproaches: Coverage-based testing Symbolic execution Data flow analysis... Functional testing (black-box) oUses function specifications to develop test cases oTypically used in system testing oApproaches: Equivalence partitioning Border case analysis... time develop black-box test cases develop white-box test cases perform white-box testing perform black-box testing

13. PVK-HT04bella@cs.umu.se13 Test Preparation Exhaustive testing is prohibited, because of the combinatorial explosion of test cases Choose representative test data for i := 1 to 100 do if a = b then X else Y; ipaths to test#tests 1X, Y2 2XX, XY, YX, YY4 3XXX, XXY,...8... 1002 100 2  2 100 - 2 > 2,5  10 30 With 10 6 tests/sec this would take 8  10 16 years Choose test data (test cases)

14. PVK-HT04bella@cs.umu.se14 How to Choose Test Data Example 1 Both paths must be tested! Example 2 How can I know there is a “path”? if ((x + y + z)/3 = x) then writeln( “x, y, z are equal”) else writeln( “x, y, z are unequal”); Test case 1:x=1, y=2, z=3 Test case 2:x=y=z=2 if (d = 0) then writeln( “division by zero”) else x = y/n; (*-----------------------------*) x = y/n;

15. PVK-HT04bella@cs.umu.se15 Test Case Development Problems: oSystematic way to develop test cases oFind a satisfying set of test cases Test case  test data Test data: Inputs devised to test the system Test case: oSituation to test oInputs to test this situation oExpected outputs è Test are reproducible è Equivalence partitioning è Coverage-based testing

16. PVK-HT04bella@cs.umu.se16 Equivalence Partitioning System Input data Output data Inputs causing anomalous behaviour Outputs which reveal the presence of faults Input- and output data can be grouped into classes where all members in the class behave in a comparable way. Define the classes Choose representatives u Typical element u Borderline cases x  [25.. 100] class 1:x < 25 class 2: x >= 25 and x <= 100 class 3: x > 100

17. PVK-HT0417 Equivalence Partitioning Examples oModule keeps a running total of amount of money received oInput: AmountofContribution, specified as being a value from $0.01 to $99,999.99 oModule prints an appropriate response letter oInput: MemberStatus, specified as having a value of {Regular, Student, Retiree, StudioClub} o 3 equivalence classes: Values from $0.01 to $99,999.99 (valid) Values less than $0.01 (invalid) Values greater than $99,999.99 (invalid) o 2 equivalence classes: Values of {Regular, Student, Retiree, StudioClub} (valid) All other input (invalid)

18. PVK-HT04bella@cs.umu.se18 Statement Coverage 1 3 2 4 5 6 7 8 2 test cases: 12467; 13567 Every statement is at least executed once in some test

19. PVK-HT04bella@cs.umu.se19 Branch Coverage 1 3 2 4 5 6 7 8 2 test cases: 12467; 1358 For every decision point in the graph, each branch is at least chosen once

20. PVK-HT04bella@cs.umu.se20 Condition Coverage Test all combinations of conditions in boolean expressions at least once if (X or not (Y and Z) and... then b; c := (d + e * f - g) div op( h, i, j);

21. PVK-HT04bella@cs.umu.se21 Path Coverage Assure that every distinct paths in the control-flow graph is executed at least once in some test

22. PVK-HT04bella@cs.umu.se22 Path Coverage 1 3 2 4 5 6 7 8 Assure that every distinct paths in the control-flow graph is executed at least once in some test

23. PVK-HT04bella@cs.umu.se23 Path Coverage 1 3 2 4 5 6 7 8 4 test cases: 12467; 1358; 1248; 13567 Assure that every distinct paths in the control-flow graph is executed at least once in some test

24. PVK-HT04bella@cs.umu.se24 Test Coverage–Example Statement Coverage: 5/10 = 50% Branch Coverage: 2/6 = 33% Path Coverage: 1/4 = 25%

25. PVK-HT04bella@cs.umu.se25 Data-flow testing Def-use analysis: match variable definitions (assignments) and uses. Example: x = 5; … if (x > 0)... Does assignment get to the use?

26. PVK-HT04bella@cs.umu.se26 Data Flow Coverage All-uses coverage x :=2 x :=1 z := 2*r x :=3 z := 2*x-y z :=x+y y :=2 r :=4 Red path covers the defs y :=2; r :=4; x :=1 Blue path covers y :=2; x :=3. Does not cover x :=2

27. PVK-HT04bella@cs.umu.se27 Coverage-based Testing Advantages oSystematic way to develop test cases oMeasurable results (the coverage) oExtensive tool support Flow graph generators Test data generators Bookkeeping Documentation support Disadvantages oCode must be available oDoes not (yet) work well for data-driven programs

28. PVK-HT04bella@cs.umu.se28 Integration Testing How to integrate & test the system Top-down Bottom-up Critical units first Functionality-oriented (threads) Implications of build order Top-down => stubs; more thorough top-level Bottom-up => drivers; more thorough bottom- level Critical => stubs & drivers. A DCB GFE Test E Test F Test G Test B,E,F Test C Test D,G Test all

29. PVK-HT04bella@cs.umu.se29 Further phases of testing System testing: Test the functionality, performance, reliability, security of the entire system. Acceptance testing: Operate system in user environment, with standard user input scenarios. Regression testing: Test of modified versions of previously validated system.

30. 30 Testing Tools / Support Test data generators oInput: Program + testing strategy oOutput: Sets of input data Profilers oInstrument code to collect run-time data Time spent in operations Number of calls to operations Number of loop iterations... è Find bottle-necks è Indicate dead code Simulators oCommon in hard-/software systems and/or real- time systems oEmulate critical parts by software

31. PVK-HT04bella@cs.umu.se31 Testing Tools / Support Debuggers oManual code instrumentation oInspect/trace variables o... File comparators oE.g. for regression testing Test-stub/-driver generators oSimulate client or server components, which are not yet available ClientServer