1. © 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Testing 29/Apr/2009 Petr Panuška petr.panuska@hp.com QA Manager & Offshore Team Manager, SOA Center HP Software R&D

2. Agenda 1. Why is SW testing necessary 2. Testing principles 3. Testing design techniques

3. Why is SW testing necessary SW does not always work as we wish Defects in SW can cause harm −Not all defects in SW result in a failure Testing reduces the probability of undiscovered defects remaining in the software Testing gives us the confidence about the tested SW

4. Testing Principles

5. ISTQB defines 7 Testing Principles 1.Testing shows presence of defects 2.Exhaustive testing is impossible 3.Early testing 4.Defect clustering 5.Pesticide paradox 6.Testing is context dependent 7.Absence-of-errors fallacy

6. Testing shows presence of defects Tests show the presence not the absence of defects. Testing can’t prove that SW is defect free Testing reduces the probability of undiscovered defects remaining in the software However, the fact that no defects were found, is not a proof of correctness.

7. Exhaustive testing is impossible It is not cost-effective −Defects are not of an equal risk We need to prioritize our tests −Use risk analysis and priorities to focus testing efforts −Use appropriate testing techniques to meet the risk, giving us confidence in what testing has been performed Exhaustive testing is equivalent to Halting problem −There is no algorithm that can for an arbitrary program and its input validates if the program works (with the given input) fine or not

8. Early testing In SW development, what can be tested −Requirement specification −Use-case Analysis document −Technical Analysis document −Design document −Functional Implementation −Performance −Usability What can be also tested −Test Description −Documentation Price of potential defect fix increases

9. Example

10. Example – Early testing Defect in H310: JBoss 4.3.0 not supported on Win 2008 (although Hermes 310 PRD requires this combination to be supported) −7 other duplicates reported −Involving 4 people from QA −9 people from DEV

11. Defect clustering Defects get clustered since different reasons −A component might be more complex than others −A component might be developed by a less experienced developer −A component might be developed by a less careful developer −A component might have a poorer specification −A component might need more refactoring (introducing more other defects) Another explanation: http://parlezuml.com/blog/?postid=242 http://parlezuml.com/blog/?postid=242

12. Pesticide paradox Old tests will eventually stop finding new defects −Defects create immunity to these tests To find defects, new tests need to be introduced −Or old tests refactored Conclusion: Regression tests do not find majority of new defects

13. Testing is context dependent Different kinds of tests are run in different periods −Defect testing To discover faults or defects in the software where its behavior is incorrect or not in conformance with its specification; A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system. −Validation testing To demonstrate to the developer and the system customer that the software meets its requirements; A successful test shows that the system operates as intended. −We start with defect testing and perform validation testing later in the SW development process.

14. Testing is context dependent II. We can also test differently because of −The type of industry (safety-critical, business, nuclear) −Number of customers (impact the SW makes) One customer patch New version of SW for potentially many customers

15. Absence-of-errors fallacy

16. We test a system to see if it meets the documented requirements We find and fix defects to demonstrate that the system meets these specifications Finding and fixing defects does not help if the system is unusable and does not fulfill the user’s needs and expectations

17. Testing Techniques

18. Testing design techniques Black-box techniques −Equivalence partitioning −Boundary value analysis −Decision tables −State transitions White-box techniques −Statement testing and coverage −Decision testing and coverage −Other structural-based coverage No details about the product implementation are known Tester knows how the tested requirement is implemented

19. Equivalence partitioning Tax Calculator based on annual income Input – integral number Partitions of equivalence Invalid partValid part #1Valid part #2Valid part #3 … -2, -10, 1, … 1999920000, … 5000050001, … IncomeTax < $20000No tax $20000 - $5000015% > $5000020%

20. Equivalence partitioning II Split input into partitions −Valid partitions −Invalid partitions Have one test-case for each partition −Even invalid ones One more example: Enter number of a month Invalid partition 1Valid partitionInvalid partition 2 … -2, -1, 01, 2, … 1213, 14, …

21. Boundary value analysis Similar as Equivalence partitioning but −Test boundary values N-BVA (BVA = 0-BVA) −Test N boundary values −BV, BV+1, BV-1, … BV+N, BV-N

22. Decision tables

23. Decision tables - Example Example – HP SOA Systinet Licensing Framework HP SOA Systinet consists of −Core, Reporting, Lifecycle, etc. −Policy Manager (optional) −Contract Manager (optional) HP SOA Systinet License can limit −Number of users −Number of days Default license included in the installer −Policy & Contract Manager included −Limited to 60 days, unlimited to number of users

24. Decision tables Example – Testing license application Conditions Actions

25. Decision tables Example – Testing license application Conditions Default PM on CM on # days limited # users limited Actions Try PM Try CM Login after expiration Login more users than allowed

26. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM Try CM Login after expiration Login more users than allowed

27. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM X Try CM X Login after expiration X Login more users than allowed Default license

28. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM X Try CM X Login after expiration X Login more users than allowed Default license

29. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM X X Try CM X X Login after expiration X X Login more users than allowed X Default license Visibility Edition

30. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM X X X Try CM X X X Login after expiration X X Login more users than allowed X Default license Unlimited Standard Edition Visibility Edition

31. Decision tables Example – Testing license application Conditions DefaultYYYYYYYYYYYYYYYYNNNNNNNNNNNNNNNN PM onYYYYYYYYNNNNNNNNYYYYYYYYNNNNNNNN CM onYYYYNNNNYYYYNNNNYYYYNNNNYYYYNNNN # days limitedYYNNYYNNYYNNYYNNYYNNYYNNYYNNYYNN # users limitedYNYNYNYNYNYNYNYNYNYNYNYNYNYNYNYN Actions Try PM X X X X Try CM X X X X Login after expiration X X X Login more users than allowed X X Default license Unlimited Standard Edition Limited Standard Edition Visibility Edition

32. Decision tables II Catches all possible combinations Helps to analyze the situation and to decide −Which use-cases must be tested −Which use-cases does not make sense −Requires knowledge of the business environment Helps to prioritize the use-cases

33. State transitions Example – Testing ‘Contract Request Lifecycle’ Create Request Accept Request Reject Request Revoke Request Delete Request

34. State transitions II Transition diagram shows valid transitions −Does not show invalid (that should be also tested) Good for testing use-cases that are possible to be described by transitions between states The scenarios may contain test-cases for each −State −Transition −Event that triggers state change (transition) −Action that may result from those transitions

35. White-box techniques

36. Statement testing and coverage

37. Decision testing and coverage

38. Other structure-based techniques Condition Coverage −All single boolean conditions in a single statement must be evaluated to true and false −Full condition coverage does not imply full decision one! Condition/Decision Coverage −Hybrid metric composed by the union of condition coverage and decision coverage.

39. Other structure-based techniques II Multiple Condition Coverage −All possible combinations of all boolean conditions in a single statement must be evaluated to true and false Path Coverage −Whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit. Function, Call, LCSAJ, Loop, Race, etc. coverages FMI, see http://www.bullseye.com/coverage.htmlhttp://www.bullseye.com/coverage.html

40. Example RTCA published DO-178B that requires minimal coverage for aeronautics SW systems based on their criticality: Effect of System Failure ExampleRequired Code Coverage CatastrophicCrash 100% condition/decision coverage 100% statement coverage HazardousPassenger fatality 100% decision coverage 100% statement coverage MajorPassenger injury100% statement coverage MinorFlight plan changeNo code coverage requirement No effect Entertainment system failureNo code coverage requirement

41. Summary

42. Testing Principles −Testing shows presence of defects −Exhaustive testing is impossible −Early testing −Defect clustering −Pesticide paradox −Testing is context dependent −Absence-of-errors fallacy Testing Techniques −Black-box techniques −White-box techniques

43. Q&A