Testing techniques and methods Ana Kovaliova Natalja Šiškina 2016-10-12

Content Types of techniques Static Dynamic Structural Specification based (functional) Experience based

Static testing technique do not execute the code test any form of document (source code, design documents, functional specifications, requirement specifications) static testing techniques do not execute the code being examined and are generally used before any tests are executed on the software. They could be called non-execution techniques. Most static testing techniques can be used to 'test' any form of document including source code, design documents and models, functional specifications and requirement specifications.

Dynamic testing techniques Functional (Black Box) based on behaviour / functionality of software Structural (White Box) based on structure of software Dynamic testing techniques Experience-based based on knowledge, skills and background of technical and business people Black-box: black box testing is software testing method for functional specification of a software, i.e. we are doing testing without no understanding of internal software structure. This method can used in all testing levels: unit, integration, system and acceptance testing. In this case the tester don’t need any specific info about internal software structure. White-box (code and design): Test cases are created depending on specification and requirements, i.e. what program is going to do. Tester checks both possibilities: right and wrong input data and states right output data without any understanding of internal program structure. So, they test internal software structure or program process in order to evaluate the functionality of the system. In white-box testing the developer’s skills are very useful to create tests.

Black, White and Experienced based (Specification Based) White (Structure Based on code and the design of the system The tests provide the ability to derive the extent of coverage of the whole application Techniques Statement coverage Branch Coverage Decision Coverage Experience (Black box) Error Guessing Exploratory Testing Based on the knowledge of the tester Using past experienced use & intuition to “guess” where errors may occur Based on requirements From the requirements, tests are created Specification Models can be used for systematic test case design Equivalence Partitioning Boundary Value Analysis Decision Tables State Transition Testing Pairwise Testing

White box techniques use the internal structure of the software to derive test cases this technique requires knowledge of how the software is implemented Types Statement Coverage Branch Coverage Decision Coverage

Black box testing methods Equivalence Partitioning Boundary Value Analysis Decision Table State Transition Testing Pairwise Testing

Equivalence partitioning (1) Typically the universe of all possible test cases is so large that you cannot try them all You have to select a relatively small number of test cases to actually run Which test cases should you choose? Equivalence partitioning helps answer this question

Equivalence Partitioning (2) Partition the test cases into "equivalence classes" Each equivalence class contains a set of "equivalent" test cases Two test cases are considered to be equivalent if we expect the program to process them both in the same way If you expect the program to process two test cases in the same way, only test one of them, thus reducing the number of test cases you have to run

Equivalence Partitioning (3) First-level partitioning: Valid vs. Invalid test cases Valid Invalid

Equivalence Partitioning (4) Partition valid and invalid test cases into equivalence classes

Equivalence Partitioning (5) Create a test case for at least one value from each equivalence class

Equivalence partitioning (6) divide (partition) the inputs, outputs, etc. into areas which are the same (equivalent) assumption: if one value works, all will work one from each partition better than all from one invalid valid 1 100 101

Boundary Value Analysis (1) When choosing values from an equivalence class to test, use the values that are most likely to cause the program to fail Errors tend to occur at the boundaries of equivalence classes rather than at the "center" In addition to testing center values, we should also test boundary values Right on a boundary Very close to a boundary on either side

Boundary Value Analysis (2) Create test cases to test boundaries of equivalence classes

Boundary value analysis (3) faults tend to occur near boundaries good place to look for faults test values on both sides of boundaries invalid valid 1 100 101

Why do both EP and BVA? If you do boundaries only, you have covered all the partitions as well technically correct and may be OK if everything works correctly! if the test fails, is the whole partition wrong, or is a boundary in the wrong place - have to test mid- partition anyway testing only extremes may not give confidence for typical use scenarios (especially for users) boundaries may be harder (more costly) to set up

Decision tables (1) capture system requirements that contain logical conditions record complex business rules DT testing creates combinations of conditions that otherwise might not have been exercised during testing

Decision tables (2) Table based technique where Inputs to the system are recorded Outputs to the system are defined Inputs are usually defined in terms of actions which are Boolean (true or false) Outputs are recorded against each unique combination of inputs Using DT the relationships between the inputs and the possible outputs are mapped together The input conditions and actions are most often stated in such a way that they must be true or false (Boolean). The decision table contains the triggering conditions, often combinations of true and false for all input conditions, and the resulting actions for each combination of conditions.

Decision Table Structure Test 1 Test 2 Test 3 Input 1 T F Input 2 Input 3 DON’T CARE Input 4 Inputs / Actions Output / Response Response 1 Y N Response 2 Response 3 Each column of the table corresponds to a business rule that defines a unique combination of conditions that result in the execution of the actions associated with that rule

Decision Table Example Test 1 Test 2 Test 3 > 55 yrs old F T Smoker Exercises 3 times a week + History of Heart Attacks Insure Y N Offer 10% Discount Offer 30% Discount What will be the outcome of the following business rules? Joe is a 22 year old non smoker who goes to the gym 4 times / week and has no history of heart attacks in his family Kevin is 62 year old non smoker who swims twice a week and plays tennis. He has no history of heart attacks in his family START - A decision table which records the business rules associated with Life Insurance policy decisions, based on the applicant’s age and health record. Joe – Insure and offer a 30% discount. Kevin - Insure and offer a 10% discount.

State Transition Testing State Transition Testing uses the following terms: State diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. State table: A grid showing the resulting transitions for each state combined with each possible event ( includes both valid and invalid transitions) State transition: A transition between two states of a component or system. State transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. Also known as N-switch testing.

State Transition Diagram State A State B Event/Action etc Starting State End State Transition Between start and end states Event from outside the system Action triggered by Event

State Transition Example Simplified Car Gears Change Down/ Move Back Reverse Neutral 1st Gear 2nd Gear 3rd Gear Change Up/ Accelerate Change Up/ Accelerate Change Down/ Decelerate Change Down/ Decelerate

State Transition - Switch Coverage Switch Coverage is a method of determining the number tests based on the number “hops” between transitions. Sometimes known as Chow 0-Switch Coverage (1 hop) RN NR N1 1N 12 21 23 32 1-Switch Coverage (2 hops) RN1 RNR NRN N12 N1N 1NR Etc. These hops can be used to determine the VALID tests

State Transition – Another example for a Theatre Show reservation Show Options provided Request Show Options Show selected Choose Show Change Mind/ Return to Options Show Reservation Made Reserve Show Paid For Pay for Show Ticket Received Issue Cancelled Cancel reservation Cancel reservation (return ticket)/Issue Refund Cancel reservation/ Issue Refund 2' 6 5 4 3 2 1 5' 4' 6'

Pairwise testing (All-Pairs Testing) popular approach to combinatorial testing problems. test cases are designed to execute all possible discrete combinations of each pair of input parameters. test suite covers all combinations; therefore it is not exhaustive, but still effective Pairwise testing is a wildly popular approach to combinatorial testing problems. A black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters. All-Pairs technique is very helpful for designing tests for applications involving multiple parameters. Tests are designed such that for each pair of input parameters to a system, there are all possible discrete combinations of those parameters.

Pairwise testing. Example Car Ordering Application : 1) Order category a. Buy b. Sell 4) Registration numbers a. Valid (5000) b. Invalid 2) Location a. Delhi b. Mumbai 5) Order type a. E-Booking b. In store 3) Car brand a. BMW b. Audi c. Mercedes 6) Order time a. Working hours b. Non-working hours Car Ordering Application : The car ordering application allows for Buying and Selling cars. It should support trading in Delhi and Mumbai. The application should have a registration numbers, may be valid or invalid. It should allow the trade of following cars: BMW, Audi, and Mercedes. Two types of booking can be done: E-booking and In Store. Orders can be placed only during trading hours.

Pairwise testing. Example If we want to test all possible valid combinations: = 2 X 2 X 3 X 5000 X 2 X 2= 240000 valid test cases combinations :( There are also an infinite number of invalid combinations! Let’s simplify: Reduce Registration Number to Two: 1. Valid registration number 2. Invalid registration number Now all possible combinations = 2 X 2 X 3 X 2 X 2 X 2 = 96

Pairwise testing. Example All pairs in 8 cases, instead of all combinations in 96! Pairwise testing technique can dramatically reduce the number of combinations to be covered but remains very effective in terms of fault detection. It is indeed a smart test design technique that guarantees a win-win situation for both test effort and test effectiveness. During the Test planning phase of software testing, Pairwise testing technique should always be taken into consideration. Either we are doing it manually or using any tool to generate test cases, it becomes a necessary component of the test plan because it in turn affects Test estimation.

Pairwise testing This technique has some limitations as well: It fails when the values selected for testing are incorrect. It fails when highly probable combinations get too little attention. It fails when interactions between the variables are not understood well

Choosing test Techniques (1) How do you choose the right technique? Type of system Standards Customer or contractual requirements Level of risk Type of risk Testing objectives Documentation available Knowledge / skills of the testers Time and budget Development processes Pick the right techniques for the right situation

Choosing test Techniques (2) Specification-based techniques are appropriate at all levels of testing Developers use structure-based techniques in component testing and component integration testing Experience-based techniques are used to complement both techniques (useful under extreme time pressure)

Thank you!

References D. Graham, E. Veenendaal, I. Evans, R. Black, ISTQB Certification, Foundation of software testing Certified Tester, Foundation Level Syllabus, http://www.istqb.org/downloads/send /2-foundation-level-documents/3- foundation-level-syllabus-2011.html4 Pairwise testing, http://www.softwaretestinghelp.com/wh at-is-pairwise-testing/