1. Software Metric; defect removal efficiency, Cyclomate Complexity Defect Seeding Mutation Testing

2. Cyclomate Complexity Basis Path Testing During Integration Basis path testing can also be applied to integration testing when units/components are integrated together McCabe’s Design Predicate approach: Draw a “structure chart” Calculate integration complexity Select tests to exercise every “type” of interaction, not every combination

3. Types of Interactions Types of unit/component interactions:

4. Unconditional – Unit/component A always calls unit/component B. A calling tree always has an integration complexity of at least one. The integration complexity is NOT incremented for each occurrence of an unconditional interaction. Conditional – Unit/component A calls unit/component B only if certain conditions are met. The integration complexity is incremented by one for each occurrence of a conditional interaction in the structure chart. Mutually Exclusive both) – Unit/component A calls either unit/component B or unit/component C (but not based upon certain conditions being met.) The integration complexity is incremented by one for each occurrence of a mutually exclusive conditional interaction in the structure chart. Iterative – Unit/component A calls unit/component B one or more times based upon certain conditions being met. The integration complexity is incremented by one for each occurrence of an iterative interaction in the structure chart.

5. Integration Basis Path Test Cases Basis path testing can also be applied to integration testing when units/components are integrated together Complexity: 5 Basis set of paths: – 1. A,B,C,E & then A calls I – 2. A,B,C,F & then A calls I – 3. A,B,D,G (only once) & then A calls I – 4. A,B,D,G (more than once) & then A calls I – 5. A,B,D,G,H & then A calls I

6. Integration Basis Path Testing Interaction types: – 2 conditional – 1 mutually exclusive conditional – 1 iteration – 1 simply because we have a graph Basis set of paths: 1. A,C,F (only once) 2. A,C,F (more than once) 3. A,B,D,G & then B calls E & then A calls C 4. A,B,D,H & then B calls E & then A calls C,F 5. A,B,D,H & then B calls E, I & then A calls C,F

7. Defect removal efficiency (contd.) Defect removal efficiency is very complicated, as different kinds of defect removal operations have different removal efficiency. As illustrated in next example; various defect removal operation at various phases of software development yields different defect removal efficiency.

8. Defect Removal Effectiveness DRE= Defects removed during development phase x100% Defects latent in the product Defects latent in the product = Defects removed during development phase+ defects found later by user The Defect Removal Effectiveness for each of the phases would be as follows: Requirements DRE = 10 / (10+3+0+2+1) x 100% = 63% Design DRE = (3+18) / (3+0+2+1+18+4+5+2) x 100% = 60% Coding DRE = (0+4+26) / (0+2+1+4+5+2+26+8+7) x 100% = 55% Testing DRE = (2+5+8) / (2+1+5+2+8+7) x 100% = 60% Development DRE = (Pre-release Defect) / (Total Defects) x 100% (10+3+2+18+4+5+26+8) / (10+3+2+1+18+4+5+2+26+8+7) x 100 = 88% Example:

9. Defect seeding The concept of defect seeding is to insert a known errors into a software deliverable prior to review, inspection or testing. For example suppose 100 known errors are inserted into the system prior to integration testing and test case detects only 30 errors, it concludes that integration testing is 30% efficient.

10. Fault injection technique In the context of software engineering, fault injection is a technique or methodology meant for improving the test coverage. This is usually done by introducing faults in the program source code in order to test the code paths. In particular it is done to test the error handling test paths that might otherwise be left un- followed and thus, untested.

11. Fault injection technique (contd.) The injected fault goes through a well defined cycle before it becomes an observable failure. After the fault is injected, the program or the application is executed. Upon execution, the injected fault may cause an error which is to be considered as an invalid state within the boundaries of the system. This error may cause many further errors within the boundaries of the system. In this way each error acts as an injected fault and propagates to the system boundaries and comes to observance. When the states of the errors are observed at the boundaries of the system they are known as failures. This whole mechanism is known as “fault – error – failure – cycle”.

12. The fault injection technique was used to introduce faults at the level of hardware system. This type of fault injection method is known as HWIFI or hardware implemented fault injection. It tends to simulate failures within a hardware system. Soon after that it was found that faults could also be introduced in the software system and it could be helpful in accessing the software system more appropriately. These fault injection techniques are collectively known as SWIFI or software implemented fault injection.

13. Categories of fault injection techniques Techniques for software implemented fault injection can be classified into two major categories namely: Compile time injection: – This can be defined as an injection technique in which the source code is modified so that the simulated faults can be injected into the system. – One popular method for doing this is “mutation testing” in which the existing code is changed so as to induce faults. – the faults produced are very similar to those added unintentionally by the programmers. – Another technique which is modification of code mutation technique known as code insertion fault injection adds code rather than modifying the existing piece of code. – This is done with the use of simple functions which take an existing value and perturb (disturb) it using some logic into another value. Such functions are called perturbation functions.

14. Run time injection: – This technique can be defined as the technique which makes use of a software trigger to inject fault in the system which is executing. – Using this technique faults can be injected in a variety of ways such as those listed below: – Time based triggers An interrupt is generated when the timer reaches a specified time. The interrupt handler associated with the timer will inject the fault. – Interrupt based triggers Software trap mechanisms and hardware exceptions are effectively used to generate a fault in the code of the system at a particular point. This gives instant access to a specific memory location.

15. Hardware faults Hardware/Physical Fault that arise during system operation are best classified by their duration: – Permanent – transient – intermittent. Permanent faults: Caused by irreversible component damage, such as a semiconductor junction that has shorted out because of thermal aging, improper manufacture, or misuse. Since it is possible that a chip in a network card that burns causing the card to stop working, recovery can only be accomplished by replacing or repairing the damaged component or subsystem. Transient faults: Triggered by environmental conditions such as power-line fluctuation, electromagnetic interference, or radiation. These faults rarely do any lasting damage to the component affected, although they can induce an erroneous state in the system. According to several studies, transient faults occur far more often than permanent ones, and are also far harder to detect. Intermittent faults: Caused by unstable hardware or varying hardware states. They can be repaired by replacement or redesign.

16. Software faults Function faults: Incorrect or missing implementation that requires a design change to be corrected. Algorithm faults: Incorrect or missing implementation that can be fixed without the need of design change. Timing/serialization faults: Missing or incorrect serialization of shared resources. Checking fault: Missing or incorrect validation of data, or incorrect loop, or incorrect conditional statement. Assignment fault: Values assigned incorrectly or not assigned.

17. Mutation Testing Can trace birth of Mutation Testing back to a student paper written in 1971 To assess the quality of the tests by performing them on mutated code To use these assessments to help construct more adequate tests To thereby produce a suite of valid tests which can be used on real programs

18. Testing Method Mutant processes are created to try to mimic typical syntactic errors made by programmers Many differing mutants are run against the specified tests to assess the quality of the tests The tests are attributed a score between 0 and 1, as to whether they can distinguish between the original and the mutants

19. Traditional Syntactical Mutation Operators Deletion of a statement Boolean: Replacement of a statement with another eg. == and >=, < and <= Replacement of boolean expressions with true or false eg. a || b with true Replacement of arithmetic eg. * and +, / and - Replacement of a variable

20. If process is not error-free, fix it Test Mutants The Mutation Process Process Mutation Tests Test Process Create Mutants Yes Test Complete No Any Live Mutants? Problem with Tests? Any Mutations that are caught by tests are killed New Test Data

21. limitation Mutation testing can only target a sub-set of all the possible faults a programmer could possibly make. Therefore it cannot be guaranteed that all possible errors can be detected with the test cases.