1. Software Testing Using Model Program DESIGN BY HONG NGUYEN & SHAH RAZA Dec 05, 2005

2. Overview Software Testing Using Model Program Briefly introduction of M-mp testing Briefly introduction of M-mp testing What is M-mp testing? What is M-mp testing? How it work? How it work? Why we need program base testing Why we need program base testing The problem of software testing The problem of software testing Oracle test Oracle test N-Version programming N-Version programming What is N-Version Programming? What is N-Version Programming? Disadvantages Disadvantages Advantages Advantages

3. Overview (continues) M-mp testing M-mp testing Framework for M-mp testing Framework for M-mp testing Asymptotic behavior of M-mp testing Asymptotic behavior of M-mp testing Input domain partitioning Input domain partitioning An M-mp Experiment An M-mp Experiment Background and motivation Background and motivation Selecting the application Selecting the application Phases of the experiment Phases of the experiment Conclusion Conclusion

4. Introduction What is M-mp testing? What is M-mp testing? Testing using M Model program Testing using M Model program Abbreviated to ‘M-mp testing’ Abbreviated to ‘M-mp testing’ It is alternative to software testing base on manual outcome prediction It is alternative to software testing base on manual outcome prediction M-mp testing is base on N-Version programming technique M-mp testing is base on N-Version programming technique Which is apply specially in the well-known software fault tolerant technique. Which is apply specially in the well-known software fault tolerant technique.

5. Introduction (Continues) How it work How it work Model program implements selected part of functional specification of the software to be tested. Model program implements selected part of functional specification of the software to be tested. The M-mp testing strategy requires that M (M>= 1) The M-mp testing strategy requires that M (M>= 1) Model programs as well as the program under test, should be independently developed Model programs as well as the program under test, should be independently developed P and M model program are then subjected to the same test data. P and M model program are then subjected to the same test data. Difference analysis is conducted on the outputs and appropriate corrective action taken Difference analysis is conducted on the outputs and appropriate corrective action taken P and the M model program jointly constituted an approximate test oracle (out come correct or not) P and the M model program jointly constituted an approximate test oracle (out come correct or not)

6. Introduction (Continues) Why we need program base testing Why we need program base testing For small number of test data, it may be feasible to work out the expected output manually For small number of test data, it may be feasible to work out the expected output manually For thorough testing which required large test datasets manual testing could be time consuming and error prone For thorough testing which required large test datasets manual testing could be time consuming and error prone That is why M-mp base testing may be more feasible for those kinds of data set That is why M-mp base testing may be more feasible for those kinds of data set

7. The problem of software testing The difficult task in software testing is to assess the correctness to the outcome of a program that is subjected to particular test input The difficult task in software testing is to assess the correctness to the outcome of a program that is subjected to particular test input In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle problem. In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle problem. Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems. Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems.

8. The problem of software testing (continues) Oracle Test Oracle Test It means of determining whether a program passed or failed a test. It means of determining whether a program passed or failed a test.

9. The problem of software testing (continues) Oracle test (continues) Oracle test (continues) P is the program to be tested P is the program to be tested Test strategy determines the set of test data to be used Test strategy determines the set of test data to be used To get the test result we compare P’s output with the oracle ‘s output To get the test result we compare P’s output with the oracle ‘s output For small out put oracle manually base oracle technique is feasible For small out put oracle manually base oracle technique is feasible But for large industry base application manually base oracle testing is not feasible But for large industry base application manually base oracle testing is not feasible One of alternative of manually base oracle is N- Version programming One of alternative of manually base oracle is N- Version programming

10. N-Version Programming N-Version is comprises N independently written version of the software N-Version is comprises N independently written version of the software All has the same functional specification All has the same functional specification At runtime voting base on majority agreement is used to decided the probable output At runtime voting base on majority agreement is used to decided the probable output Disadvantage : Disadvantage : Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliability Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliability N-Version systems should not be casually assumed to be completed reliable. N-Version systems should not be casually assumed to be completed reliable.

11. N-Version (continues) Disadvantage (continues) Disadvantage (continues) Is the increased development cost because at least three version being required for a voting system to work Is the increased development cost because at least three version being required for a voting system to work Advantage Advantage N-Version design appear to offer more reliability that we can gain from any other way N-Version design appear to offer more reliability that we can gain from any other way Even though the cost of failure is high: Even though the cost of failure is high: It would be more cost effective to build N-Version systems rather than focusing on building ‘one good version’ It would be more cost effective to build N-Version systems rather than focusing on building ‘one good version’

12. M-mp Testing Framework for M-mp testing Framework for M-mp testing

13. M-mp Testing (continues) Framework (continues) Framework (continues) In M-mp testing P is primary program (the program under test) In M-mp testing P is primary program (the program under test) mp1 –mp M are M so called ‘model program of P mp1 –mp M are M so called ‘model program of P M-mp testing treat P and mp1-mp M to be same test input M-mp testing treat P and mp1-mp M to be same test input Any disagreement on the output indicates the presence of specification defects Or software faults in at least one of program Any disagreement on the output indicates the presence of specification defects Or software faults in at least one of program Once defects are detected and removed. The program are re-run Once defects are detected and removed. The program are re-run

14. M-mp Testing (continues) Framework (continues) Framework (continues) The cycle is repeat until all disagreements for a particular dataset are resolved. The cycle is repeat until all disagreements for a particular dataset are resolved. Asymptotic behavior of M-mp testing Asymptotic behavior of M-mp testing Difference between the M-mp and manual approaches to testing is the cost of outcome verification Difference between the M-mp and manual approaches to testing is the cost of outcome verification

15. M-mp Testing (continues) Asymptotic behavior of M-mp testing (continues) Asymptotic behavior of M-mp testing (continues) On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly. On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly. The cost growth is modeled as linear The cost growth is modeled as linear In manual approach, the growth of the output verification cost may be expected to be much steeper. In manual approach, the growth of the output verification cost may be expected to be much steeper. Input domain partitioning Input domain partitioning Divided into a standard domain and an exception domain Divided into a standard domain and an exception domain Exception domain may be split into an incomplete data domain and an invalid data domain. Exception domain may be split into an incomplete data domain and an invalid data domain. The standard domain consists of inputs that a program can process ‘as is’ The standard domain consists of inputs that a program can process ‘as is’

16. M-mp Testing (continues) Input domain partitioning (continues) Input domain partitioning (continues) The invalid domain contains the input that a program can not process it should be rejected with appropriate error message The invalid domain contains the input that a program can not process it should be rejected with appropriate error message The incomplete data domain is ‘between’ the standard and invalid data domains The incomplete data domain is ‘between’ the standard and invalid data domains It is made up of inputs that the program has to completed with default values before processing them It is made up of inputs that the program has to completed with default values before processing them An M-mp Experiment An M-mp Experiment Background and motivation Background and motivation

17. M-mp Testing (continues) Background and motivation (continues) Background and motivation (continues) The primary program is a scheduling application, a non- interactive data processing subsystem of a large logistics management system The primary program is a scheduling application, a non- interactive data processing subsystem of a large logistics management system The program can be characterized as algorithmically complex The program can be characterized as algorithmically complex In practice, the testing process in the company was well defined and supported by standards and procedures but had several drawback In practice, the testing process in the company was well defined and supported by standards and procedures but had several drawback Designing test cases based on manual outcome prediction was not only difficult, error prone and time consuming Designing test cases based on manual outcome prediction was not only difficult, error prone and time consuming But consequently also unattractive and demoralizing But consequently also unattractive and demoralizing

18. M-mp Testing (continues) Selecting the application Selecting the application The scheduling application was chosen as the primary program for following reasons The scheduling application was chosen as the primary program for following reasons Developer was not familiar with the scheduling domain and application Developer was not familiar with the scheduling domain and application The cost of developing from scratch could be assesses more authentically The cost of developing from scratch could be assesses more authentically The risk of correlated failures caused by common design error was reduces The risk of correlated failures caused by common design error was reduces scheduling application is representative of the family of operation management program scheduling application is representative of the family of operation management program The time the experiment started, the scheduling subsystem had been in use for more than year The time the experiment started, the scheduling subsystem had been in use for more than year This seem a good opportunity to test the fault detection ability of M-mp testing, as the likelihood of finding defect was expected to be low This seem a good opportunity to test the fault detection ability of M-mp testing, as the likelihood of finding defect was expected to be low

19. M-mp Testing (continues) Phase of the experiment Phase of the experiment

20. M-mp Testing (continues) Phase of the experiment Phase of the experiment The experiment entailed a test design, a test execution and test evaluation phase as depicted The experiment entailed a test design, a test execution and test evaluation phase as depicted Test design phase comprised test data selection Test design phase comprised test data selection Which includes activities related to test data generation, Model program design and model program implantation Which includes activities related to test data generation, Model program design and model program implantation The test execution phase includes activities relating to setting up the test environment and executing the program The test execution phase includes activities relating to setting up the test environment and executing the program The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the program need to be arbitrated. The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the program need to be arbitrated.

21. Conclusion M-mp approach could test a scheduling program more adequately than manually designed test and at lower cost M-mp approach could test a scheduling program more adequately than manually designed test and at lower cost M-mp testing should be judged on a case by case basic M-mp testing should be judged on a case by case basic M-mp approach will tend to be better option than the manual one when the number of tests required to achieve a particular adequacy level become large M-mp approach will tend to be better option than the manual one when the number of tests required to achieve a particular adequacy level become large

22. Question Finished Finished Please be easy on us.Thank you Please be easy on us.Thank you