Testing, Reliability, and Interoperability Issues in CORBA Programming Paradigm 11/21/2018

Outline Motivation Testing CORBA Program Reliability Evaluation Interoperability Future Work Q&A 11/21/2018

Motivation Implement distributed applications with CORBA become popular Testing, reliability, and interoperability for the CORBA programming paradigm remain unexplored These issues is important to develop the reliable distributed system in CORBA Why we talk about issues about CORBA? Lots of company use CORBA as its enterprise distributed system solution. Although CORBA can help develop distributed system. Mechanisms for reliability and testing important! 11/21/2018

What is CORBA? Architecture and specification Allow object in the network communicate through an "interface broker" Developed by Object Management Group (OMG) Standard architecture for distributed objects (or Components) An architecture and specification for creating, distributing, and managing distributed program objects in a network Allow programs at different locations and developed by different vendors to communicate in a network through an "interface broker" Developed by a consortium of vendors through the Object Management Group (OMG)--currently includes nearly 800 member companies ISO and X/Open sanction CORBA as the standard architecture for distributed objects (or components) 11/21/2018

Experiment Project Description 19 Programs,same specification Soccer Management System 10 operations team rules Different ORBs(Visigenic/Orbix) Different Languages (C++/Java) Based on same spec provide basic management operation lots of rules of team 11/21/2018

Program Metrics 7 programs use Iona Orbix(C++) 12 programs use Visigenic 9 uses Java for client and server 2 uses C++ for client and server 1 uses Java as client and C++ as server Table? 11/21/2018

General Software Metrics The software metrics of these 19 programs are listed in Table 2. The metrics were collected using etags and some perl scripts. These programs range from 500 to 5000 lines of code (LOC). The large size of program P12 was due to fancy user interface and on-line help commands. The distribution of the client code versus server code is 1.79. 11/21/2018

Testing the Programs Test Preparation Test Execution Result Analysis Generate Test Cases Test Procedure Test Execution Result Analysis In order to evaluate the reliability We try to testing the program 11/21/2018

Generate Test Cases Specification IDL Test Cases 11/21/2018 base on spec and ID, For each operation IDL can help general test cases 11/21/2018

Test Procedure 11/21/2018 Minimum operation use for each test case Test each operation by operation step ? Scripts automatic ? The test procedure is shown in. In order to reduce the testing work for these program versions, we define the test sequence for each operation to cover all the test cases with minimum operations. In each step of test, we only apply one test case. However, we need some extra operations for some test cases to work properly. In total we have 76 operations in the test procedure to cover all the test cases. 11/21/2018

Pass Rate Pj - number of “Pass” cases for program j. Mj -number of “Maybe” cases program j. C- total number of cases apply to the programs (57) The test cases, designed to raise exceptions, can not apply to the program because the client side of the program deliberately forbids them. In this situation, we do not know whether the server is designed properly to raise the exceptions, so we put down “maybe” as the result. The definitions of the pass rate and the reliability in this paper, therefore, consider two conditions, one including the “maybe” case and the other excluding it. Oi -number of “Pass” test cases for operation i OMi- number of “Maybe” test cases for operation i Ti - total cases for operation i 11/21/2018

Test Result for Each Team From Table 4 we see that Team 16 and Team 19 have low pass rate as they fail to process many exceptions properly. In addition, Team 13 has many “maybe” cases due to their special user interface which avoids many illegal test cases designed to test for exception handling. Test Results for Each Team 11/21/2018 Maybe: Some Test cases not applicable

Test Results for Each Operation Operation Pass Rate= A/B discussion From Table 5 we can see that the operations CreateTeam and MovePlayer have the lowest pass rates. The reason is the complexity of these operations, as they need to consider more for the process of normal cases and exceptions. Furthermore, the CreateTeam operation has a large number of “Maybe” results as many programs forbid the test cases which would raise exceptions. 11/21/2018

Defects Classification Exceptions Server Side Missing Extra or Wrong Client Side Wrong Memory Management Ref Count Language Mapping Other This happens when the server side program does not throw the necessary exception. This situation usually comes from the IDL definition problem as addressed above. On the other hand, the implementation may also fails to check if there should be an exception. Extra exception is considered as a wrong one since the client will not be able to recongnize the unexpected exception. The server throws an unexpected exception. This defect seldom occurs. The programmers forget to catch some kind of the expected exception. The programmers catch the exception, but give an incorrect response. This kind of defect occurs when a special process is needed in the exception handling code. "_duplicate()/_release()," sequencs/string/array user interface,parameter process, etc. 11/21/2018

Distribution of Defects Exception: server: missing extra or wrong client: wrong Memory management: traditional/CORBA CORBA: reference count; squence, array, string Other From Table 6, we can see that over 70 percent of the total defects come from exception handling. This indicates that exception handling routines are the most difficult part of CORBA programming for distributed systems. Distribution of Defects *Missing/Extra or Wrong +Missing/Wrong 11/21/2018

Distribution of Defects Exception: Memory management other 11/21/2018

Reliability Develop the Operational Profile Define the Reliability of the Program Analysis the Results Software reliability [LYU96] is the probability of failure-free software operation for a specified time in a specified environment. We use the similar procedure specified in [MUSA97] to evaluate software reliability in our experiment. We note, however, that it is not easy to obtain execution time for CORBA programs, as many factors affect the operation execution time. They include, for example, programming language, platform, ORB implementation, user implementation, etc. Since it is difficult for us to get an accurate execution time measure for each operation in these programs, we evaluate the reliability of each accepted program based on the defects we get from our test and the probability of each operation. We test and evaluate the client and server for each program as a whole, and assume that each test case has the same execution time for the same program. 11/21/2018

Operational Profile Operation Profile: List of occurrence probabilities of each element in the input domain of the program. Because the application in our experiment is a new information management system, the operational profile can not be obtained from any historical data. Consequently, we have to estimate the occurrence probability of each operation. This is shown in Table 7. 11/21/2018

Reliability Evaluation Rj - Reliability for program j Rj' - Reliability for program j (treat “maybe” as pass) OPi,j-“Pass” test cases for operation i , program j Mi,j-“Maybe” test cases for operation i , program j Ti - total cases for operation i i- Probability for operation I n-number of operation Software reliability is the probability of failure-free software operation for a specified time in a specified environment. Since it is difficult for us to get the accurate execution time of each operation in these programs, evaluate the reliability of each accepted program by the defects we get from our test and the probability of each operation. We test and evaluate the client and server for each program as a whole, and assume that each test case has the same execution time for the same program. 11/21/2018

Evaluation Results 11/21/2018

Evaluation Results 11/21/2018

Comparison We also list the average reliability for Visiginic/Orbix program, and Java/C++ programs, as shown in Table 9. From Table 9, we can see that the reliability of Visiginic programs is higher than that of Orbix programs. Moreover, the reliability of the teams using Java is higher than those using C++. The result does not necessary mean that using Visiginic and Java is better than using Orbix and C++. Instead, this may be due to the CORBA mapping for C++, which is more complicated than that for Java. Moreover, the programmers are generally more familiar with Java than with C++. 11/21/2018

Why Interoperability? Same Specification Try to Exchange the Client and Server Availability Higher Reliability Server/Client lead to higher system reliability? 11/21/2018

Interoperability Evaluation The difficulty of Interoperability: 1: very difficult to inter-operate 2: possible to inter-operate, but with considerable effort 3: interoperable with moderate effort. 4: interoperable with some effort. 5: readily interoperable with minimal effort. 11/21/2018

Evaluation Results From Table 10 we can see that there is clearly a lack of interoperability among these 18 program versions (indicated by the low mark of 1.42), even though they are developed based on the same specification requirements. Programs P3, P6, P7, P11, P14, P17, P19, in particular, are extremely difficult to inter-operate with other program versions. Only a few pairs of programs achieve higher interoperability marks, but they are sparse. Same ORB better, subset P2,P10,P12 is better 11/21/2018

Difficulties in Interoperability IDL Interface Attribute/Operation Exception Other Between ORBs CORBA Service Specification 11/21/2018

Summary Introduction to CORBA Testing the CORBA program Evaluating the reliability Evaluating the interoperability 11/21/2018

Future Work Testing Techniques Reliability Evaluation Techniques Apply Software Fault Tolerance Techniques Reliability Model Implement Software Reliability Middleware 11/21/2018

Q&A 11/21/2018

Thank You! 11/21/2018