Testing, Reliability, and Interoperability Issues in CORBA Programming Paradigm Gang Xing and Michael R. Lyu The Chinese University of Hong Kong 6th Asia-Pacific Software Engineering Conference December 7-10 1999 2019/1/17

Outline Motivation Testing CORBA Program Reliability Evaluation Interoperability Future Work Q&A 2019/1/17

Motivation Implementing distributed applications with CORBA becomes popular Testing, reliability, and interoperability for the CORBA programming paradigm remain unexplored These issues are important to develop the reliable distributed system in CORBA Why we talk about issues about CORBA? Lots of company use CORBA as its enterprise distributed system solution. Although CORBA can help develop distributed system. Mechanisms for reliability and testing important! 2019/1/17

What is CORBA? Architecture and specification Allow object in the network communicate through an "interface broker" Developed by Object Management Group (OMG) Standard architecture for distributed objects (or Components) An architecture and specification for creating, distributing, and managing distributed program objects in a network Allow programs at different locations and developed by different vendors to communicate in a network through an "interface broker" Developed by a consortium of vendors through the Object Management Group (OMG)--currently includes nearly 800 member companies ISO and X/Open sanction CORBA as the standard architecture for distributed objects (or components) 2019/1/17

Experiment Project Description 19 Programs,same specification Soccer Management System 10 operations team rules Different ORBs(Visigenic/Orbix) Different Languages (C++/Java) Based on same spec provide basic management operation lots of rules of team 2019/1/17

Program Metrics 7 programs use Iona Orbix(C++) 12 programs use Visigenic 9 uses Java for client and server 2 uses C++ for client and server 1 uses Java as client and C++ as server Table? 2019/1/17

General Software Metrics The software metrics of these 19 programs are listed in Table 2. The metrics were collected using etags and some perl scripts. These programs range from 500 to 5000 lines of code (LOC). The large size of program P12 was due to fancy user interface and on-line help commands. The distribution of the client code versus server code is 1.79. 2019/1/17

Testing the Programs Test Preparation Test Execution Result Analysis Generate Test Cases Test Procedure Test Execution Result Analysis In order to evaluate the reliability We try to testing the program 2019/1/17

Generate Test Cases Specification IDL Test Cases 2019/1/17 base on spec and ID, For each operation IDL can help general test cases 2019/1/17

Test Procedure 2019/1/17 Minimum operation use for each test case Test each operation by operation step ? Scripts automatic ? The test procedure is shown in. In order to reduce the testing work for these program versions, we define the test sequence for each operation to cover all the test cases with minimum operations. In each step of test, we only apply one test case. However, we need some extra operations for some test cases to work properly. In total we have 76 operations in the test procedure to cover all the test cases. 2019/1/17

Pass Rate Pj - number of “Pass” cases for program j. Mj -number of “Maybe” cases program j. C- total number of cases apply to the programs (57) The test cases, designed to raise exceptions, can not apply to the program because the client side of the program deliberately forbids them. In this situation, we do not know whether the server is designed properly to raise the exceptions, so we put down “maybe” as the result. The definitions of the pass rate and the reliability in this paper, therefore, consider two conditions, one including the “maybe” case and the other excluding it. Oi -number of “Pass” test cases for operation i OMi- number of “Maybe” test cases for operation i Ti - total cases for operation i 2019/1/17

Test Result for Each Team From Table 4 we see that Team 16 and Team 19 have low pass rate as they fail to process many exceptions properly. In addition, Team 13 has many “maybe” cases due to their special user interface which avoids many illegal test cases designed to test for exception handling. Test Results for Each Team 2019/1/17 Maybe: Some Test cases not applicable

Test Results for Each Operation Operation Pass Rate= A/B discussion From Table 5 we can see that the operations CreateTeam and MovePlayer have the lowest pass rates. The reason is the complexity of these operations, as they need to consider more for the process of normal cases and exceptions. Furthermore, the CreateTeam operation has a large number of “Maybe” results as many programs forbid the test cases which would raise exceptions. 2019/1/17

Defects Classification Exceptions Server Side Missing Extra or Wrong Client Side Wrong Memory Management Ref Count Language Mapping Other This happens when the server side program does not throw the necessary exception. This situation usually comes from the IDL definition problem as addressed above. On the other hand, the implementation may also fails to check if there should be an exception. Extra exception is considered as a wrong one since the client will not be able to recongnize the unexpected exception. The server throws an unexpected exception. This defect seldom occurs. The programmers forget to catch some kind of the expected exception. The programmers catch the exception, but give an incorrect response. This kind of defect occurs when a special process is needed in the exception handling code. "_duplicate()/_release()," sequencs/string/array user interface,parameter process, etc. 2019/1/17

Distribution of Defects Exception: server: missing extra or wrong client: wrong Memory management: traditional/CORBA CORBA: reference count; squence, array, string Other From Table 6, we can see that over 70 percent of the total defects come from exception handling. This indicates that exception handling routines are the most difficult part of CORBA programming for distributed systems. Distribution of Defects *Missing/Extra or Wrong +Missing/Wrong 2019/1/17

Distribution of Defects Exception: Memory management other 2019/1/17

Reliability Develop the Operational Profile Define the Reliability of the Program Analyze the Results Software reliability [LYU96] is the probability of failure-free software operation for a specified time in a specified environment. We use the similar procedure specified in [MUSA97] to evaluate software reliability in our experiment. We note, however, that it is not easy to obtain execution time for CORBA programs, as many factors affect the operation execution time. They include, for example, programming language, platform, ORB implementation, user implementation, etc. Since it is difficult for us to get an accurate execution time measure for each operation in these programs, we evaluate the reliability of each accepted program based on the defects we get from our test and the probability of each operation. We test and evaluate the client and server for each program as a whole, and assume that each test case has the same execution time for the same program. 2019/1/17

Operational Profile Operation Profile: List of occurrence probabilities of each element in the input domain of the program. Because the application in our experiment is a new information management system, the operational profile can not be obtained from any historical data. Consequently, we have to estimate the occurrence probability of each operation. This is shown in Table 7. 2019/1/17

Reliability Evaluation Rj - Reliability for program j Rj' - Reliability for program j (treat “maybe” as pass) OPi,j-“Pass” test cases for operation i , program j Mi,j-“Maybe” test cases for operation i , program j Ti - total cases for operation i i- Probability for operation I n-number of operation Software reliability is the probability of failure-free software operation for a specified time in a specified environment. Since it is difficult for us to get the accurate execution time of each operation in these programs, evaluate the reliability of each accepted program by the defects we get from our test and the probability of each operation. We test and evaluate the client and server for each program as a whole, and assume that each test case has the same execution time for the same program. 2019/1/17

Evaluation Results 2019/1/17

Evaluation Results 2019/1/17

Comparison We also list the average reliability for Visiginic/Orbix program, and Java/C++ programs, as shown in Table 9. From Table 9, we can see that the reliability of Visiginic programs is higher than that of Orbix programs. Moreover, the reliability of the teams using Java is higher than those using C++. The result does not necessary mean that using Visiginic and Java is better than using Orbix and C++. Instead, this may be due to the CORBA mapping for C++, which is more complicated than that for Java. Moreover, the programmers are generally more familiar with Java than with C++. 2019/1/17

Why Interoperability? Same Specification Try to Exchange the Client and Server Availability Higher Reliability Server/Client lead to higher system reliability? 2019/1/17

Interoperability Evaluation The difficulty of Interoperability: 1: very difficult to inter-operate 2: possible to inter-operate, but with considerable effort 3: interoperable with moderate effort. 4: interoperable with some effort. 5: readily interoperable with minimal effort. 2019/1/17

Evaluation Results From Table 10 we can see that there is clearly a lack of interoperability among these 18 program versions (indicated by the low mark of 1.42), even though they are developed based on the same specification requirements. Programs P3, P6, P7, P11, P14, P17, P19, in particular, are extremely difficult to inter-operate with other program versions. Only a few pairs of programs achieve higher interoperability marks, but they are sparse. Same ORB better, subset P2,P10,P12 is better 2019/1/17

Difficulties in Interoperability IDL Interface Attribute/Operation Exception Other Between ORBs CORBA Service Specification 2019/1/17

Summary Introduction to CORBA Testing the CORBA program Evaluating the reliability Evaluating the interoperability 2019/1/17

Future Work Testing Techniques Reliability Evaluation Techniques Apply Software Fault Tolerance Techniques Reliability Model Implement Software Reliability Middleware 2019/1/17

Thank You! 2019/1/17