1. Testing strategy of DILIGENT 4D Soft Ltd.

2. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 2 Introduction Testing of complex systems is very difficult and necessitates a detailed plan of the testing methods, strategy and adequacy criteria to be applied. New techniques require new testing methods as well. Real study of lack of test planning: Web application required of serving 10 000 user in parallel. After installation only 3 users could access in the same time. One of the most critical effect on testing is the stability of the specification and the design of application

3. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 3 Summary of testing process Comprehensive testing plan - the fundamental guidance for the whole testing process Contains: starting conditions of present plan – accepted specification documents subject of the testing testing organisation Selecting methods and tools for different testing phases defines the constraints risk management elaboration of quality assurance plan defines the exit criterion Start date: Jan. 2005 – end date: Apr. 2005

4. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 4 Elaboration of testing plan – Subject of the testing Defining the subject of testing Diligent’s services Interface testing with gLite (all the relevant interfaces with Diligent, we do not test gLite) Interface testing with other systems Specification documents define the subjects more precisely

5. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 5 Elaboration of testing plan -Test management Setting up the testing organisation, test management István Forgács: project manager Responsibilities: general project leader, professional guidance, deciding on strategic testing issues Éva Takács Responsibilities: technical coordinator, relationship with Diligent and other organisations involved in the project, relationship with development (tutorial on unit testing, etc for development) Klára Tauszig Responsibilities: administrative competencies Test team

6. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 6 Elaboration of testing strategy - Test methodology for the different testing phase Test planning – we planned to use category-partition method for all phases of testing. This eases test planning and gives a general view of testing. Maybe the original method should be extended to involve performance, load and stress, etc. testing Unit testing – we propose test driver and coverage tools to be applied. All-decision test data adequacy criterion seems to be enough and a minimum of 90% coverage of each unit, but the goal is 100%. Black box and white box testing together. Integration and component testing – black box testing. Test automation would be useful.

7. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 7 Test methodology for the different testing phase - continued System testing – it covers lots of testing activities on the entire system. Functional testing is essential. Load and performance testing is critical. Traditional testing tools probably cannot be applied. New methods and may be home tools are necessary. DILIGENT and EGEE testing effort can be united. Acceptance testing – the only criterion is that different team and methods are to be applied. Regression testing. Also very critical.

8. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 8 Elaboration of testing strategy - Using of testing tools Large projects such as DILIGENT necessitate of using testing tools The selection of testing tools is a difficult task since there are numerous tools but most of them is not applicable at all. 4D Soft gives advice for the selection of unit testing tools, and methodology for unit testing. The main criteria are: Easy-to-use test driver and stub functions Test code and data should be separated New test data are possible to be inserted by the tester without any recompilation of the system Simple coverage analyser promoting all-decision criterion

9. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 9 Elaboration of testing strategy - Selecting of testing tools Only some tools should be selected for evaluation Only downloadable tools are to be selected Existing experience is taken into consideration Goals and requirements should be clearly identified and quantified Requirements are usability, learning curve, price, etc Requirements are usability, learning curve, price, etc. Real-life applications instead of toy programs should be applied for tool evaluation Multi-functional tools have an advantage over more different tools

10. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 10 Elaboration of testing strategy - Test automation Test automation means test repetition and using testing tools. Test automation adds an additional cost of 25%, therefore we apply testing tools where intensive regression testing is necessary. This is almost always the case, but we should investigate the components of the system one by one carefully. If test automation is necessary, but no applicable tools on the market, home tools should be implemented. Test planning can be automated only in part.

11. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 11 Elaboration of testing strategy - Selection of testing criteria indicates when the testing can be finished Unit testing all the program statements should run at least once – too week all the program branches should run at least once - appropriate all the conditions should run at least once – too strong Component and system testing The total number of test cases is an appropriate criterion (between LOC/10 and LOC/20) MTBF

12. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 12 Elaboration of testing strategy - Testing environment Currently there are several open questions: Which are the services\functions that can be tested on one computer? Is it possible to develop a minimal environment at 4D Soft site A more distributed testing environment for Diligent Nr of sites (minimum (3?), average, maximum Nr of computers at each site? Is there or will be any Grid testing environment for DILIGENT? Test environment should be specified in common with the design team

13. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 13 Elaboration of testing strategy - Test documents It would be reasonable to use the same tool for bug reporting than used by the EGEE team Using gLite we may find failures in it, since correct program does not exist Detailed test documents are generated by the testing tools. Final test evaluation documents are made manually

14. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 14 Elaboration of testing strategy – Walk through of documents Methods for error discovery Walk-through: (T1.1-Functional and Architectural description) Tests the documents integrity, understandability, clarity and self-consistence Usability for the test cases design What kind of test cases can be designed having these documents?

15. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 15 Elaboration of testing strategy - Risk analysis We should assume the main risks specific to DILIGENT defining risk matrix Contingency planning for the critical risks

16. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 16 Elaboration of testing strategy - Risk analysis Generally, the main risks of testing are as follow: risk of the project – changing the specification and the design on the fly, the code does not conform with the original plan, etc. The budget of testing is insufficient – almost always the case Late delivery risk of the resources (knowledge, human factors) risk of technology (new, without experience) human related risks (missing co-operation)

17. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 17 Test planning Test plans defines the test specification and then the test cases for different testing phases (unit, integration, system, etc.) Method used for designing test plans: Category partitioning Basic function: functionality that corresponds to a use-case Category: basic entity of a function that is expressed in the related design document Choice: different states of a category Advantage: test case set is reduced from “universe”, no important test cases are left out

18. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 18 Designing of the test plans – using specification Test case design method used: category partitioning Required documents that serve as input for designing test cases are the following: Functional test cases – functional specification (T1.1) Integration test cases – architectural specification (T1.2) ---------------------------------------------------------------- Unit test cases – based detailed design documents

19. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 19 Tutorial for developers The focus is on unit testing (methods, tools) Guidelines on: Defining test specification Generating black box test cases from test specification Running black box test cases Checking coverage by the testing criterion Selecting white box test data Selecting unit testing tools

20. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 20 Component and integration test Performed on a single computer All the unit tested components will be tested using test cases defined by category-partitioning method Component’s integration will be tested together with the component test We intend to apply EBIT technique Components are tested in a predefined order according to the data flow Testing tool could be applied Some test code should be inserted into the code

21. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 21 DILIGENT testing emphasizing Grid Traditional functional testing requires complex functionality to be validated involving distributed environment Load, stress and performance testing require new methods We try to extend category-partition method to plan test cases for all testing phases above. Since the method is very general we believe that it is possible. We also develop an in house tool the automate category- partition planning We try free load testing tools to see whether existing tools are applicable

22. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 22 Regression testing Traditional regression testing executes all the test cases and investigates those cases for which the original and the executed results are different Selective tools are not available on the market We should plan regression for different test phases. Only tests that requires more than two iterations is well worth automating If possible we apply static regression testing, new tools are to be developed at 4D Soft

23. 4D SOFT DILIGENT-EGEE Interaction - Content and Metadata Management and Testing meeting CERN, 16th December 2004 23 Static and selective regression testing Static regression testing determines all the (output) variables that are influenced by a given code modification The programmer select which variables are intentionally affected, the other influences are due to program defect, and should be fixed Only tests that influences the required variables are executed, thus few test cases should be executed Now the goal is that the original and the modified result be different. If this is not the case the test case should be transformed to be effective.