1. www.eu-etics.org INFSOM-RI-026753 WP4: Testing tools and methodologies WP4: Testing tools and methodologies István Forgács 4D SOFT

2. INFSOM-RI-026753 Team 4D SOFT’s staff – some changes4D SOFT’s staff – some changes –István Forgács - responsible for –leading WP4 –4D Soft’s own methods and tools –Key scientific issues –Cooperate with the coordinator, visiting PMBs, etc. –Anna Bánsághi, –Study and evaluate testing tools –Katalin Gergely, (Thalmeiner Zsolt) –Study and evaluate testing methods –Cooperates with Univ. of Wisconsin staff –Andor Dirner –DILIGENT connection –Gyöngyi Kispál –Financial issues –Klára Tauszig –Administrative issues

3. INFSOM-RI-026753 A4.1 requirements A4.1 Collect requirements for unit, functional and regression testing. There is a large number of research papers on the test adequacy criteria of unit testing. However, there are no related tools in the practice for most of them. We are going to collect applicable and practical requirements based on our knowledge, experience and the testing tools currently available and widely used such as JUnit. The task involves the requirements and applicability criteria for test planning, dynamic testing and the testing tools.

4. INFSOM-RI-026753 Work has been done General requirements for test automationGeneral requirements for test automation –Studying the literature especially the book of Fewster and Graham: Software Test Automation –Key result: test automation should be based on good test methods Requirements for test planningRequirements for test planning –Easy to learn, –Easy to use –Easy to maintain –Can be automated –Can be generally used –Needs more invention and less manual work

5. INFSOM-RI-026753 Methods Methods have been studiedMethods have been studied –Method for the whole testing process –V model –Test planning –Category-partitioning (detailed description tomorrow) –Syntax testing – can be automated, checks correct and incorrect syntax –Equivalence Partitioning – input domain is partitioned into sub domains so that one sub domain behaves uniquely w.r.t. the errors –widely used –disadvantages Doesn't test every inputDoesn't test every input No guidelines for choosing inputsNo guidelines for choosing inputs Heuristic basedHeuristic based very limited focusvery limited focus

6. INFSOM-RI-026753 Methods Methods have been studiedMethods have been studied –Test planning –Boundary Value Analysis – together with the equivalence partitioning –Well-known requirements that boundaries and close to the boundaries should be tested –The boundaries of both valid and invalid partitions are considered –Disadvantages Does not test all possible inputsDoes not test all possible inputs Does not test dependencies between combinations of inputsDoes not test dependencies between combinations of inputs –Error guessing – the experience of the tester is used to postulate what faults might occur, and to design tests specifically to expose them. –These three are used together.

7. INFSOM-RI-026753 Methods Methods have been studiedMethods have been studied –Test planning –Cause-Effect Graphing: old fashioned, not maintainable –State transition diagrams: widely used if the software contains states and different events cause the transition from one to another example ATM –Random testing – random input for test cases. The distribution (normal, uniform, etc.) to be used in the generation of random input values shall be based on the expected operational distribution of inputs. –Ad hoc testing unfortunately it’s widely used –Fuzz input testing - lots of arbitrary, mainly invalid input –Smoke testing: is a cursory examination of all of the basic components of a software system to ensure that they work –Equivalence testing this is the best method if you have two different software aiming to do the same things

8. INFSOM-RI-026753 methods Unit, component and integration testingUnit, component and integration testing –Black box methods –Test planning methods –EBIT (tomorrow) –White box methods –Test bed –Test driver – missing callers –Test stubs – missing callees –Coverage analyses –All statement criterion –All branch criterion – most widely used minimum requirement –MC/DC criterion –Data flow criteria – better but rarely applied –Reduced path testing – cycles are tested with 0,1 and more iterations

9. INFSOM-RI-026753 methods System testingSystem testing –Scripting languages –Capture and play back techniques –Static analysis –Dependence analysis (slicing) –Metrics –Exception handling analysis –Null pointer analysis –Non-initialised variable analysis

10. INFSOM-RI-026753 methods Regression testingRegression testing –Dynamic regression testing –Full regression testing –Selective regression testing –Selection modification-traversing tests –Classifying regression tests –At least one output has been influenced –Influences through predicates –No influences exist –If necessary transforming tests to be effective –Static regression testing –Original and modified sources are available –Dependence analysis considering the modifications –Selecting the non-expected influences

11. INFSOM-RI-026753 A4.1 Collect requirements for tools Testing tool selectionTesting tool selection –Requirements can only be collected based on existing tools –We studied tools for different purposes –Test planning - No specification-based tools have been considered studying the specification, detailed design, etc. documents for DILIGENT –CatGen –Unit testing –Junit –GJTester –Functional, regression –vTest –Static analysis –INDUS

12. INFSOM-RI-026753 A4.1 Collect requirements for tools Testing tool selectionTesting tool selection –Acceptance testing –AutAT (web appl) –Accessibility testing –CSE Lite (web appl test tool) –A-Prompt (web appl test tool for people with disabilities) –PowerTools HTTP proxy –PowerTools Connection Test –PowerTools HTTP Request Editor –PowerTools Expression Test –PowerTools Encode/Decode –Performance testing (load, stress) –OpenSTA –TestMaker –Grinder –Jmeter –LoadRunner –Analog –Coverage –JavaCov

13. INFSOM-RI-026753 Immediate future plan More testing tools to studyMore testing tools to study –Functional –Qengine (performance) –Enterprise Web Test (stress, reliability) –AppPerfect WebTest (load) –JFunc –Conformance –SuiteRunner (unit) –Regression, smoke –ITP –Integration –iValidator –Performance –TPTP

14. INFSOM-RI-026753 Immediate future plan More testing tools to studyMore testing tools to study –Component –ASP Component Test –Code analyser –PMD –Coverage –Jcoverage –Unit –AppPerfect DevTest

15. INFSOM-RI-026753 Immediate future plan We have finished studying testing methodsWe have finished studying testing methods Requirement analysisRequirement analysis –Starting points –Methods available –Testing tools available –Testing research –Test books for testing –Setting requirements –Weighting requirements –Validating and comparing requirements

16. INFSOM-RI-026753 Deliverables We haven’t any scheduled deliverables yetWe haven’t any scheduled deliverables yet Future deliverables in this yearFuture deliverables in this year –D4.1 - Distributed test execution system (prototype) in PM06. Lead participant: UoW –D4.2 - Requirements for unit, functional and regression testing in PM06. Lead participant: 4DS

17. INFSOM-RI-026753 metrics Number of testing tools downloaded and studiedNumber of testing tools downloaded and studied –Plan 20-25current status 20 Number of test methods studiedNumber of test methods studied –Plan 20-30current status 23 Number of quality metrics identified for test methods and tool selectionNumber of quality metrics identified for test methods and tool selection –Plan 15-20current status 0

18. INFSOM-RI-026753 Future plan Selection of the testing tools and methodologies based on the requirements derived in the previous partSelection of the testing tools and methodologies based on the requirements derived in the previous part Working out special aid on how to implement and use these tools/methods with clear examples explaining the main difficulties of the application.Working out special aid on how to implement and use these tools/methods with clear examples explaining the main difficulties of the application.