Complex Test Suites Organization Victor Kuliamin ISP RAS, Moscow

2/16 Modern Software Complexity Modern software systems Are huge and have a lot of functions Have very complex interfaces Are developed by hundreds of people System Year Size(MLOC) Windows Windows NT Windows NT Windows Windows NT Solaris Red Hat Linux Debian Windows Red Hat Linux Sun StarOffice Debian Red Hat Linux Windows XP Red Hat Linux Debian Windows Vista Beta Fedora Core Debian Mac OS X System Year DevTeam Size Windows NT Windows NT Windows NT Debian * Debian * Windows Debian * Debian * Debian *

3/16 Test Suites Complexity Scalability of testing techniques is strongly related with test suite organization System Size(MLOC) Test suite size(MLOC) Java (JCK) System Year TestTeam Size Windows NT Windows NT Windows NT (0.9) Windows (1.2) System Test Cases MS Word XP Oracle 10i Windows XP >

4/16 Traditional Test Suite Organization Set of test cases Suitable for test suite size management Suitable for failure analysis Test suites for complex systems became huge and poorly manageable Many common parts and low reuse Hard to modify Test suite

5/16 Full Set of Problems Test execution Selection the necessary tests Test suite configuration Test result analysis Failure diagnostics Test adequacy reporting Test creation and maintenance Explicit relations with requirements, testing goals, and components under test Fight with complexity Reuse

6/16 Approaches Attributing Configuration parameters Modularization

7/16 Attributing Adding a set of test case attributes Very simple Provides multi-aspect tests classification Provides links to requirements and components under tests Can help to measure test completeness Test suite 1x3F7 /tests/ usr/t1 2AB41 /tests/ sys/t2 3x9E /tests/ sys/t3

8/16 Configuration Parameters A set of parameters that control test execution and test results assessment Provide additional flexibility, portability and adaptability of test suite Help to check optional requirements May be assigned dynamically May make test suite too complex

9/16 Modularization Separation of modules performing specific tasks Facilitates reuse Enhances maintainability Facilitates requirements traceability May complicate test suite May complicate failure analysis

10/16 Right Modules Test run-time support Scheduler Timer Event monitor Test adapter Test data generator Test oracle

11/16 Test Adapters Make possible reusable and abstract tests TestSUTAdapter

12/16 Test Data Generators Provide data for tests Can contain filters or selectors Can be composed from smaller data generators

13/16 Test Oracles Provide verdict of SUT behavior correctness Test Data Generator System under Test Test Oracle Partial oracles - Event or operation oracle - Data integrity constraint Enhance reuse Simplify requirements traceability ?

14/16 Oracle Composition Simple case Any failure detected by partial oracle is global ­ Complex data integrity constraints Behavior of distributed systems Interleaving semantics

15/16 Conclusion Test suite organization techniques improving testing scalability already exist The most powerful one is modularization But usage of these techniques is restricted by existing tools All the techniques presented need proper support in tools

16/16 Thank you for attention! Any questions?