C IT L AB : a Laboratory for Combinatorial Interaction Testing Angelo Gargantini Università di Bergamo - Italy Joint work with Paolo Vavassori In collaboration with Andrea Calvagna Università di Catania Workshop on Combinatorial Testing ICST 2012 April 17, 2012

user Some open issues in CT user perspective Still missing A common abstract language A common abstract language with a precise semantics for concepts like constraints, seeds … A standard concrete syntax/grammar A standard concrete syntax/grammar or exchange format for tools: to guarantee interoperability Limited assistance in writing the models Limited assistance in writing the models generation tools do not offer any editing capabilities (only a grammar and a parser) and are rarely integrated in any IDE for programming or design. Angelo Gargantini - CitLab: a Laboratory for Combinatorial

researcher Some open issues in CT researcher perspective No standard language  Comparison among tools and algorithms is difficult No standard language  Comparison among tools and algorithms is difficult Conceptual point of view: some features may be supported in an approach but not in another or different meanings Practical: every comparison is quite unreliable since every user must redefine in its own language and tool the examples No help in defining new algorithms and tools No help in defining new algorithms and tools In order to experiment a new test generation algorithm, a researcher should define a proper grammar and a parser, develop the libraries to manipulate the model data, and translate the benchmarks Angelo Gargantini - CitLab: a Laboratory for Combinatorial

C IT L AB in brief (Domain Specific) Language abstract language 1.A rich abstract language with a precise formal semantics concrete syntax 2.A concrete syntax with a well defined grammar by XTEXT editor 3.A textual editor integrated in the eclipse IDE A Framework 1.based on the Eclipse Modeling Framework (EMF) which provides tools and run-time support to manipulate combinatorial problems in Java 2.A rich collection of Java utility classes and methods 3.For introducing new test generation algorithms 4.For introducing code translators for importing and exporting models and tests Angelo Gargantini - CitLab: a Laboratory for Combinatorial

A DSL for CIT problems Domain-specific languages (DSLs) are languages tailored to a particular problem domain. They offer substantial gains in expressiveness and ease of use compared with general-purpose programming languages CIT can be considered a specific application domain for which developing a domain-specific language is worthwhile. DSL development is supported by several tools and technologies. Xtext Xtext Angelo Gargantini - CitLab: a Laboratory for Combinatorial Xtext is a framework for development of domain specific languages. It covers all aspects of a complete language infrastructure, from parsers, over linker, compiler or interpreter to fully-blown top- notch Eclipse IDE integration.

.ecore (metamodel) Language project Angelo Gargantini - CitLab: a Laboratory for Combinatorial.xtxt Grammar Generator Constraints Antlr ParserEditor Syntax coloring Manual (required) Manual (optional) Editor Project Generated Java API Language Model Outline Content Assist Template Formatting…

A DOMAIN SPECIFIC LANGUAGE FOR CIT PROBLEMS Angelo Gargantini - CitLab: a Laboratory for Combinatorial

C IT L AB Language in a glance Model Types:... end Parameters:... end Constraints: #... # end Seeds: #... # end TestGoals: #... # end Angelo Gargantini - CitLab: a Laboratory for Combinatorial Parameters Types Constraints Seeds Test goals

Parameters and their types To describe a combinatorial problem would be sufficient to specify the number of variables and their cardinality. CITLAB language forces the designer to name parameters and to specify their types by listing all the values in their domain. Angelo Gargantini - CitLab: a Laboratory for Combinatorial Parameters: Boolean Viewer; Range textLines [ ]; Enumerative display { 16MC 8MC BW }; end 1.Enumerative 1.Enumerative for parameters that can take a value in a set of symbolic constants. 2.Boolean 2.Boolean for parameters that can be either true or false. 3.Numerical 3.Numerical value for parameters that take any value in an integer range. Choice: explicit parameter names to facilitate the modeling of real systems and to ease the specification of constraints and seeds From ACTS

Named Types Types can be defined with their name in the Types section to be used in parameters declaration Instead of an “anonymous” type. Angelo Gargantini - CitLab: a Laboratory for Combinatorial Types: EnumerativeType cameraType { 2MP 1MP NOC }; end Parameters: Enumerative rearCamera : cameraType; Enumerative frontCamera : cameraType;... end Advantages: the use of named parameter types to make more compact and more maintainable the models in case many parameters share the same domain.

Constraints Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Constraints Semantics Angelo Gargantini - CitLab: a Laboratory for Combinatorial

GF vs forbidden combinations Most CIT that support constraints, allow only the use of forbidden combinations or tuples A forbidden combination would be translated in our model as a not statement. a forbidden pair x = a; y = b would be represented # not (x = a and y = b) # Angelo Gargantini - CitLab: a Laboratory for Combinatorial Advantages: Much more compact than forbidden tuples Example: if b {x1 … xn } “a = x => b = xk” would need n-1 forbidden tuples More expressive (whilst in a finite domain …) More similar to the natural language No need to distinguish between implicit and explicit constraints

Seeds The testers can also force the inclusion of their favorite test cases by specifying them as seed tests. They can be tests generated by other criteria Critical complete combinations The seeds must be included in the generated test set without modification CITLAB considers only complete seeds, i.e., seeds that assign a valid value to each parameter # frontCamera==1MP, display==16MC, Viewer=true, rearCamera==2MP, textLines=30 # Partial seeds? Critical partial combination the tester would like to be included in the test suite Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Test Goals Critical situation that must be tested Predicates that must be covered by some tests in GF Test goals can be again in GF (as the constraints): // at least one camera is missing and the display has at least threshold lines ( rearCamera==NOC or frontCamera==NOC) and textLines>=threshold Partial combinations // both rearCamera and frontCamera are NOC and display is 16MC rearCamera==NOC and frontCamera==NOC and display==16MC Remarks: Checking feasibility of test goals is an NP-complete problem, since it can be reduced to a SATisfiability problem Checking if a test covers a test goal is however linear Angelo Gargantini - CitLab: a Laboratory for Combinatorial

C IT L AB EDITOR Angelo Gargantini - CitLab: a Laboratory for Combinatorial DEMO

Editor features Syntax Coloring Content Assist Template Proposals Rich Hover Rename Refactoring Quick Fixes Outline Folding Hyperlinks for all Cross References Find References Toggle Comment Mark Occurrences Formatting Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Example, model validation XTEXT provides several levels of validation for the defined language the user can specify additional constraints for the model by providing validation fragments Angelo Gargantini - CitLab: a Laboratory for Combinatorial Example: No seed can violate any constraint Note: checking if a seed violates a constraints is decidable in a linear time

AN EXTENSIBLE FRAMEWORK FOR CIT Angelo Gargantini - CitLab: a Laboratory for Combinatorial

C IT L AB : as a extensible framework Angelo Gargantini - CitLab: a Laboratory for Combinatorial CitLab framework Random generator test generators exporters/importers For the definition and implementation of test generators and a set of exporters/importers to and from other languages Based on the eclipse plugin development environment Goals to foster tools interoperability to favor reuse of artifacts to allow fair comparison Casa exporter gene rator exporter

Extension points and extensions Angelo Gargantini - CitLab: a Laboratory for Combinatorial CitLab accepts new contributions as plugins extension points by defining extension points extensions Contributors add new plugins by implementing extensions Declare the extension in.xml Declare the extension in.xml Implement an Interface Implement an Interface

C IT L AB : extension points Extension PointInterface/Methods TestGeneratorICitLabTestGenerator TestSuite generateTests(Model m, int nWise) boolean acceptConstraints(List c) boolean acceptSeeds(List s) boolean acceptTestGoals(List tg) ExporterICitLabExporter void export(Model m) boolean acceptConstraints(List c) boolean acceptSeeds(List s) boolean acceptTestGoals(List tg) ImporterICitLabImporter Model import(Reader r) TestSuiteExporterICitLabTestSuiteExporter void save(TestSuite ts, String file) Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Using Model to Text or Model to Model MDE approach: a possible way to define importers and exporters is to use Model to Text (M2T) or Model to Model (M2M) transformations For instance, Xtend2 for CASA exporter def toCasaCode(Model sm,int n){ ''' «n» «sm.parameters.size» «FOR param :sm.parameters»«param.getSize» «ENDFOR» ''' } Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Test Suite meta-model and Java utils a simple meta-model for tests and test suites We have developed several classes and methods capable to perform routine tasks like generating all the test requirements for a given strength, checking if a test or a seed violates some constraints checking if a test goal is covered by a given test suite … Angelo Gargantini - CitLab: a Laboratory for Combinatorial

CitLab site and Licence issues All the code is available at CitLab is released under the Eclipse Public License (EPL) an open source software license used by the Eclipse Foundation a business-friendly free software license copyleft but weaker than other, like GNU General Public License (GPL) can use, modify, copy and distribute the work and modified versions, in some cases being obligated to release their own changes CitLab Plugins can be commercial (distribute as binaries) Any modification of the CitLab framework must be open sourced The EPL is approved by the Open Source Initiative (OSI) and is listed as a "free software license" by the Free Software Foundation (FSF). Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Status and Future work Adding new plugins is easy (as student projects) Paolo Vavassori has a part-time contract for 1 year Already implemented Generator Random with seeds support IPO and AETG work in progress Model exporter To CASA (constraints still in progress) Test suite exporter To excel /CSV file New planned features New planned features Language Language, e.g. output for oracles, arrays, … Meta-algorithms Meta-algorithms (like: run 50 times and take the best) postprocessor, Test suite postprocessor, e.g. remove unnecessary test cases Angelo Gargantini - CitLab: a Laboratory for Combinatorial

Objectives and conclusions CitLab Language & editor Establish a CIT “language” with a precise semantics New concepts and constructs? Exchange models and benchmarks CIT “knowledge base” A user friendly textual editor integrated with the eclipse IDE CitLab plugin-based environment defining new generators to foster contributions from new parties making comparison easier and more reliable Angelo Gargantini - CitLab: a Laboratory for Combinatorial