Methodological Issues in Model-Based Testing (MBT)

Related definitions What is MBT? What about handling the real-world complexity in models? What about generating the production code from models?

Definitions Test Case – structure of input and expected ouput behaviours Deterministic systems – finite sequence Non-deterministic systems – tree-like structure Deterministic Output and next state are unambigously determined by the current state and input Non-deterministic Reaction for input is non-deterministic, ie More than one outgoing transition may be available for some states Test Suite – set of test cases Test Purpose – a property that has to be tested. May be expressed formally (statement, branch, path, … coverage) or informally (reference to system requirement or sub-requirement)

Model-based testing - components Results Model Test Generator Tool Test Suite Test Execution Tool/ Human Tester Drivers/ Adapters SUT <<Implements>> Does the implementation conform to specification model?

Model-based testing Based on the specification model of SUT’s behaviour Therefore model-based testing is Black-Box testing Specification model Allows to automatically generate tests Can be used as the oracle that checks whether the SUT behaves correctly Theoretically, specification models allow us to generate production code also! (model-based development)

Types of testing/test generation Offline / a priori Test suite is generated before executing it (offline) Online / on-the-fly Each test is generated after executing the previous test during the testing

Why bother?! Modeling-the-real-world issue Model has to be valid. Why build a model, validate it, derive tests, run tests rather than directly validating the SUT? Model-Based Development issue Can we use the same model for generating tests and generating production code? If we can generate production code from the valid model does it eliminates the need for testing because generated code is valid itself?

Modeling-the-real-world issue

Modeling the real “world” Model is a simplification of the real world. Models do not reflect all attributes of the real world Two basic approaches for simplification Omission of details Encapsulation of details We can’t understand and describe the whole “world” at once We must break the “world” into reasonable pieces

Omission of details Abstract models are easier to understand Stepwise refinement is used High-level models include fundamental ideas High-level models are too abstract for testing Abstract models are detailed, if necessary, and with relevant details for some aspect of the SUT only Detailed models with respect of requirements to be validated, limited set of requirements at a time Drivers/adapters insert the implemention-specific information so that tests can be executed against the real SUT

Drivers/Adapters Used for interaction between abstract test cases and the real-world SUT via Concretization Abstraction Abstract TCs Test Execution Tool/ Human Tester Drivers/ Adapters SUT Concretization Concrete results Abstraction

Drivers/Adapters Test Cases generated from abstract model are too abstract to be fed to the real SUT Concretization is needed to be able to pass information to the real SUT Actual results from the real SUT are too detailed to be compared to the expected results from the model Abstraction is needed to be able to validate real-world concrete results against the abstract model

Encapsulation of details In other words – using references to detailed models of certain aspects, for example ISO/OSI model, where each layer relies on lower layers Libraries in programming languages “Parts” of the real world are put togehter when executing tests

Model-Based Development issue

Models in SW development Testing is not the only discipline using models The main question – can we use the same model for test and production code generation? Let’s investigate two alternatives Common model Separate models

The common model approach Test Generator Tool Suite Requirements Code With common model - the code is tested against itself since the source (model) is the same! Only Code generator and code-environment interaction can be tested with this Test Suite!

The separate models approach Requirements Pricipial problem of the common model approach is resolved by different sources Development Model Test Model Development of models and manual development must be carefully synchronized! This approach is expensive Code Test Suite Code Generator Tool Test Generator Tool

Summary Modeling itself helps to reveal defects in logic In MBT, Models are executable artifacts written in a very high-level programming language Test code can be generated Production code can be generated Models need to be abstract enough to be maintainable – can’t model the whole world at once Omission of details Encapsulation of details Adapters/drivers are responsible for adding/removing Implemetation-specific details for interaction with real-world SUT Great interest on MBT in embedded systems industry

Sources and further reading Methodological issues in Model-Based testing Alexander Pretschner Jan Phillips MBT Example - Synthesis of Test Purpose Directed Reactive Planning Tester for Non-deterministic Systems Jüri Vain, Kullo Raiend, Andres Kull, Juhan P. Ernits