Simulated Time for Host-Based Testing with TTCN-3 Stefan Blom (Innsbruck Univ.), Thomas Deiss (Nokia Research Center, NRC), Natalia Ioustinova (CWI), Ari Kontio (NRC), Jaco van de Pol (CWI), Axel Rennoch (FOKUS), Natalia Sidorova (TU/e)

Roadmap Motivation for host-based testing General structure of a TTCN-3 test system Simulated time for host-based testing Solution for host-based testing in simulated time with TTCN-3 Applications (transportation, telecom) Conclusion

Systems around us... Embedded Systems Distributed Systems

Many components/processes/services working in parallel Asynchronous communication Complex behavior driven by data and time limitations Open: in continuous interaction with the outside world Scalability etc. are complex and often unpredictable V&V Verification: are we building the product right? Validation: are we building the right product? (testing) (peer reviews)

Why host-based testing? Target environment: the environment where software is supposed to work Host environment: the environment used for developing the software Many of components do not exist until the latest phase of development => they are not always available for integration testing => testing in the target environment can be expensive and even dangerous host-based testing Find and fix as many errors as possible prior to testing in target environment => host-based testing

Host-based testing Host environment makes use of  simulated environments (1) (represent missing software/hardware components)  emulation of target OS services (2)  monitoring (3)  instrumentation (4) Time constraints are imposed on the system and its environment

Host-based testing Test results are adequate iff the effect of 1-3 on timed behavior of the system is negligible Simulators/emulators/monitoring/instrumentation with negligible effect are often expensive and system-specific In practice: probe effects are often “not negligible”  performance constraints (time limits on system) [target]  behavioral constraints (time limits on environment) [host] Here we focus on host-based testing of behavioral time-dependant features

TTCN-3 Test System

Timers in TTCN-3 Timer states: deactivated or active Active timer keeps information on time left until its expiration Timer operations: set, reset, timeout Time progression decreases time left until expiration of active timers When time left until expiration becomes 0, the timer expires Expiration results in producing a timeout (message) Timeout is enqueued at the component using the timer

Simulated Time Time is modeled by a discrete logical clock We work with systems where external delays are significantly larger compared to duration of computations within the system To obtain test results independent of probe effects we assume  instrumentation/monitoring are done at zero time  message passing, scheduling/synchronization and other emulated OS services are provided at zero time Actions are instantaneous => time progression has the least priority Time progresses by action tick decreasing the time left until expiration of active timers by one Periods between two ticks are time slice

Idleness We focus on closed systems (SUT+Test System) consisting of multiple components communicating with each other Local idleness: a component is idle iff it cannot proceed by performing computations, receiving messages or processing timeouts Global idleness: a closed system is idle iff all components are idle and there are no messages or timeouts pending

Challenge Time semantics in TTCN-3 has been intentionally left open to enable use of TTCN-3 with different time semantics Focus has been on using TTCN-3 for real time testing => existing TCI and TRI interfaces provide excellent support for real time but lack operations necessary for implementing simulated time. In simulated time all entities of TTCN-3 test system and SUT should agree on simulated time Time may progress only if the system is idle We need a mechanism that  detects global idleness of the closed system  progresses time if idleness is detected

Idleness of Closed System All components are idle No messages pending

Global Idleness of TTCN-3 test system All components are idle No messages pending from TS to SUT from SUT to TS from TSi to TSj from TEi to PAi & SAi from PAi & SAi to TEi

Distributed Termination Detection We extend DTD algorithm of Dijkstra-Safra to detect idleness and to progress time

Time Manager Initiates global idleness detection by sending a token consisting of a global message counter and a global flag along the ring. Initially, global message counter is 0 and the global flag has value IDLE_TAG. If time manager receives the token back with the global message counter equal to 0 and the global flag equal to IDLE_TAG, it detects global idleness. Otherwise, time manager repeats idleness detection. If global idleness is detected, time manager reactivates components in the new time slice, then progresses time by sending the token with flag TICK_TAG and restarts global idleness detection in the next time slice

idle flag Time Manager 0 idle Initially msg 0 idle sends a message to 4 4 and 1 update their message counters

idle flag 00 Time Manager 1 and 2 become idle 0 idle idle 1 updates the token 1 running 1 sends the token to the next running 1 1 updates local flag flag idle

flag 00 Time Manager idle 2 becomes idle and updates token running 1 flag 2 sends token to the next and updates local flag idle running 1 flag

idle flag 00 Time Manager idle 3 becomes idle and updates token flag 3 sends token to the next and updates local flag idle running 1 flag idle running 1 flag

idle flag 00 Time Manager idle flag 4 sends token to the next and updates local flag idle running 0 flag 4 updates token idle flag idle running 1 flag 0 Maybe one of the components is active, repeat detection 4 becomes idle flag

idle flag 00 Time Manager idle flag idle 0 flag idle flag idle flag The system is idle, progress time! idle flag idle

flag 00 Time Manager idle flag idle flag idle flag idle flag Reactivate components in the new time slice idle flag restart flag restart flag restart flag 0 restartflag 0 restart

idle flag 00 Time Manager idle flag idle flag idle flag idle flag Progress time idle flag tick flag tick flag tick flag 0 tickflag 0 tick Start new idleness detection round

TTCN-3 Test System

Simulated Time Interface Idle PAIdle(int TRISentPA, int TRIEnqdPA) SAIdle(int TRISentSA, int TRIEnqdSA, int SASentSUT, int SUTEnqdSA) TEIdle(int TRISentTE, int TRIEnqdTE, int TCISentTE, int TCIEnqdTE) Active PAActive() SAActive() TEActive()

Idleness Handler for TSi Counts number of messages exchanged by the TSi via TCI Counts number of messages exchanged by the TSi via TRI Counts number of messages exchanged by the TSi (actually SAi) with SUT Keeps idle variables for TEi, SAi and PAi Keeps flag variables for TEi and SAi If idleness token present, SAi/PAi/TEi are idle and there are no messages pending at TRI, then the idleness handler of a TSi detects local idleness of the TSi updates the idleness token and sends it to the next handler

Testing Railway Interlockings (FATES’05) Provide a framework for testing railway interlockings with TTCN-3 Railway Control System Apply the framework for testing interlocking software for Hoorn- Kersenboogerd station

Testing layer 3 of a 2G/3G

Conclusion We proposed host based testing with simulated time in TTCN-3 Possible to implement on the level of TTCN-3 code (FATES’05) Pro: does not require new interfaces Con: test suite is suitable for testing with simulated time only Proposed a solution on the level of adapters for distributed host-based testing Con: require new interface (submitted as change request to ETSI by Nokia) Pro: test suites are suitable for testing both in real and in simulated time