Formal Conformance Testing of Systems with Refused Inputs and Forbidden Actions Igor Burdonov, Alexander Kossatchev, Victor Kuliamin ISP RAS, Moscow

Outline Introduction Formal Testing with LTSes and ioco Proposed conformance relation LTS Completion Conclusion

Formal Conformance Testing Requirements System under Test Specification Test Suite Implementation Model Test Suite Real WorldModel World conforms???conforms formally passes formallypasses derivation modeling translation  !!!

Labeled Transition Systems Model world – LTSes States Transitions Labels  Inputs ?a, ?b  Outputs !x, !y  Internal transitions τ Initial state ?a?a !x ?b?b !y τ

ioco Conformance Relation Testing abilities  Ability to provide inputs  Ability to observe outputs  Ability to observe quiescence I ioco S  for each δ -trace (trace with quiescence) σ of S  an output is possible in I after σ only if its is possible in S  quiescence is possible in I after σ only if it is possible in S ?a?a !x ?b?b δ

Examples ?a?a !x ?b?b δ !y τ S δ δ ?a?a !x ?b?b δ ?b ?a I1I1 δ !y ?a δ ?a?a !x ?b?b δ ?b ?a I2I2 δ !y ?a ?a?a ?b ?b?b δ ?a I3I3 !y ?a δ I 1 ioco SI 2 ioco SI 3 ioco S τ

? More Subtle Abilities Testing abilities  Ability to provide inputs  Ability to observe outputs  Ability to observe quiescence Constraints  Implementation should be input-enabled  Tester has full control over action firing of implementation - synchronous testing ( parallel composition semantics ) ?a?a !x ?b?b ?b ?a !y I ?b ?a?a τ !b T

Practical Considerations Some implementations are not input-enabled  Launch button  Memory allocation Testing in almost cases is asynchronous  Especially for distributed systems free(void *ptr) “ptr should be earlier returned by malloc(), calloc(), realloc()” (POSIX)

Environment Asynchronous Testing System under Test Tester Real WorldModel World Model Tester Implementation Context T I C I || C

Compositional Conformance We may try to check that ( I || C ) ioco ( S || C ) But ioco interacts with || in a bad way!  Unexpected conformance ( I || C ) ioco ( S || C ) while I ioco S  Unexpected non-conformance ( I || C ) ioco ( S || C ) while I ioco S

Bad Examples ?a?a !y ?b?b S !x?a?a I1I1 C – input and output queues I 1 ioco S ( I 1 || C ) ioco ( S || C ) ?a?a !y I2I2 !x ?b?b ?b?b!z I 2 ioco S ( I 2 || C ) ioco ( S || C )

Possible Ways out  ioco is asymmetric – I should be input-enabled, while S should not  ioco is preserved when S is input-enabled Consider input-enabled specs only Too narrow Replace S with S ’ such that  I I ioco S  I ioco S ’ ( S ~ ioco S ’) and S ’ is input-enabled Consider not input-enabled implementations and ioco analogue for them Consider context-specific conformance relations Queues already considered, but others are also needed

Outline Introduction Proposed conformance relation Extension of ioco for non-input-enabled implementations LTS Completion Conclusion

Meaning of Undefined Inputs Forbidden Should not be provided Refused Can be provided and its refusal can be observed Ignored Can be provided, does nothing  should be specified

Proposed Model LTS with forbidden actions and refused inputs Additional elements  Forbidden action γ  Refused inputs {?a}, {?b} βγδ -traces Contain inputs, outputs, δ, input refusals, γ γ can only be the last symbol ?a?a !y ?b?b !x τ {?a,?b}, δ {?a} {?b} δ ?b?b ?a?a ?a,?b ?b?b γ ?a?a

Safe Traces βγδ -traces that cannot cause forbidden action to occur ?b?b ?b ?a?a !x τ ?a?a γ ?a?a τ !y τ γ ?b?b Safe βγδ -traces: ?b !x ?a δ ?b{?b}?a !x?a δ {?a} δ {?b} δ, {?a} !x

Safety of Testing Test should not destroy implementation Safety hypothesis (weakening of input-enabledness hypothesis) Each safe βγδ -trace of S, which is also βγδ -trace of I, can be safely extended in I by each symbol safe after this trace in S Such I is called safe for S

ioco βγδ Conformance Relation Testing abilities  Ability to provide inputs  Ability to observe outputs  Ability to observe quiescence  Ability to observe input refusal Constraints  Implementation is safe for specification  Tester has full control over action firing of implementation (synchronous testing)

ioco βγδ Conformance Relation I ioco βγδ S  I is safe for S and for each safe βγδ -trace σ of S  an output safe in S after σ is possible in I after σ only if its is possible in S  quiescence safe in S after σ is possible in I after σ only if it is possible in S  an input safe in S after σ is possible in Ι after σ only if it is possible in S  an input refusal safe in S after σ is possible in I after σ only if it is possible in S

Examples ?a?a,?b γ SI2I2 I1I1 I 1 ioco βγδ SI 3 ioco βγδ S I 2 is not safe for S !y !x ?a?a !y ?b?b !x τ {?a,?b}, δ {?a} {?b} δ ?b?b γ ?a?a ?b?b ?a?a ?b?b γ !y !x ?b?b ?b {?b} {?a} !y δ {?b} {?a,?b}, δ δ ?a?a γ I3I3 ?a?a ?b?b!y ?b?b ?b {?b} {?a} !y δ I 4 ioco βγδ S I4I4 δ I5I5 I 5 ioco βγδ S !x?b I6I6 I 6 ioco βγδ S {?b} {?a}

Test Case Derivation Very similar to ioco Differences  We should escape forbidden action – Use safe traces only  We should test input refusals – Try to provide an input and observe refusal as a deadlock

Examples S ?a?a !y ?b?b !x τ {?a} {?b} δ ?b?b γ ?a?a {?a,?b}, δ T1T1 θ !a ?y?y ?x, θ !b!b ?x,?y !b θ !a θ θ θ θ ?x,?y θ θ !a !b!b T2T2 !x ?y, θ ?x,?y !b θ θ θ ?x,?y θ θ !a !b!b

Outline Introduction Proposed conformance relation LTS Completion  Trying to force composition to preserve conformance Conclusion

Completion Operations ioco is preserved when S is input-enabled Replace S with S ’ such that  I I ioco S  I ioco S ’ ( S ~ ioco S ’) and S ’ is input-enabled Then, I ioco S  I ioco S ’   C ( I || C ) ioco ( S ’ || C ) S ’ is more correct form of S – it can be used in any context

Demonic completion Ξ Tretmans et al. LTS May add more conforming implementations Undefined inputs τ All inputs All outputs?a?a !x !y τ SI I ioco S ?a?a ?a !y ?a Ξ(S) ?a?a?a?a τ !x ?a?a !y I ioco Ξ(S)

Basic Completion States Bc( S ) = δ -traces of S For each δ -trace σ of S add the following transitions R( σ ) means all δ -traces obtained from σ by deleting some δ -s  Add σ  σ marked with ?a if  μ  R( σ ) μ – δ -trace of S  Add σ  σ marked with !x if  μ  R( σ ) μ – δ -trace of S  Add internal σ  σ if  μ  R( σ ) μ – δ -trace of S and σ does not end with δ  Add σ  σ marked with !error if  μ  R( σ ) μ cannot be extended with any output, nor with δ ?a?a !x !y τ ?a?a

Proposed Completions Bc(LTS) Undefined inputs τ All inputs All outputs Bc(LTS) Undefined inputs γ ?a?a !x !y τ SΔ(S) ?a?a?a?a τ !x ?a?a !y ?a?a γ Γ(S) Δ Γ For each I and S in ioco domain I ioco S  I ioco βγδ Δ(S)  I ioco βγδ Γ(S)

Outline Introduction Proposed conformance relation LTS Completion Conclusion

Summary ioco ioco βγδ Extension for non-input-enabled implementations Completion – construction of equivalent input-enabled spec Δ, Γ ?

Announcement Notation  U – non-empty subsets of reachable states of S  A  U and z is safe in A  A z – states reachable from A by z and τ  A  U and z is unsafe in A  A z = γ  A  U and z is refusal set  A z – maximal stable subset of A that for each refusal in z it exists in each element of the subset States of F( S ) – U, U ×{outputs, δ }, γ Transitions  γ  γ marked with γ  If for symbol z A z is nonempty  A  A z and ( A, δ)  A z marked with z  If !y is safe and exists in A z where z is refusal set  internal A  (A z,!y) and (A z,!y)  (A z ) !y marked with !y  If each !y is unsafe or not exists in nonempty A z where z is refusal set  internal A  (A z, δ )  If for ?a A ?a is nonempty  ( A, !y )  A ?a marked with ?a

Practical Implications Not any specs are written correctly for use in component-based systems The transformation rules can serve for specification adjustment The rules can be rephrased into characteristics of correctly written specs

Contacts Igor Burdonov Alexander Kossatchev Victor Kuliamin

Thank you!