(or what I learned from Neil about software testing) John Hughes Chalmers University/Quviq AB

Company founded May 2006 – Tools for testing software – Profitable in the first year! Customers include… Soon TBA

QuickCheck Tests software automatically against a formal specification prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)). prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)).

Testing… 19> eqc:quickcheck(ex:prop_reverse()) Failed! After 10 tests. {[2],[3,2]} Shrinking....(4 times) {[0],[1]} false 19> eqc:quickcheck(ex:prop_reverse()) Failed! After 10 tests. {[2],[3,2]} Shrinking....(4 times) {[0],[1]} false

QuickCheck Tests software automatically against a formal specification prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)). prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)).

QuickCheck in Brief Specification Test case Test outcome GenerateExecute Simplify Minimal counter- example e.g. {[0],[1]}

A More Realistic Test Case A test case for the Erlang process registry test_register() -> Pid = spawn(), register(name,Pid), Pid2 = whereis(name), assert(Pid==Pid2), unregister(name). test_register() -> Pid = spawn(), register(name,Pid), Pid2 = whereis(name), assert(Pid==Pid2), unregister(name). Spawn a new process (set up test data) Register it—a side- effect Inspect the results Did the test pass? Restore the state, ready for next test

Industrial Test Cases A sequence of – Calls to an API under test – Checks on the results Not much like {[0],[1]}… How can we convert this kind of test case into a logical property?

QuickCheck in Industry Specification Test case: a program Test case: a program Test outcome GenerateExecute Simplify Minimal program provoking a fault

Programs as Data Objects Why? – So failing test cases can be printed – So failing test cases can be repeated – So failing test cases can be simplified

Can we mix generation and execution? Randomly generate and perform each call – Save a test case as a list of functions and actual parameters? – NO!!! When we repeat the test, Pid has a different value – We must use a symbolic variable! test_register() -> Pid = spawn(), register(name,Pid),… test_register() -> Pid = spawn(), register(name,Pid),…

Test Case Language Do we need conditionals? Tests should be deterministic – Check the right branch, don’t test Users need simplicity—straight-line code is simple, and suffices X = foo(…), case p(X) of true -> baz(X); false -> bar(X) end X = foo(…), case p(X) of true -> baz(X); false -> bar(X) end X = foo(…), assert not(p(X)), bar(X) X = foo(…), assert not(p(X)), bar(X)

Generating Test Cases How do we know – Which commands are valid at each point? – What test data is available at each point? – What results are expected? Track a test case state – Check preconditions before generating each command – Store available data in the state – Check postconditions wrt test case state

Process Registry State type state() = record pids::list(pid()), regs::list({atom(),pid()}) end. type state() = record pids::list(pid()), regs::list({atom(),pid()}) end. Available process ids Currently registered processes

Process Registry State Two-level programs! – Static = test case generation time – Dynamic = test execution time type state() = record pids::list(symbolic(pid())), regs::list({atom(),symbolic(pid())}) end. type state() = record pids::list(symbolic(pid())), regs::list({atom(),symbolic(pid())}) end. Available process ids Currently registered processes

Two-level State Machines Preconditions – Checked during generation, purely static Postconditions – Checked during execution, purely dynamic Next state function – Used at both times, two-level Command generator – Used during generation, two-level

Process Registry spawn() – Adds a (dynamic) pid to the state unregister(Name) – Pre: Name must be registered – Removes Name from the state register(Name,Pid) – Adds {Name,Pid} to the state – Post: exception if Name or Pid already registered

A Failing Test Case [{set,{var,2},{call,…,spawn,[]}}, {set,{var,3},{call,…,register,[a,{var,2}]}}, {set,{var,6},{call,…,register,[b,{var,2}]}}, {set,{var,8},{call,…,spawn,[]}}, {set,{var,9},{call,…,register,[b,{var,8}]}}] [{set,{var,2},{call,…,spawn,[]}}, {set,{var,3},{call,…,register,[a,{var,2}]}}, {set,{var,6},{call,…,register,[b,{var,2}]}}, {set,{var,8},{call,…,spawn,[]}}, {set,{var,9},{call,…,register,[b,{var,8}]}}] V2=spawn(), register(a,V2), register(b,V2), V8=spawn(), register(b,V8) V2=spawn(), register(a,V2), register(b,V2), V8=spawn(), register(b,V8) CodeSpec Inconsistency!

A Buggy Spec spawn() – Adds a (dynamic) pid to the state unregister(Name) – Pre: must be registered – Removes Name from the state register(Name,Pid) – Adds {Name,Pid} to the state – Post: exception if Name or Pid already registered …provided there is no exception!

”The Trick” What if we know part of the structure? – Check the known part in the postcondition – Add the known part statically to the state ”The Trick” is as useful as ever! [{set,{var,2},{call,…,spawn,[]}}, …] [{set,{var,2},{call,…,spawn,[]}}, …] Added to the state Purely dynamic

Test Case Generator Two-level languages are helping to find thorny bugs in industrial systems! Randomized Generating Extension =

Best Bug! In Ericsson’s Media Proxy (Multimedia IP- telephony product, SIP) A serious bug Obtained from a simple specification, by simplifying 160-command sequence! Add Sub Add Sub Add Sub