1. VDMUnit Testing, Combinatorial Testing, Model Quality and Code Generation Professor Peter Gorm Larsen Department of Engineering, Aarhus University (pgl@eng.au.dk)pgl@eng.au.dk and PhD student Peter Jørgensen (pvj@eng.au.dk)pvj@eng.au.dk

2. 2 Agenda  The VDMUnit Testing Classes The Combinatorial Testing Principles Internal Consistency External Consistency Code Generation Project Proposals

3. 3 The VDMUnit Framework

4. 4 The TestSuite Class class TestSuite is subclass of Test instance variables tests : seq of Test := []; operations public Run: () ==> () Run () == (dcl ntr : TestResult := new TestResult(); Run(ntr); ntr.Show()); public Run: TestResult ==> () Run (result) == for test in tests do test.Run(result); public AddTest: Test ==> () AddTest(test) == tests := tests ^ [test]; end TestSuite

5. The TestCase Class public Run: TestResult ==> () Run (ptr) == trap with ptr.AddFailure(self) in (SetUp(); RunTest(); TearDown()); protected SetUp: () ==> () SetUp () == is subclass responsibility; protected RunTest: () ==> () RunTest () == is subclass responsibility; protected TearDown: () ==> () TearDown () == is subclass responsibility end TestCase class TestCase is subclass of Test instance variables name : seq of char operations public TestCase: seq of char ==> TestCase TestCase(nm) == name := nm; public GetName: () ==> seq of char GetName () == return name; protected AssertTrue: bool ==> () AssertTrue (pb) == if not pb then exit ; protected AssertFalse: bool ==> () AssertFalse (pb) == if pb then exit ;

6. 6 The TestResult Class class TestResult instance variables failures : seq of TestCase := [] operations public AddFailure: TestCase ==> () AddFailure (ptst) == failures := failures ^ [ptst]; public Print: seq of char ==> () Print (pstr) == def - = new IO().echo(pstr ^ "\n") in skip; public Show: () ==> () Show () == if failures = [] then Print ("No failures detected") else for failure in failures do Print (failure.GetName() ^ " failed") end TestResult

7. 7 Enigma Unit Tests

8. 8 Agenda The VDMUnit Testing Classes  The Combinatorial Testing Principles Internal Consistency External Consistency Code Generation Project Proposals

9. 9 9 Combinatorial Testing Perspective Regular expression Overview of results Detailed test case and results

10. 10 Combinatorial Testing Overview VDM source Parser AST Pattern tree Fully expanded tests Shape reduced tests Filtered test results T: let x in set {1,..., 100} in obj.op(x){1,3};

11. 11 Test case execution Test cases Test engine Check if filtered by other test cases Failed Inconclusive Passed Filtered Test results Filter out test cases with a prefix resulting in a test failure

12. 12 operations public insert : int ==> () insert(val)== skip pre val > 1; traces T1 : let x in set {1,2} in insert(x);

13. 13 class A values obj : A = new A(); operations public op : nat ==> nat op (x) == return x; traces T2: let x,y in set {1,..., 10} in (obj.op(x);obj.op(y)); end A

14. 14 Regular Expressions in Traces Let definitions (local naming) Let be such that definitions (all selection) Repeat traces (fixed or variable number of times) Concurrency traces Application expression (calling operations)

15. 15 Agenda The VDMUnit Testing Classes The Combinatorial Testing Principles  Internal Consistency External Consistency Code Generation Project Proposals

16. 16 Introduction What is the value of the models you have produced? How do we assess the quality of a model? Internal consistency: Does the model describe something? Syntax, type checking and proof obligations No potential run-time errors External consistency: Does the model describe the right thing? Validation with domain expert Does the model have desirable properties?

17. 17 POP3: Protection of Partial Operators class POP3Server... instance variables maildrop : MailDrop;... types public MailDrop = map POP3Types`UserName to MailBox; public MailBox :: msgs : seq of POP3Message locked : bool; operations GetUserMessages: POP3Types`UserName ==> seq of POP3Message GetUserMessages(user) == return GetUserMail(user).msgs pre UserKnown(user); end POP3Server

18. 18 Booking of Flights: Invariant Preservation class Trip types Flight :: departure : seq of char destination : seq of char instance variables journey: seq of Flight; inv forall i in set {1,...,len journey -1} & journey(i).destination = journey(i+1).departure operations AddFlight: Flight ==> () AddFlight(f) == journey := journey ^ [f] pre journey(len journey).destination = f.departure end Trip journey <> [] =>

19. 19 Robot Routes: Satisfiability 1 class Route instance variables points: set of Point; inv forall p1, p2 in set points & p1.GetCoord() = p2.GetCoord() => p1 = p2 and forall p in set points & p.GetIndex() <> card points => GetNext(p).GetCoord() in set {n.GetCoord() | n in set p.Neighbour()} … end Route A pre/post condition specification is satisfiable, if, for all states and inputs satisfying the precondition, there exists some output and “after” state satisfying the postcondition and any relevant invariants.

20. 20 Robot Routes: Satisfiability 2 class Route functions static public AvoidanceRoutes( obstacles: set of (nat * nat), currentPosition: Point, nextWaypoint: Point) routes: set of Route post forall r in set routes & r.GetFirst().GetCoord() = currentPosition.GetCoord() and r.GetLast().GetCoord() = nextWaypoint.GetCoord() and r.GetCoords() inter obstacles = {}; end Route

21. 21 Robot Routes: Satisfiability 3 For implicit definitions there must exist at least one potential result for each input satisfying the pre-condition Proof Obligation (or integrity constraint): forall obstacles: set of (nat * nat), currentPosition: Point, nextWaypoint: Point & exists routes: set of Route & post-AvoidanceRoutes(obstances,currentPosition, nextWaypoint,routes) Can in principle be proved formally

22. 22 Agenda The VDMUnit Testing Classes The Combinatorial Testing Principles Internal Consistency  External Consistency Code Generation Project Proposals

23. 23 Dialogue with Domain Experts Typically domain experts know little about IT Understanding their intended usage may be a challenge Creating a model will create further questions to experts Model should seldom be shown directly Scenarios to be used for test purposes can typically be discussed A Java-based API can be used to ”demonstrate” ideas to domain experts/end users

24. 24 The External Interfaces of Overture External aid is needed to support communication. Overture supports two different types of external interfacing External Call Interface Only calls from the VDM model to an external interface Remote Control Interface Allows for external calls into a VDM Model

25. 25 External Call Interface Enables call to native Java code from inside a model A type of delegate scheme is created between the model and an external jar In the model an operation is created which contains an "is not yet specified" statement public receivedMessage: int ==> () receivedMessage(vecID) == is not yet specified; When a non-specified statement is encountered, the interpreter will considered it an delegate, an attempt to look it up in Java CLASSPATH.

26. 26 External Call Interface The Java class must have the same name as the module or class. Package separators are denoted with underscores in the model. VDM: class gui_Graphics Java: package gui; class Graphics If a Java class and method is found, a delegate object is created an bound to the delegate function. MATH, IO and VDMUtil standard library functions in Overture are implemented using the same delegate scheme

27. 27 External Call Interface Example : VDM class gui_Graphics public receivedMessage : int ==> () receivedMessage(vecID) == is not yet specified; end gui_Graphics

28. 28 External Call Interface Example : Java package gui; import org.overture.interpreter.runtime.ValueException; import org.overture.interpreter.values.Value; import org.overture.interpreter.values.VoidValue; public class Graphics implements Serializable { public Value receivedMessage(Value vecID) throws ValueException { model.receivedMessage(vecID.intValue(null)); return new VoidValue(); } }

29. 29 External Call Interface Example : Java IDE In order to use VDM Types the Java program must have the Overture java library in its build path. ast.jar, parser.jar and interpreter.jar must be available on the built path. These jar files are provided with Overture \Plugins\ Once the Java program is finished, it must be exported to a Jar file and placed in the lib directory in the Overture/VDM projects directory, in order for Overture to find it.

30. 30 External Call Interface Example

31. 31 Remote Control Interface Allows for a VDM model to be controlled by an external Java programs Essentially a console like access to the Overture interpreter which can be bootstrapped through the Overture Debugger A RemoteControl interface is defined, which must be implemented by the external programs. public interface RemoteControl { public void run(RemoteInterpreter interpreter) throws Exception; }

32. 32 RemoteInterpreter RemoteInterpreter is passed from Overture to the external programs It has four central methods Void Init() Re-initializes the interpreter Void Create(String, String) Takes a variable name as the first argument and expression as the second. String Execute(String) Takes expression as string argument, returns a string value. Value ExecuteValue(String) Takes expression as string argument, returns a generic VDM Value represented in Java

33. 33 Example model to use class A operations public op: int ==> int op(n) == return n + 1 pre n > 0 end A

34. 34 Use of the RemoteInterpreter interpreter.init() interpreter.create(("a", "new A()"); //create a := new A(); Value result = interpreter.valueExecute("a.op(5)"); System.out.println(result.intValue(null)); //prints 6

35. 35 Use of the Remote Control Interface Using the Remote Control Interface to create an interactive GUI for a model Connecting a GUI to the KLV /CSLaM model Create a GUI Implement RemoteControl interface Bind GUI with the RemoteInterpreter through the RemoteControl.

36. 36 Remote Control Interface Example (RemoteControl) import org.overture.interpreter.debug.RemoteControl; import org.overture.interpreter.debug.RemoteInterpreter public class KlvRemote implements RemoteControl{ @Override public void run(RemoteInterpreter intrprtr){ Thread remoteThread = new Thread(new Runnable(){ public void run(){ KLVgui gui = new KLVgui(); KLVgui.vdmklv = new KlvOvertureComm(intrprtr, gui); gui.initialising(); }); remoteThread.setDaemon(true); remoteThread.start(); intrprtr.processRemoteCalls(); }

37. 37 Remote Control Interface Example (KlvOvertureComm) public int getMaxSpeed() { try{ Value g = execute("klv.getMaxSpeed()"); return new Long(g.intValue(null)).intValue(); } catch (Exception err) { printError("API error: " + err.getMessage());} return 0; } private Value execute(String arguments) throws Exception{ System.out.println("Calling Overture with: " + cmd); Value result = interpreter.valueExecute(cmd); return result; }

38. 38 Setting up the Remote Control Interface External Java class must be configured in the Overture Debug Configurations

39. Validating KLV using the API

40. 40 Agenda The VDMUnit Testing Classes The Combinatorial Testing Principles Internal Consistency External Consistency  Code Generation Project Proposals

41. 41 Overture Code Generation A VDM++ to Java Code Generator Available in Overture releases 2.0.6 onwards Early work: VDM++ to C++ Code Generator

42. 42 Code generation challenges (Java) VDM: public op : () ==> nat op () == ( dcl v1 : Vector2D := mk_Vector2D(1,2); dcl v2 : Vector2D := v1; v1.x := 2; return v2.x;) Java: public Number op() { Vector2D v1 = new Vector2D(1L, 2L); Vector2D v2 = v1; v1.x = 2L; return v2.x;} Code generating VDM++ models to Java Preserving the semantics in the generated code Code generation across paradigms How do we generate: {x | x in set S & pred(x)} Or Lambdas, e.g. (lambda x : int & (lambda y : int & x + y)) Java (correct): public Number op() { Vector2D v1 = new Vector2D(1L, 2L); Vector2D v2 = v1.clone(); v1.x = 2L; return v2.x;}

43. 43 Example: Set comprehensions public f : () -> set of nat f () == let a = {x | x in set S & pred(x)} in g(a,a); public op : () ==> set of nat op () == ( dcl setCompResult : set of nat := {}; for all x in set S do if pred(x) then setCompResult := setCompResult union {x}; (dcl a : set of nat := setCompResult; return g(a,a)) );

44. 44 Code Generation Platform

45. 45 Agenda The VDMUnit Testing Classes The Combinatorial Testing Principles Internal Consistency External Consistency Code Generation  Project Proposals

46. 46 Project Proposals R&D projects (5 ECTS) or MSc thesis (30 ECTS) Validating a code generator for embedded systems Code generate for an embedded platform NXT, Raspberry Pi etc. Identify and address issues Validating the code generation platform architecture Adding (partial) support for a new language Identify changes/extensions to the IR Feedback for the platform architecture Code Generating VDM-SL models Converting a VDM-SL model into a VDM++ model Without changing the code generator Implementation of the conversion rules

47. 47 Project Proposals Code Generating Union Types How can union types be represented in (say) Java Comparing different approaches Implementing support for union types Code Generating VDM-RT In VDM-RT objects can be deployed on different CPUs CPUs communicate via buses Objects message exchange results in bus communication Code generation of VDM concurrency concepts Other projects: http://wiki.overturetool.org/index.php/Overture_projects #Additional_Potential_Projects_for_.22Technical_IT.22_st udents_at_Aarhus_University http://wiki.overturetool.org/index.php/Overture_projects #Additional_Potential_Projects_for_.22Technical_IT.22_st udents_at_Aarhus_University

48. 48 Summary What have I presented today? Introduced VDMUnit for testing Introduced combinatorial testing principles Assessing model quality Internal consistency External consistency Code Generation Project Proposals What do you need to do now? Read chapter 13 Assess your own mini-project VDM model’s consistency Carry out either the testing exercise or the exercise with a graphical front-end using java

49. 49 Quote of the day Bertrand Meyer Formal specifications may become for software engineers what, say, differential equations are for engineers of other fields