1. Feedback-Directed Random Test Generation Automatic Testing & Validation CSI5118 By Wan Bo

2. Introduction

3. Random Testing VS System Testing – No certain conclusion which is better – Advantages of random testing for OO – Normal random test generation is unguided.

4. Purpose of this work: – random generation of unit tests for object- oriented programs. – a sequence of method calls for unit test. – random selecting or previous computed object as parameters for each method call.

5. Benefits of guided generation: – only sensible to build a legal sequence of method calls – Speed up generate process and reduce redundant sequences. – Example: Double a = sqrt(-1); Double b = log(a); With feedback, such call sequence will not be extended.

6. RANDOOP- random tester for object-oriented programs. – A software developed by this group – It can generate random test cases (more details later) – Example:

7. Methodology

8. Method Sequence – a method sequence, or simple sequence, is a sequence of method calls – Each call named S.i which means the value returned by the ith previous method call in sequence S

9. Extending Sequence – extension operation: takes zero or more sequences and produces a new sequence. – operator extend(m, seqs, vals) m: is a method with formal parameters of type T1; : : : ; Tk. seqs: is a list of sequences. vals: is a primitive values v1 : T1; : : : ; vk : Tk, or it is the return value s:i of the i-th method call for a sequence s appearing in seqs.

10. Example:

11. Feedback-directed Generation – build sequences incrementally – as soon as a sequence is built, execute to ensure non-redundant and legal objects (filters & contracts) – GenerateSequences(classes, contracts, filters, timeLimit) classes: a list of classes for which to create sequences contracts: a list of contracts filters: a list of filters timeLimit: a time limit to stop generation process

13. Generate Sequence – Step1 Create new Sequence – Step2 Discard duplication – Step3 Execute new sequence – Step4 Check contracts – Step 5 added in test sequence set.

14. Boolean vector – a boolean flag s:i.extensible. (whither it can be used as input in new method call) – Be used in filters (explain later) m(T1...Tk) at line 5 – Use reflection to load dll (being tested), store all method infos and classes infos into collection. Select randomly. – RandomMethodOrConstructorUniform(),return a reflecton.runtimemethodinfo

15. randomSeqsAndVals(T1...Tk) – Types is from methodinfo – The length of sequence is decided by limited time – When a new method call created, put in plan, when time out, write in cs file. Parameters – Primitive type, select a primitive value form fixed pool of values – Reference type Use one already exist in sequence (in plan). New one Null

16. Primitive Value Pool:

17. Extend Sequence The method is explained in previous slide Discard duplicates – Check whither equivalent sequence was already created in previous – Non error sequence set – Error sequence set Execute sequence – Execute calls in sequence one by one. And check contracts after each all. – If conflict with contract, set violated be true

18. Default Contracts

19. Filtering – setExtensibleFlags (newSeq; flters; ~o) line19 Function: determine which values of a sequence are extensible and should be used as inputs to new method calls – Equality. uses the equals() method to determine if the resulting object has been created before. – Null. set newSeq:i.extensible to false iff the corresponding object is null. Such as void method

20. – Exceptions. If exception occurs, even it does not conflict with contract, still set extendable false

21. Evaluation

22. Block Coverage

23. Block Coverage Running Time

24. Test common libs – REDUCE similar with Junit and Nunit – Used feedback-RANDOOP

25. Open Source: For Java: – http://code.google.com/p/randoop/ http://code.google.com/p/randoop/ For.Net: – http://randoop.codeplex.com/ http://randoop.codeplex.com/

26. RUNDOOP2 for C# published 09/08/2010 Main structure:

27. Randoop – Load target dll by reflection, analysis and generate config file (xml) RandoopBare – According to config file, generate test cases. – Staticmanager calculates figures. – Generate log

28. Example 1 – Test target.Net System.Data.dll – Right now just can test dlls in C:\WINDOWS\Microsoft.NET\Framework\v2.0.50 727 – Randoop needs use csc.exe to compile test cases

29. Test Case:

30. My test cases: Lib- RandoopTestLib – Addition – Subtract – Arithmetic

31. Subtract

33. Put dll into C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727 For Int: -10 -3 -2 -1 0 1 2 3 10 For Long: -10 -3 -2 -1 0 1 2 3 10 For String and Object I can not find it.

34. The test case