Feedback-Directed Random Test Generation Automatic Testing & Validation CSI5118 By Wan Bo

Introduction

Random Testing VS System Testing – No certain conclusion which is better – Advantages of random testing for OO – Normal random test generation is unguided.

Purpose of this work: – random generation of unit tests for object- oriented programs. – a sequence of method calls for unit test. – random selecting or previous computed object as parameters for each method call.

Benefits of guided generation: – only sensible to build a legal sequence of method calls – Speed up generate process and reduce redundant sequences. – Example: Double a = sqrt(-1); Double b = log(a); With feedback, such call sequence will not be extended.

RANDOOP- random tester for object-oriented programs. – A software developed by this group – It can generate random test cases (more details later) – Example:

Methodology

Method Sequence – a method sequence, or simple sequence, is a sequence of method calls – Each call named S.i which means the value returned by the ith previous method call in sequence S

Extending Sequence – extension operation: takes zero or more sequences and produces a new sequence. – operator extend(m, seqs, vals) m: is a method with formal parameters of type T1; : : : ; Tk. seqs: is a list of sequences. vals: is a primitive values v1 : T1; : : : ; vk : Tk, or it is the return value s:i of the i-th method call for a sequence s appearing in seqs.

Example:

Feedback-directed Generation – build sequences incrementally – as soon as a sequence is built, execute to ensure non-redundant and legal objects (filters & contracts) – GenerateSequences(classes, contracts, filters, timeLimit) classes: a list of classes for which to create sequences contracts: a list of contracts filters: a list of filters timeLimit: a time limit to stop generation process

Generate Sequence – Step1 Create new Sequence – Step2 Discard duplication – Step3 Execute new sequence – Step4 Check contracts – Step 5 added in test sequence set.

Boolean vector – a boolean flag s:i.extensible. (whither it can be used as input in new method call) – Be used in filters (explain later) m(T1...Tk) at line 5 – Use reflection to load dll (being tested), store all method infos and classes infos into collection. Select randomly. – RandomMethodOrConstructorUniform(),return a reflecton.runtimemethodinfo

randomSeqsAndVals(T1...Tk) – Types is from methodinfo – The length of sequence is decided by limited time – When a new method call created, put in plan, when time out, write in cs file. Parameters – Primitive type, select a primitive value form fixed pool of values – Reference type Use one already exist in sequence (in plan). New one Null

Primitive Value Pool:

Extend Sequence The method is explained in previous slide Discard duplicates – Check whither equivalent sequence was already created in previous – Non error sequence set – Error sequence set Execute sequence – Execute calls in sequence one by one. And check contracts after each all. – If conflict with contract, set violated be true

Default Contracts

Filtering – setExtensibleFlags (newSeq; flters; ~o) line19 Function: determine which values of a sequence are extensible and should be used as inputs to new method calls – Equality. uses the equals() method to determine if the resulting object has been created before. – Null. set newSeq:i.extensible to false iff the corresponding object is null. Such as void method

– Exceptions. If exception occurs, even it does not conflict with contract, still set extendable false

Evaluation

Block Coverage

Block Coverage Running Time

Test common libs – REDUCE similar with Junit and Nunit – Used feedback-RANDOOP

Open Source: For Java: – For.Net: –

RUNDOOP2 for C# published 09/08/2010 Main structure:

Randoop – Load target dll by reflection, analysis and generate config file (xml) RandoopBare – According to config file, generate test cases. – Staticmanager calculates figures. – Generate log

Example 1 – Test target.Net System.Data.dll – Right now just can test dlls in C:\WINDOWS\Microsoft.NET\Framework\v – Randoop needs use csc.exe to compile test cases

Test Case:

My test cases: Lib- RandoopTestLib – Addition – Subtract – Arithmetic

Subtract

Put dll into C:\WINDOWS\Microsoft.NET\Framework\v For Int: For Long: For String and Object I can not find it.

The test case