Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign Work.

Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign http://taoxie.cs.illinois.edu/courses/testing/ Work described in the slides was done in collaboration with the Pex team (Nikolai Tillmann, Peli de Halleux, Pratap Lakshman, et al.) @Microsoft Research, students @Illinois ASE, and other collaborators

© Ammann & Offutt Faults, Errors & Failures □ Fault : A static defect in the software (i.e., defect, bug) □ Infected State: An incorrect internal state that is the manifestation of some fault (often also referred to as error) □ Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior

Mistake, Fault, Error, Failure Programmer makes a mistake. Fault (defect, bug) appears in the program.Fault remains undetected during testing (running test inputs). The program fails (based on test oracles) during execution i.e. it behaves unexpectedly. Error: difference between computed, observed, or measured value or condition and true, specified, or theoretically correct value or condition What does Bug mean in “Bug Report”?

What is fault, error, failure? □ Doubling the balance and then plus 10 int calAmount () { int ret = balance * 3; ret = ret + 10; return ret; } 1-4 void testCalAmount() { Account a = new Account(); Account.setBalance(0); int amount = Account.calAmount(); assertTrue(amount == 10); } Where is test input? Where is test oracle?

What is fault, error, failure? □ Doubling the balance and then plus 10 int calAmount () { int ret = balance * 3; ret = ret + 10; return ret; } 1-5 void testCalAmount() { Account a = new Account(); Account.setBalance(1); int amount = Account.calAmount(); assertTrue(amount == 12); } Where is test input? Where is test oracle?

What is fault, error, failure? □ Should not allow withdrawal when there is a balance of 100 or less boolean doWithdraw(int amount) { if (Balance<100) return false; else return wthDraw(amount); } 1-6 void testWithDraw() { Account a = new Account(); Account.setBalance(100); boolean success = Account.doWithdraw(10); assertTrue(success == false); }

Who should test? □ Developer? Separate “quality assurance” group? □ Programmer? User? Someone with a degree in “testing”? 7

Types of Test Activities □ Testing can be broken up into four general types of activities 1. Test Design 2. Test Automation 3. Test Execution 4. Test Evaluation □ Each type of activity requires different skills, background knowledge, education and training © Ammann & Offutt

Test Design □ This is the most technical job in software testing □ Requires knowledge of : ◊ Discrete math ◊ Programming ◊ Testing □ Requires much of a traditional CS degree □ This is intellectually stimulating, rewarding, and challenging □ Test design is analogous to software architecture on the development side □ Using people who are not qualified to design tests is a sure way to get ineffective tests Design test values to satisfy coverage criteria or other engineering goal © Ammann & Offutt

Test Automation □ This is slightly less technical □ Requires knowledge of programming ◊ Fairly straightforward programming – small pieces and simple algorithms □ Requires very little theory □ Very boring for test designers □ Programming is out of reach for many domain experts □ Who is responsible for determining and embedding the expected outputs ? ◊ Test designers may not always know the expected outputs ◊ Test evaluators need to get involved early to help with this Embed test values into executable scripts © Ammann & Offutt

Test Execution □ This is easy – and trivial if the tests are well automated □ Requires basic computer skills ◊ Interns ◊ Employees with no technical background □ Asking qualified test designers to execute tests is a sure way to convince them to look for a development job □ If, for example, GUI tests are not well automated, this requires a lot of manual labor ◊ Test executors have to be very careful and meticulous with bookkeeping © Ammann & Offutt Run tests on the software and record the results

Test Evaluation □ This is much harder than it may seem □ Requires knowledge of : ◊ Domain ◊ Testing □ Usually requires almost no traditional CS ◊ A background in the domain of the software is essential ◊ An empirical background is very helpful (biology, psychology, …) ◊ A logic background is very helpful (law, philosophy, math, …) □ This is intellectually stimulating, rewarding, and challenging ◊ But not to typical CS majors – they want to solve problems and build things © Ammann & Offutt Evaluate results of testing, report to developers

Types of Test Activities – Summary □ These four general test activities are quite different □ It is a poor use of resources to use people inappropriately © Ammann & Offutt 1.Test designDesign test values to satisfy coverage criteria or other engineering goal Requires technical knowledge of discrete math, programming and testing 2.Test automationEmbed test values into executable scripts Requires knowledge of scripting 3.Test executionRun tests on the software and record the results Requires very little knowledge 4.Test evaluationEvaluate results of testing, report to developers Requires domain knowledge But most test organizations use the same people for ALL FOUR activities !!

Testing Model – Black Box Testing □ You know the functionality ◊ Given that you know what it is supposed to do, you design tests that make it do what you think that it should do ◊ From the outside, you are testing its functionality against the specs ◊ For software, this is testing the interface ○ What is input to the system? ○ What you can do from the outside to change the system? (controllability) ○ What is output from the system? (observability) ◊ Impossible to thoroughly exercise all inputs ○ Exhaustive testing grows without bound ◊ Tests the functionality of the system by observing its external behavior ◊ No knowledge of how it goes about meeting the goals ©L. Williams

Testing Model – White Box Testing □ You know the code ◊ Given knowledge of the internal workings, you thoroughly test what is happening on the inside ◊ Close examination of procedural level of detail ◊ Logical paths through code are tested ○ Conditionals ○ Loops ○ Branches ◊ Status is examined in terms of expected values ◊ Impossible to thoroughly exercise all paths ○ Exhaustive testing grows without bound ◊ Can be practical if a limited number of “important” paths are evaluated ◊ Can be practical to examine and test important data structures ©L. Williams

Group Exercise  A program needs to be developed so that given an integer value  it outputs 0 when the integer value is 0  it outputs 1 when the integer value > 0  It outputs -1 when the integer value < 0  What would be your black box tests?  How would you generate your white box tests?  Would black box tests alone be good enough to find bugs/faults in the program? Why?  Would white box tests alone be good enough be find bugs/faults in the program? Why?

Black-box vs. White-box □ White-box - look at code to write test ◊ Tests are based on code ◊ Better for finding crashes, out of bounds failures, file not closed failures ◊ Better at finding faults of extra logic □ Black-box - don’t look at code to write test ◊ Tests are based on specifications ◊ Better at telling if program meets spec ◊ Better at finding faults of omission 17

Types of Testing  Unit Testing (white) ◊ testing of individual hardware or software units or groups of related units ◊ Done by programmer(s) ◊ Generally all white box ◊ Automation desirable for repeatability  Integration Testing (black and white) ◊ testing in which software components, hardware components, or both are combined and tested to evaluate the interaction between them ◊ Done by programmer as they integrate their code into code base ◊ Generally white box, maybe some black box ◊ Automation desirable for repeatability ©L. Williams

Types of Testing -II  Functional/System Testing (black) ◊ testing conducted on a complete, integrated system to evaluate the system compliance with its specified requirements ◊ stress testing, performance testing, usability testing ◊ it is recommended that this be done by external test group ◊ mostly black box so that testing is not ‘corrupted’ by too much knowledge ◊ test automation desirable  Acceptance Testing (black) ◊ formal testing conducted to determine whether or not a system satisfies its acceptance criteria (the criteria the system must satisfy to be accepted by a customer) and to enable the customer to determine whether or not to accept the system ◊ Generally done by customer/customer representative in their environment through the GUI... Definitely black box ©L. Williams

Types of Testing -III  Regression Testing (black and white) ◊ Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements ◊ Smoke test group of test cases that establish that the system is stable and all major functionality is present and works under “normal” conditions  Beta Testing (black) ◊ (1~many) potential users or beta testers install software and use it as they wish and report any revealed errors to the development organization. □ A/B Testing ◊ https://en.wikipedia.org/wiki/A/B_testing https://en.wikipedia.org/wiki/A/B_testing ©L. Williams

Techniques for writing tests □ Black-box (from specifications) ◊ Equivalence partitioning ◊ Boundary value analysis □ White-box (from code) ◊ Branch coverage □ Fault-based testing (from common errors) ◊ http://www.exampler.com/testing- com/writings.html from Brian Marick http://www.exampler.com/testing- com/writings.html 21

Planning a Black Box Test Case ©L. Williams

Important Consideration for Black Box Test Planning □ Look at requirements/problem statement to generate. □ Test cases need to be traceable to a requirement. □ You must write the repeatable test case so anyone on the team can run the exact test case and get the exact same result/sequence of events. ◊ The inputs must be very specific. ○ Example: “Students who receive a grade of 70 or higher pass the exam.” ○ Correct test cases: Grade = 80; Grade =20 ○ Incorrect test cases: “input a passing grade” “input a failing grade” ◊ The expected results must be very specific. “Pass” “Fail” ©L. Williams

Equivalence Class Partitioning □ Divide your input conditions into groups (classes). ◊ Input in the same class should behave similarly in the program. □ Be sure to test a mid-range value from each class. □ Example: for tests of “Go to Jail” the most important thing is whether the player has enough money to pay the $50 fine ◊ Test input values clearly in the two partitions: 25 and 75. ©L. Williams

Equivalence partitioning □ Example: sorting □ sort(array,len) takes an array of integers of length len and sorts them in ascending order, i.e. permutes them so that each element of the array is less than or equal to the succeeding one. □ len = 0,1, 2, 17 □ Array is already sorted, has duplicates, has negative numbers 25

Equivalence class test ideas □ Any object: the null pointer □ Strings: the empty string □ Collections: ◊ The empty collection ◊ Contains exactly one element ◊ Contains the maximum number of elements (or at least more than one) 26

Equivalence class test ideas □ Linked structures (trees, queues, etc.) ◊ Empty ◊ Minimal but non-empty ◊ Circular ◊ Depth greater than one (or maximally deep) □ Equality comparison of objects ◊ Equal but not identical ◊ Different at lowest level, the same at upper 27

Boundary Value Analysis □ Focus on boundaries... because a greater number of faults tend to occur at the boundaries of the input domain ◊ Range input, a to b, test with a, b, a-1, a+1, b-1, b+1 if integer range; otherwise, slightly less than a and slightly more than b. ◊ If you can only have a certain quantity (q) of something, try to create q-1, q, q+1 ©L. Williams

Decision Table Testing ©L. Williams

Monopoly Decision Table □ If a Player (A) lands on property owned by another player (B), A must pay rent to B. If A does not have enough money to pay B, A is out of the game. ©L. Williams

Dirty/Failure Test Cases □ Can something cause division by zero? □ What if the input type is wrong (You’re expecting an integer, they input a float. You’re expecting a character, you get an integer.)? □ What if the customer takes an illogical path through your functionality? □ What if mandatory fields are not entered? □ What if the program is aborted abruptly or input or output devices are unplugged? ©L. Williams Robustness Testing

Techniques for writing tests □ Black-box (from specifications) ◊ Equivalence partitioning ◊ Boundary value analysis □ White-box (from code) ◊ Branch coverage □ Fault-based testing (from common errors) ◊ Reading online: catalog from Brian Marick 32

Coverage □ Make sure tests cover each part of program ◊ Every statement ◊ Every branch ◊ Every condition ◊ Every pass through a loop ◊ Every path(?) □ Measures the quality of tests □ How much of the program do the tests cover ? 33

34 Coverage tools □ Tool “instruments” the program □ You run your tests, it builds database □ Tool looks at database to see which parts of the program were executed, and reports test coverage □ Some Java open source tools: EclEmma, Quilt, NoUnit, InsECT, Jester, jcoverage, Coverlispe, Hansel…

35 White-box tests □ Purpose: exercise all the code □ Large number - take a long time to write □ Good for finding run-time errors ◊ Null object, array-bounds error □ In practice, coverage is better for evaluating tests than for creating them

White Box Testing - Review □ You know the code ◊ Given knowledge of the internal workings, you thoroughly test what is happening on the inside ◊ Close examination of procedural level of detail ◊ Logical paths through code are tested ○ Conditionals ○ Loops ○ Branches ◊ Status is examined in terms of expected values ◊ Impossible to thoroughly exercise all paths ○ Exhaustive testing grows without bound ◊ Can be practical if a limited number of “important” paths are evaluated ◊ Can be practical to examine and test important data structures ©L. Williams

Devising a prudent set of test cases □ Equivalence Class/Boundary Value Analysis ◊ Still applies! □ A metric for assessing how good your test suite is ◊ Method Coverage ◊ Statement Coverage ◊ Decision/Branch Coverage ◊ Condition Coverage □ Think diabolically ©L. Williams

Recall: Mistake, Fault, Error, Failure Programmer makes a mistake. Fault (defect, bug) appears in the program.Fault remains undetected during testing (running test inputs). The program fails (based on test oracles) during execution i.e. it behaves unexpectedly. Error: difference between computed, observed, or measured value or condition and true, specified, or theoretically correct value or condition What does Bug mean in “Bug Report”?

© Ammann & Offutt 39 Fault & Failure Model Three conditions necessary for a failure to be observed 1.Execution/Reachability : The location or locations in the program that contain the fault must be reached 2.Infection : The state of the program must be incorrect 3.Propagation : The infected state must propagate to cause some output of the program to be incorrect PIE model

41 Conversation 1: Tester  Test Manager □ Test Manager: Looks like the code under test is not achieving high statement coverage. Please work hard to achieve high statement coverage. □ Tester: Hmm… boss, our goal is to detect faults. I don’t think I need to spend more efforts to achieve high statement coverge. □ Test Manager: Well, according to the PIE model, …..[You fill in here]

42 Conversation 2: Tester  Test Manager □ Tester: Boss, following your command, I work very hard and I have already achieved 100% statement coverage! I would like to take a vacation in Hawaii. Could you approve? □ Test Manager: Well, according to the PIE model, …..[You fill in here]

43 Testing & Debugging □ Testing : Finding inputs that cause the software to fail □ Debugging : The process of finding a fault given a failure

© Ammann & Offutt 44 Test Case □ Test Case Values/Test Input/Test Data : The values that directly satisfy one test requirement □ Expected Results : The result that will be produced when executing the test if the program satisfies it intended behavior ◊ Related Term: Test Oracles □ Tests can mean different things in different contexts

© Ammann & Offutt 45 Observability and Controllability □ Software Observability : How easy it is to observe the behavior of a program in terms of its outputs, effects on the environment and other hardware and software components ◊ Software that affects hardware devices, databases, or remote files have low observability □ Software Controllability : How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors ◊ Easy to control software with inputs from keyboards ◊ Inputs from hardware sensors or distributed software is harder ◊ Data abstraction reduces controllability and observability

46 What is a Unit Test? A unit test is a small program with assertions. [TestMethod] public void Add() { HashSet set = new HashSet(); set.Add(3); set.Add(14); Assert.AreEqual(set.Count, 2); } Many developers write such unit tests by hand. This involves □d□determining a meaningful sequence of method calls, □s□selecting exemplary argument values (the test inputs), □s□stating assertions.

Unit Testing: Benefits □ Design and specification ◊ by example □ Code coverage and regression testing ◊ confidence in correctness ◊ preserving behavior □ Short feedback loop ◊ unit tests exercise little code ◊ failures are easy to debug □ Documentation

Unit Testing: Measuring Quality □ Coverage: Are all parts of the program exercised? ◊ statements ◊ basic blocks ◊ explicit/implicit branches ◊ … □ Assertions: Does the program do the right thing? ◊ test oracle Experience: □ Just high coverage or large number of assertions is no good quality indicator. □ Only both together are!

Advantages of tests as specs □ Concrete, easy to understand □ Don’t need new language □ Easy to see if program meets the spec □ Making tests forces you to talk to customer and learn the problem □ Making tests forces you to think about design of system (classes, methods, etc.) 49

Disadvantages of tests as specs □ Too specific □ Hard to test that something can’t happen ◊ Can’t withdraw more money than you have in the system ◊ Can’t break into the system ◊ Can’t cause a very long transaction that hangs the system □ Tends to be verbose 50

Tests as specifications □ Tests show how to use the system □ Tests need to be readable ◊ Need comments that describe their purpose or need good names ◊ Keep short, delete duplicate or redundant 51

Parameterized Unit Test □ A parameterized unit test is a small program that takes some inputs and states assumptions and assertions. 52 JUnit: @Theory (multiple parameters) @Parameters (single parameter)

Parameterized Unit Test Parameterized Unit Testing Parameterized Unit Tests □ serve as specifications □ can be leveraged by (automatic) test input generators □ fit in development environment, evolve with the code

Hand-written Test Generation Process 54 // FooTest.cs [TestClass, PexClass] partial class FooTest { [PexMethod] void Test(Foo foo) {…} // FooTest.Test.cs partial class FooTest { [TestMethod] void Test_1() { this.Test(new Foo(1)); } [TestMethod] void Test_1() { this.Test(new Foo(2)); } … } Pex User writes parameterized tests Lives inside a test class Generated unit tests Pex not required for re-execution xUnit unit tests xUnit Attributes Parameterized Unit Test Partial Class Generated http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx

PUTs separate concerns PUTs separate two concerns: (1) The specification of external behavior (i.e., assertions) (2) The selection of internal test inputs (i.e., coverage) In many cases, a test generation tool (e.g., Pex) can construct a small test suite with high coverage !

PUTs are algebraic specs A PUT can be read as a universally quantified, conditional axiom.  int name, int data. name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name, WriteResource(name, data)), data)

Pex4Fun – Turning Pex Online 1,750,069 clicked 'Ask Pex!' http://pex4fun.com/default.aspx?language=CSharp&sample=_Template

http://research.microsoft.com/pex/ Dynamic Symbolic Execution (DSE) aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths Behind the Scene of Pex4Fun

http://research.microsoft.com/pex/

Walkthrough: Unit Testing in VS □ Create Project □ Create Test Class □ Create Tests ◊ passing, failing, expected to fail □ Run Tests □ View Coverage Note: Other unit test frameworks exist for.Net, e.g. Nunit Use [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] to force generation/displaying of all explored test data

void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a.Length>0 a[0]==123… T F T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Input null {} {0} {123…} Execute&Monitor Solve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890 Done: There is no path left. Dynamic Symbolic Execution in Pex http://pex4fun.com/HowDoesPexWork

Pex is Part of Visual Studio 2015 Eneterprise Edition! □ As new feature of “IntelliTest” https://www.visualstudio.com/news/vs2015-vs#Testing

Domain Matrix for Testing Complex Condition 63

Guide Pex to Generate Test Data //[PexMethod(TestEmissionFilter = PexTestEmissionFilter.All)] [PexMethod] public void TestBoundaryValuesAndInputPartition(int x, int y) { //boundary values/partitions for x > 0 && x = 1 PexAssume.IsTrue((x > 0)); if (x == 1) { } else if (x > 0) { } PexAssume.IsTrue((x <= 10)); if (x == 10) { } else if (x <= 10) { } PexAssume.IsTrue((y >= 1)); if (y == 1) { } else if (y > 1) { } } 64 Details see http://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdfhttp://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdf

Parameterized Unit Tests Supported by Pex/Pex4Fun using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } …. } 65

Interface for IntSet Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet 66 http://www.cs.unc.edu/~stotts/723/adt.html

(Buggy) Implementation for IntSet Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } See the Set.cs that can be downloaded from http://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex” 67

Parameterized Unit Tests Supported by Pex/Pex4Fun using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } …. } 68 Pex4Fun supports only one PexMethod at a time; you can write multiple PexMethods but comment out other lines of “[PexMethod]” except one

Axioms for IntSet variables i, j : Int; s : IntSet Axioms: member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i) 69 http://www.cs.unc.edu/~stotts/723/adt.html Is this complete? How do we know?

Guidelines for Completeness □ Classify methods: ◊ constructors: return IntSet ◊ inspectors: take IntSet as argument, returning some other value. □ Identify key constructors, capable of constructing all possible object states ◊ e.g., insert, new. □ Identify others as auxiliary, ◊ e.g., remove is a destructive constructor □ Completeness requires (at least): ◊ every inspector/auxiliary constructor is defined by one equation for each key constructor. 70

Add More Axioms □ remove(new(), i) = new() □ remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j) 71 Are we done yet? The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.

Guidelines for Completeness □ But does this really specify sets? Do the following properties hold? □ Order of insertion is irrelevant. ◊ insert(insert(s, i), j) = insert(insert(s, j), i) □ Multiple insertion is irrelevant. ◊ insert(insert(s, i), i) = insert(s, i) 72

Interface (Implementation) for UIntStack Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … } } 73 See the UIntStack.cs that can be downloaded from http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

Group Exercise: Write Parameterized Unit Tests (PUTs) Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … } } 74 Let’s copy it to http://pex4fun.com/default.aspx?language= CSharp&sample=_Template And Click “Ask Pex” Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time) See the UIntStack.cs that can be downloaded from http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

(Buggy) Implementation for IntSet Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } See the Set.cs that can be downloaded from http://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex” 75

Recall: Parameterized Unit Tests Supported by Pex/Pex4Fun using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } …. } 76

Force Pex/Pex4Fun to Display All Explored Test Inputs/Paths using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } …. } 77

Factory Method: Help Pex Generate Desirable Object States In class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i); return uIntStack; // TODO: Edit factory method of UIntStack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. } 78

Factory Method: Help Pex Generate Desirable Object States Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; } 79

One Sample PUT Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); } 80

Pex4Fun Not Supporting Factory Method - Workaround If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [PexMethod] public void TestPush(int[] elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); } 81

Guideline of Writing PUT Setup: basic set up for invoking the method under test Checkpoint: Run Pex to make sure that you don't miss any Pex assumptions (preconditions) for the PUT Assert: add assertions for asserting behavior of the method under test, involving Adding Pex assertions Adding Pex assumptions for helping assert Adding method sequences for helping assert

Setup Select your method under test m Put its method call in your PUT Create a parameter for your PUT as the class under test c (annotated it with [PexAssumeUnderTest] ) Create other parameters for your PUT for parameters of m if any Add Pex assumptions for preconditions for all these parameters of PUT if any

Setup - Example [PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { s.Push(i); } You may write your factory method to help Pex in test generation If you get exceptions thrown if indicating program faults, fix them If indicating lack of PUT assumptions, add PUT assumptions If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.

Assert Think about how you can assert the behavior Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)? Do you need to add assumptions so that your assertions can be valid? Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

Targets for Asserting Return value of the method under test (MUT) Argument object of MUT Receiver object properties being modified by MUT (if public fields, directly assertable) How to assert them? Think about the intended behavior! If you couldn't do so easily, follow the guidelines discussed next

Cached Public Property Value A property value before invoking MUT may need to be cached and later used. Pattern 2.1/2.2: Assume, Arrange, Act, Assert [PexMethod] void AssumeActAssert(ArrayList list, object item) { PexAssume.IsNotNull(list); // assume var count = list.Count; // arrange list.Add(item); // act Assert.IsTrue(list.Count == count + 1); // assert }

Argument of MUT Argument value of MUT may be used Pattern 2.3:Constructor Test [PexMethod] void Constructor(int capacity) { var list = new ArrayList(capacity); // create AssertInvariant(list); // assert invariant Assert.AreEqual(capacity, list.Capacity); // assert }

Reciever or Argument of Earlier Method Receiver or argument value of a method before invoking MUT Pattern 2.4/5:Roundtrip [ PexMethod] void ToStringParseRoundtrip(int value) { // two-way roundtrip string s = value.ToString(); int parsed = int.Parse(s); // assert Assert.AreEqual(value, parsed); } value  s  parsed

Observer Methods Invoking observer methods on the modified object state Pattern 2.6: State Relation [PexMethod] void InsertContains(string value) { var list = new List (); list.Add(value); Assert.IsTrue(list.Contains(value)); } Each modified object property should be read by at least one observer method.

Observer Methods cont. Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios [PexMethod] void PushIsFull( [PexAssumeUnderTest] UIntStack s, int value) { PexAssume.IsTrue(s.GetSize() == (s.GetMaxSize()-1)); s.Push (value); Assert.IsTrue(s.IsFull ()); }

Alternative Computation Invoking another method/method sequence to produce a value to be used Pattern 2.7: Commutative Diagram [ PexMethod] void CommutativeDiagram1(int x, int y) { // compute result in one way string z1 = Multiply(x, y).ToString(); // compute result in another way string z2 = Multiply(x.ToString(), y.ToString()); // assert equality if we get here PexAssert.AreEqual(z1, z2); }

Divide and Conquer Split possible outcomes into cases (each with pre and post condition) Pattern 2.8: Cases [PexMethod] void BusinessRules(int age, Job job) { var salary = SalaryManager.ComputeSalary(age, job); PexAssert.Case(age < 30).Implies(() => salary < 10000).Case(job == Job.Manager && age > 35).Implies(() => salary > 10000).Case(job == Job.Manager && age < 20).Implies(() => false); }

Class Invariant Checker If class invariant checker (repOk) exists or you would be willing to write one, use it to assert Pattern 2.3:Constructor Test [PexMethod] void Constructor(int capacity) { var list = new ArrayList(capacity); // create AssertInvariant(list); // assert invariant Assert.AreEqual(capacity, list.Capacity); // assert }

Other Patterns Pattern 2.9: Allowed exceptions [PexAllowedException(typeof(ArgumentNullException))] [ ExpectedException(typeof(ArgumentNullException))] Pattern 2.10: Reachability [PexExpectedGoals] + throw new PexGoalException(); Pattern 2.11: Parameterized Stub No scenarios or assertions Pattern 2.12: Input Output Test void Add(int a, int b, out int result) { … } int Substract(int a, int b) { … } Pattern 2.13/14: Regression Tests bool Parse(string input) { … } PexStore.ValueForValidation("result", result); http://research.microsoft.com/en-us/projects/pex/patterns.pdf

Test-Driven Development (TDD) □ Basic Idea: ◊ Write tests before code ◊ Refine code with new tests □ In more detail, TDD is a cycle of steps: ◊ Add a test, ◊ Run it and watch it fail, ◊ Change the code as little as possible such that the test should pass, ◊ Run the test again and see it succeed, ◊ Refactor the code if needed.

Note: TDD and specifications □ TDD encourages writing specifications before code ◊ Exemplary specification □ Later, we will generalize TDD to Parameterized TDD ◊ Axiomatic specifications

Parameterized Test- Driven Development Write/refine Contract as PUT Write/refine Code of Implementation Fix-it (with Pex ), Debug with generated tests Use Generated Tests for Regression Run Pex Bug in PUT Bug in Code failures no failures

Coding Duels 1,750,069 clicked 'Ask Pex!'

Coding Duels Pex computes “semantic diff” in cloud secret reference implementation vs. code written in browser You win when Pex finds no differences secret For more info, see our ICSE 2013 SEE paper: http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

Behind the Scene of Pex for Fun Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } Player Implementation class Player { public static int Puzzle(int x) { return x ; } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } behavior Secret Impl == Player Impl 102 1,594,092

Code Hunt Programming Game https://www.codehunt.com/

Code Hunt Programming Game

It’s a game! iterative gameplay adaptive personalized no cheating clear winning criterion secret code test cases

Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign Work.

Similar presentations

Presentation on theme: "Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign Work."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign Work.

Similar presentations

Presentation on theme: "Parameterized Unit Testing: Theory and Practice Tao Xie University of Illinois at Urbana-Champaign Work."— Presentation transcript:

Similar presentations

About project

Feedback