QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs By Koen Claessen, Juhn Hughes ME: Mike Izbicki

Mike Izbicki / QuickCheck2 About Haskell Purely functional language Same function, same inputs -> same outputs  Functions can have no side effects Variables cannot vary Lazy evaluation -> infinite data structures Popular uses Language research High frequency trading  handles billions of dollars, so programs must be correct

Mike Izbicki / QuickCheck3 Testing in Haskell Many tests are not applicable or require significant reinterpretation Ex: What would control flow mean? Possible solutions: Small, obviously correct code Strong static typing Proofs of correctness Functional testing

Mike Izbicki / QuickCheck4 QuickCheck's goals Every function has invariants These are the specification Combine specification directly into source code  Formal method that is very expressive  Easily learned  Programmer MUST pay attention Automatically create tests from the specification

Mike Izbicki / QuickCheck5 Specification example: primes Prime sieves generate prime numbers Sieve of EratosthenesO ( N ) Sieve of AtkinO ( N / (log log N) ) Wheel sievesO ( N / (log log N) ) Difficult computations to understand, but there is a trivial verification algorithm That is our invariant It is much slower, but we only test it on some inputs

Mike Izbicki / QuickCheck6 Unconditional invariants Always easy to compute Ex: Code:

Mike Izbicki / QuickCheck7 Conditional invariants Sometimes easy to compute Ex: Sometimes hard to compute Ex:

Mike Izbicki / QuickCheck8 Generating the tests Testing simple data types, like ints, is easy Testing complex data types is hard What does a random list mean?  Size / contents / member data types Context free grammars?  Has been used before, but not powerful enough Attribute grammars?  Can define any distribution over any data type  By embedding in Haskell, we get this for free

Mike Izbicki / QuickCheck9 Special cases QuickCheck can create random, infinitely large data types Must be very careful to avoid infinite loops QuickCheck can create randomly generated functions Many functions in Haskell take other functions as arguments  Ex: map :: (a -> b) -> [a] -> [b]

Mike Izbicki / QuickCheck10 Test Adequacy By simply using enough random tests, we can make our test suite arbitrarily adequate for any coverage criteria

Mike Izbicki / QuickCheck11 Case studies Unification problem Theorem proving Edison, a data structure library Lava, a tool to simulate and formally verify hardware

Mike Izbicki / QuickCheck12 Results: Edison Every data structure, and every function was tested Maintainer of the Edison library reports: Testing requires 25% of the effort without QuickCheck QuickCheck still found more errors QuickCheck not only finds errors in the code, but also errors in the specification  Approximately a 50 / 50 split

Mike Izbicki / QuickCheck13 Results: Lava Previously proven correct QuickCheck did not find any errors QuickCheck was useful for speeding up the program Lava is an embedded language in Haskell  QuickCheck also works on Lava Very slow process to verify circuits  Can “QuickCheck” a circuit to rapidly to find trivial bugs

Mike Izbicki / QuickCheck14 QuickCheck in summary Random testing works well in functional programming languages QuickCheck forces programmers to write specifications

Mike Izbicki / QuickCheck15 QuickCheck in summary Random testing works well in functional programming languages QuickCheck forces programmers to write specifications Questions?

Mike Izbicki / QuickCheck16