LeanCheck

LeanCheck’s Build Status LeanCheck on Hackage LeanCheck on Stackage LTS LeanCheck on Stackage Nightly

LeanCheck logo

LeanCheck is a simple enumerative property-based testing library. Properties are defined as Haskell functions returning a boolean value which should be True for all possible choices of argument values. LeanCheck applies enumerated argument values to these properties in search for a counterexample. Properties can be viewed as parameterized unit tests.

LeanCheck works by producing tiers of test values: a possibly infinite list of finite sublists of same-and-increasingly-sized values. This enumeration is similar to Feat’s. However, the ranking and ordering of values are defined differently. The interface is also different.

Throughout this README lines that begin with the symbol > indicate a line entered into an interactive interpreter (ghci). The result of evaluating the expression is then printed on the following line.

LeanCheck implementation is easy to understand. LeanCheck’s core is under 190 lines of code.

Installing

To install the latest LeanCheck version from Hackage, just run:

$ cabal update
$ cabal install leancheck

Starting from Cabal v3.0, you need to pass --lib as an argument to cabal install:

$ cabal install leancheck --lib

Checking if properties are True

To check if properties are True, just use the function holds :: Testable a => Int -> a -> Bool. It takes two arguments: the number of values to test and a property (function returning Bool), then, it returns a boolean indicating whether the property holds. See (ghci):

> import Test.LeanCheck
> import Data.List
> holds 100 $ \xs -> sort (sort xs) == sort (xs::[Int])
True
> holds 100 $ \xs -> [] `union` xs == (xs::[Int])
False

As a rule-of-thumb, you should run holds for 500, 1 000, or 10 000 tests. With more than that you may run out-of-memory depending on the types being tested.

Finding counter examples

To find counter examples to properties, you can use the function counterExample :: Testable a => Int -> a -> Maybe [String]. It takes two arguments: the number of values to test and a property (function returning Bool). Then, it returns Nothing if no results are found or Just a list of Strings representing the offending arguments to the property. See (ghci):

> import Test.LeanCheck
> import Data.List

> counterExample 100 $ \xs -> sort (sort xs) == sort (xs::[Int])
Nothing

> counterExample 100 $ \xs -> [] `union` xs == (xs::[Int])
Just ["[0,0]"]

> counterExample 100 $ \xs ys -> xs `union` ys == ys `union` (xs::[Int])
Just ["[]","[0,0]"]

Checking properties like in SmallCheck/QuickCheck

To “check” properties like in SmallCheck and QuickCheck automatically printing results on standard output, you can use the function check :: Testable a => a -> IO ().

> import Test.LeanCheck
> import Data.List

> check $ \xs -> sort (sort xs) == sort (xs::[Int])
+++ OK, passed 200 tests.

> check $ \xs ys -> xs `union` ys == ys `union` (xs::[Int])
*** Failed! Falsifiable (after 4 tests):
[] [0,0]

The function check tests for a maximum of 200 tests. To check for a maximum of n tests, use checkFor n. To get a boolean result wrapped in IO, use checkResult or checkResultFor. There is no “quiet” option, just use holds or counterExample in that case.

Testing user-defined types

LeanCheck works on properties with Listable argument types. Listable instances are declared similarly to SmallCheck:

data MyType = MyConsA
            | MyConsB Int
            | MyConsC Int Char
            | MyConsD String

instance Listable MyType where
  tiers = cons0 MyConsA
       \/ cons1 MyConsB
       \/ cons2 MyConsC
       \/ cons1 MyConsD

For types that do not have a constraning data invariant, instances can be automatically derived with Template Haskell by using deriveListable like so:

deriveListable ''MyType

The tiers function return a potentially infinite list of finite sub-lists (tiers). Each successive tier has values of increasing size.

tiers :: Listable a => [[a]]

For convenience, the function list returns a potentially infinite list of values of the bound type:

list :: Listable a => [a]

So, for example:

> take 5 (list :: [(Int,Int)])
[(0,0),(0,1),(1,0),(0,-1),(1,1)]

The list function can be used to debug your custom instances.

Listable class instances are more customizable than what is described here: check source comments or haddock documentation for details.

Standard Listable Instances

LeanCheck comes out-of-the-box with Listable instances for all types in the Haskell 2010 Language Report with the intentional exception of a few types. The leancheck-instances package aims to support types in the Haskell Platform$ cabal install leancheck-instances.

Providers for Tasty, test-framework and Hspec

The following providers allow including LeanCheck properties into Tasty, test-framework or Hspec test suites.

Memory usage

Due to the way it is implemented (using lists of lists), LeanCheck can be quite memory intensive if we set the maximum number of tests of a property to millions of values (YMMV).

For the default maximum number of tests (200) you should be safe on most cases. If you use 1 000 or 10 000 as the maximum number of tests for a property you’re also generally safe. More than that, it is in a hit or miss basis.

For more details, see LeanCheck memory usage.

Beginner friendliness

LeanCheck strives to be beginner/student friendly both in the interface and its implementation. For instance, to understand LeanCheck’s core, one does not need to understand Monads as they aren’t used at all there.

In the name of keeping the implementation easy to understand, a compromise were made in terms of performance (cf. LeanCheck memory usage).

LeanCheck is mostly Haskell 98 compliant and almost Haskell 2010 compliant. With the exception of Listable derivation modules (TH and Generics), the only extension used by LeanCheck is CPP. This is to maintain compatibility with different compilers. LeanCheck even compiles and runs on Hugs98 from September 2006.

LeanCheck has 100% Haddock coverage with most functions having examples.

Further reading

For a detailed documentation of each function, see LeanCheck’s Haddock documentation.

For an introduction to property-based testing and a step-by-step guide to LeanCheck, see the tutorial on property-based testing with LeanCheck (doc/tutorial.md in the source repository).

LeanCheck is subject to a chapter in a PhD Thesis (2017).

Changes

Changelog for LeanCheck

v0.9.3

  • improve Haddock documentation
  • use consistent code format
  • improve CI scripts and Makefile

v0.9.2

  • rename most functions on Test.LeanCheck.Utils.Operators; deprecated names are provided;
  • improve documentation:
    • 100% haddock coverage;
    • LeanCheck memory usage thoroughly documented;
  • implement stub function conditionStatsT;
  • improve function display on Test.LeanCheck.Function.*;
  • fix some compiler warnings (newer GHC);
  • improve build scripts;
  • improve tests;
  • update tests scripts to support the new cabal (test/sdist).

v0.9.1

  • fix bug in genericTiers where using it bound to a recursive datatype could cause an infinite loop;
  • minor improvements in documentation and tests.

v0.9.0

  • logo for LeanCheck;
  • Listable instances to most types in the Haskell 2010 Language Report:
    • Word<n>;
    • Int<n>;
    • Complex;
    • etc…;
  • minor improvements in documentation and README.

v0.8.0

  • export tiersFractional from Core and main module;
  • improve Listable instance for Floats and Doubles;
  • improve Show instance for functions;
  • improve Haddock documentation;
  • remove experimental function enumeration modules, in favour of the working ListsOfPairs enumeration;
  • add special String and Char types to Utils.Types;
  • fix bug in the Natural type of the Utils.Types modules;
  • force non-negativity in Natural and Nat types from Utils.Types;
  • rename some exported symbols in the ShowFunction module;
  • improve tests of LeanCheck itself.

v0.7.7

  • Add a changelog.md file with the contents of git tag annotations: git tag -ln99.

v0.7.6

  • Add experimental Test.LeanCheck.Generic module with automatic derivation of Listable instances through GHC.Generics;
  • Improve Haddock documentation.

v0.7.5

  • Fix tests on systems with case-insensitive filesystems, like:
    • Windows;
    • Mac OS;
  • Fix tests on GHC 8.6.

This release fixes just the tests of LeanCheck itself. The LeanCheck library is otherwise unaffected.

v0.7.4

  • Add list of providers on README;
  • Minor fix in haddock.

v0.7.3

  • Fix bug: add missing Hugs backport file to source distribution (GHC users were not affected by this);
  • Improve tests so I don’t forget to include files in the source distribution (cabal sdist) again.

v0.7.2

  • Significantly improve documentation;
  • Slightly improve tests.

v0.7.1

  • LeanCheck now works on Hugs-200607 (only minor changes were needed);
  • Implement functions that calculate statistics: Test.LeanCheck.Stats;
  • More stuff on Utils: rational, okNum;
  • Improve tests;
  • Improve build scripts;
  • Minor assorted fixes.

v0.7.0

  • Improved cabal file;
  • Cabal package now has all files checked in on git repo;
  • Add functions to compute Listable statistics (and some stubs);
  • Improve tests;
  • Code improvements (refactoring).

v0.6.7

The only change in relation to v0.6.6 is a fixed build on Travis (the reference output files were outdated). The code of the tool is otherwise unchanged.

v0.6.6

  • Improve showing of functional counter-examples.

v0.6.5

  • Export ordering from ‘Test.LeanCheck.TypeBinding’;
  • Improve documentation;
  • Improve tests.

Earlier versions

Please refer to the git commit history.