leancheck
Enumerative property-based testing
https://github.com/rudymatela/leancheck#readme
LTS Haskell 22.43: | 1.0.2 |
Stackage Nightly 2024-12-07: | 1.0.2 |
Latest on Hackage: | 1.0.2 |
leancheck-1.0.2@sha256:5c2a164e1120e404d51a150873f4ada4738379e5fb665ea71ba289dc245c89a4,7090
Module documentation for 1.0.2
- Test
- Test.LeanCheck
- Test.LeanCheck.Basic
- Test.LeanCheck.Core
- Test.LeanCheck.Derive
- Test.LeanCheck.Error
- Test.LeanCheck.Function
- Test.LeanCheck.Generic
- Test.LeanCheck.IO
- Test.LeanCheck.Stats
- Test.LeanCheck.Tiers
- Test.LeanCheck.Utils
- Test.LeanCheck
LeanCheck
LeanCheck is a simple enumerative property-based testing library. Properties
are defined as Haskell functions returning a boolean value which should be
True
for all possible choices of argument values. LeanCheck applies
enumerated argument values to these properties in search for a counterexample.
Properties can be viewed as parameterized unit tests.
LeanCheck works by producing tiers of test values: a possibly infinite list of finite sublists of same-and-increasingly-sized values. This enumeration is similar to Feat’s. However, the ranking and ordering of values are defined differently. The interface is also different.
Throughout this README lines that begin with the symbol >
indicate a line
entered into an interactive interpreter (ghci
). The result of evaluating the
expression is then printed on the following line.
LeanCheck implementation is easy to understand. LeanCheck’s core is under 200 lines of code.
Installing
To install the latest LeanCheck version from Hackage just run:
$ cabal update
$ cabal install leancheck
Starting from Cabal v3.0, you need to pass --lib
as an argument to
cabal install
to install packages globally on the default
user environment:
$ cabal install leancheck --lib
If you already have LeanCheck installed Cabal may refuse to update to the latest version. To update, you need to reset your user’s cabal installation with:
rm -rf ~/.cabal/{bin,lib,logs,share,store} ~/.ghc/*/
WARNING: the above command will erase all user-local packages.
LeanCheck has (official) packages available on Stackage, OpenSUSE, Gentoo, Arch Linux and NixOS.
Checking if properties are True
To check if properties are True,
just use the function holds
:: Testable a => Int -> a -> Bool
.
It takes two arguments:
the number of values to test
and a property (function returning Bool),
then, it returns a boolean indicating whether the property holds.
See (ghci):
> import Test.LeanCheck
> import Data.List
> holds 100 $ \xs -> sort (sort xs) == sort (xs::[Int])
True
> holds 100 $ \xs -> [] `union` xs == (xs::[Int])
False
As a rule-of-thumb, you should run holds for 500, 1 000, or 10 000 tests. With more than that you may run out-of-memory depending on the types being tested.
Finding counter examples
To find counter examples to properties,
you can use the function counterExample
:: Testable a => Int -> a -> Maybe [String]
.
It takes two arguments:
the number of values to test
and a property (function returning Bool).
Then, it returns Nothing if no results are found or Just a list of Strings
representing the offending arguments to the property.
See (ghci):
> import Test.LeanCheck
> import Data.List
> counterExample 100 $ \xs -> sort (sort xs) == sort (xs::[Int])
Nothing
> counterExample 100 $ \xs -> [] `union` xs == (xs::[Int])
Just ["[0,0]"]
> counterExample 100 $ \xs ys -> xs `union` ys == ys `union` (xs::[Int])
Just ["[]","[0,0]"]
Checking properties like in SmallCheck/QuickCheck
To “check” properties like in SmallCheck and QuickCheck
automatically printing results on standard output,
you can use the function check
:: Testable a => a -> IO ()
.
> import Test.LeanCheck
> import Data.List
> check $ \xs -> sort (sort xs) == sort (xs::[Int])
+++ OK, passed 200 tests.
> check $ \xs ys -> xs `union` ys == ys `union` (xs::[Int])
*** Failed! Falsifiable (after 4 tests):
[] [0,0]
The function check
tests for a maximum of 200 tests.
To check for a maximum of n
tests, use checkFor
n
.
To get a boolean result wrapped in IO
, use checkResult
or checkResultFor
.
There is no “quiet” option, just use holds
or counterExample
in that case.
Testing user-defined types
LeanCheck works on properties with Listable
argument types.
Listable
instances are declared similarly to SmallCheck:
data MyType = MyConsA
| MyConsB Int
| MyConsC Int Char
| MyConsD String
instance Listable MyType where
tiers = cons0 MyConsA
\/ cons1 MyConsB
\/ cons2 MyConsC
\/ cons1 MyConsD
For types that do not have a constraning data invariant, instances can be
automatically derived with Template Haskell by using deriveListable
like
so:
deriveListable ''MyType
The tiers
function return a potentially infinite list of finite sub-lists
(tiers). Each successive tier has values of increasing size.
tiers :: Listable a => [[a]]
For convenience, the function list
returns a potentially infinite list
of values of the bound type:
list :: Listable a => [a]
So, for example:
> take 5 (list :: [(Int,Int)])
[(0,0),(0,1),(1,0),(0,-1),(1,1)]
The list
function can be used to debug your custom instances.
Listable
class instances are more customizable than what is described here:
check source comments or haddock documentation for details.
Standard Listable Instances
LeanCheck comes out-of-the-box with Listable
instances for all types in the
Haskell 2010 Language Report with the intentional exception of a few types.
The leancheck-instances package aims to support types in the
Haskell Platform – $ cabal install leancheck-instances
.
Providers for Tasty, test-framework and Hspec
The following providers allow including LeanCheck properties into Tasty, test-framework or Hspec test suites.
- LeanCheck provider for Tasty
–
$ cabal install tasty-leancheck
; - LeanCheck provider for test-framework
–
$ cabal install test-framework-leancheck
; - LeanCheck provider for Hspec
–
$ cabal install hspec-leancheck
.
Memory usage
Due to the way it is implemented (using lists of lists), LeanCheck can be quite memory intensive if we set the maximum number of tests of a property to millions of values (YMMV).
For the default maximum number of tests (200) you should be safe on most cases. If you use 1 000 or 10 000 as the maximum number of tests for a property you’re also generally safe. More than that, it is in a hit or miss basis.
For more details, see LeanCheck memory usage.
Beginner friendliness
LeanCheck strives to be beginner/student friendly both in the interface and its implementation. For instance, to understand LeanCheck’s core, one does not need to understand Monads as they aren’t used at all there.
In the name of keeping the implementation easy to understand, a compromise were made in terms of performance (cf. LeanCheck memory usage).
LeanCheck is mostly Haskell 98 compliant and almost Haskell 2010 compliant.
With the exception of Listable
derivation modules (TH and Generics),
the only extension used by LeanCheck is CPP. This is to maintain
compatibility with different compilers. LeanCheck even compiles and runs on
Hugs98 from September 2006.
LeanCheck has 100% Haddock coverage with most functions having examples.
Further reading
For a detailed documentation of each function, see LeanCheck’s Haddock documentation.
For an introduction to property-based testing
and a step-by-step guide to LeanCheck, see the
tutorial on property-based testing with LeanCheck
(doc/tutorial.md
in the source repository).
LeanCheck is subject to a chapter in a PhD Thesis (2017).
LeanCheck has a list of frequently asked questions and answers.
Changes
Changelog for LeanCheck
v1.0.2 (January 2024)
- improve error messages
- some code linting, refactoring and cleanup
- improve testing of LeanCheck itself
v1.0.0 (August 2022)
This release indicates that the Test.LeanCheck
API is now stable.
This is the same as v0.9.12 with just the version number change,
the major bump here serves just to indicate stability.
If your project works with the 0.9 series, it will work with the 1.0 series.
v0.9.12 (August 2022)
- replace enumeration of
Fractional
andFloating
types by the Calkin-Wilf sequence:- add
listFloating
andlistFractional
; - deprecate
tiersFloating
andtiersFractional
;
- add
- improve the changelog with timestamps and history down to v0.5.0;
- normalize code formatting throughout: indentation, spacing and width;
- improve testing of LeanCheck itself;
- test on GHC 9.2 and 9.4 (works with no changes).
v0.9.10 (June 2021)
Test.LeanCheck.Utils.Type
: derive Typeable instances on GHC 7.8. Behaviour on newer GHCs (>= 7.10) versions is unaffected as they automatically derive Typeable instances for all types.
v0.9.8 (June 2021)
Test.LeanCheck.Utils.Type
: ~Typeable instances on GHC 7.10.~ ~Behaviour on newer GHCs (>= 8.0) versions is unaffected~ ~as they automatically derive Typeable instances for all types.~ Update: these instances were already present on v0.9.6. LeanCheck v0.9.8 is essentially the same as v0.9.6.
v0.9.6 (May 2021)
- no code changes in what is exported by
Test.LeanCheck
Test.LeanCheck.Utils.Types
: export theA
,B
,C
,D
,E
andF
types- slightly improve README
- improve Makefile and tests
- replace Travis by GitHub Actions as the CI system
v0.9.4 (April 2021)
- no code changes in what is exported by
Test.LeanCheck
- add
errorToLeft
andanyErrorToLeft
onTest.LeanCheck.Error
- add
?==?
and!==!
onTest.LeanCheck.Error
- add
Test.LeanCheck.Function.List
- add
Test.LeanCheck.Function.Ord
- reduce default argument enumeration to 12 on
Test.LeanCheck.Function.Eq
- add FAQ
- improve Makefile and test scripts
v0.9.3 (March 2020)
- improve Haddock documentation
- use consistent code format
- improve CI scripts and Makefile
v0.9.2 (March 2020)
- rename most functions on
Test.LeanCheck.Utils.Operators
; deprecated names are provided; - improve documentation:
- 100% haddock coverage;
- LeanCheck memory usage thoroughly documented;
- implement stub function
conditionStatsT
; - improve function display on
Test.LeanCheck.Function.*
; - fix some compiler warnings (newer GHC);
- improve build scripts;
- improve tests;
- update tests scripts to support the new cabal (
test/sdist
).
v0.9.1 (February 2019)
- fix bug in
genericTiers
where using it bound to a recursive datatype could cause an infinite loop; - minor improvements in documentation and tests.
v0.9.0 (January 2019)
- logo for LeanCheck;
Listable
instances to most types in the Haskell 2010 Language Report:Word<n>
;Int<n>
;Complex
;- etc…;
- minor improvements in documentation and README.
v0.8.0 (November 2018)
- export
tiersFractional
fromCore
and main module; - improve
Listable
instance forFloat
s andDouble
s; - improve
Show
instance for functions; - improve Haddock documentation;
- remove experimental function enumeration modules,
in favour of the working
ListsOfPairs
enumeration; - add special
String
andChar
types toUtils.Types
; - fix bug in the
Natural
type of theUtils.Types
modules; - force non-negativity in
Natural
andNat
types fromUtils.Types
; - rename some exported symbols in the
ShowFunction
module; - improve tests of LeanCheck itself.
v0.7.7 (October 2018)
- Add a
changelog.md
file with the contents of git tag annotations:git tag -ln99
.
v0.7.6 (October 2018)
- Add experimental
Test.LeanCheck.Generic
module with automatic derivation of Listable instances throughGHC.Generics
; - Improve Haddock documentation.
v0.7.5 (September 2018)
- Fix tests on systems with case-insensitive filesystems, like:
- Windows;
- Mac OS;
- Fix tests on GHC 8.6.
This release fixes just the tests of LeanCheck itself. The LeanCheck library is otherwise unaffected.
v0.7.4 (September 2018)
- Add list of providers on README;
- Minor fix in haddock.
v0.7.3 (August 2018)
- Fix bug: add missing Hugs backport file to source distribution (GHC users were not affected by this);
- Improve tests so I don’t forget to include files in the source distribution (cabal sdist) again.
v0.7.2 (August 2018)
- Significantly improve documentation;
- Slightly improve tests.
v0.7.1 (July 2018)
- LeanCheck now works on Hugs-200607 (only minor changes were needed);
- Implement functions that calculate statistics:
Test.LeanCheck.Stats
; - More stuff on
Utils
:rational
,okNum
; - Improve tests;
- Improve build scripts;
- Minor assorted fixes.
v0.7.0 (December 2017)
- Improved cabal file;
- Cabal package now has all files checked in on git repo;
- Add functions to compute Listable statistics (and some stubs);
- Improve tests;
- Code improvements (refactoring).
v0.6.7 (September 2017)
The only change in relation to v0.6.6 is a fixed build on Travis (the reference output files were outdated). The code of the tool is otherwise unchanged.
v0.6.6 (September 2017)
- Improve showing of functional counter-examples.
v0.6.5 (August 2017)
- Export ordering from ‘Test.LeanCheck.TypeBinding’;
- Improve documentation;
- Improve tests.
v0.6.4 (August 2017)
- Add and export the
Xs
constructor inUtils
; - Add type binding operators for up to 12 arguments.
v0.6.3 (July 2017)
Test.LeanCheck
: Add thedelay
andreset
functions;Test.LeanCheck.Tiers
:normalizeT
stripping up to 12 empty tiers;- Some refactoring;
- Add lots of new tests for LeanCheck itself.
v0.6.2 (March 2017)
- Improve documentation;
- Improve tests of LeanCheck itself (including putting LeanCheck on CI).
v0.6.1 (March 2017)
Test.LeanCheck.Error
: exportfromError
;Test.LeanCheck.Tiers
: exportmapMaybeT
,maybeConsN
andcatMaybesT
;- Some code refactoring and internal experiments with function enumeration.
v0.6.0 (October 2016)
- Add
deriveListableCascading
; - Add
Listable
instances forWord
andOrdering
; - Improve tests of LeanCheck itself.
v0.5.0 (July 2016)
- Improve documentation;
- Rename several functions;
- Add several utilities in
Utils
.
Earlier versions
Please refer to the git commit history.