Tasty is a modern testing framework for Haskell.
It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.
- Run tests in parallel but report results in a deterministic order
- Filter the tests to be run using patterns specified on the command line
- Hierarchical, colored display of test results
- Reporting of test statistics
- Acquire and release resources (sockets, temporary files etc.) that can be shared among several tests
- Extensibility: add your own test providers and ingredients (runners) above and beyond those provided
To find out what's new, read the change log.
Here's how your
test.hs might look like:
import Test.Tasty import Test.Tasty.SmallCheck as SC import Test.Tasty.QuickCheck as QC import Test.Tasty.HUnit import Data.List import Data.Ord main = defaultMain tests tests :: TestTree tests = testGroup "Tests" [properties, unitTests] properties :: TestTree properties = testGroup "Properties" [scProps, qcProps] scProps = testGroup "(checked by SmallCheck)" [ SC.testProperty "sort == sort . reverse" $ \list -> sort (list :: [Int]) == sort (reverse list) , SC.testProperty "Fermat's little theorem" $ \x -> ((x :: Integer)^7 - x) `mod` 7 == 0 -- the following property does not hold , SC.testProperty "Fermat's last theorem" $ \x y z n -> (n :: Integer) >= 3 SC.==> x^n + y^n /= (z^n :: Integer) ] qcProps = testGroup "(checked by QuickCheck)" [ QC.testProperty "sort == sort . reverse" $ \list -> sort (list :: [Int]) == sort (reverse list) , QC.testProperty "Fermat's little theorem" $ \x -> ((x :: Integer)^7 - x) `mod` 7 == 0 -- the following property does not hold , QC.testProperty "Fermat's last theorem" $ \x y z n -> (n :: Integer) >= 3 QC.==> x^n + y^n /= (z^n :: Integer) ] unitTests = testGroup "Unit tests" [ testCase "List comparison (different length)" $ [1, 2, 3] `compare` [1,2] @?= GT -- the following test does not hold , testCase "List comparison (same length)" $ [1, 2, 3] `compare` [1,2,2] @?= LT ]
And here is the output of the above program:
(Note that whether QuickCheck finds a counterexample to the third property is determined by chance.)
tasty is the core package. It contains basic definitions and APIs and a console runner.
In order to create a test suite, you also need to install one or more «providers» (see below).
The following providers exist:
- tasty-hunit — for unit tests (based on HUnit)
- tasty-golden — for golden tests, which are unit tests whose results are kept in files
- tasty-smallcheck — exhaustive property-based testing (based on smallcheck)
- tasty-quickcheck — for randomized property-based testing (based on QuickCheck)
- tasty-hspec — for Hspec tests
- tasty-program — run external program and test whether it terminates successfully
It's easy to create custom providers using the API from
Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.
Another standard ingredient is one that simply prints the names of all tests.
It is possible to write custom ingredients using the API from
Some ingredients that can enhance your test suite are:
- tasty-ant-xml adds a possibility to write the test results in a machine-readable XML format, which is understood by various CI systems and IDEs
- tasty-rerun adds support for minimal test reruns by recording previous test runs and using this information to filter the test tree. For example, you can use this ingredient to only run failed tests, or only run tests that threw an exception.
- tasty-html adds the possibility to write the test results as a HTML file
- tasty-stats adds the possibility to collect statistics of the test suite in a CSV file.
- tasty-th automatically discovers tests based on the function names and generate the boilerplate code for you
- tasty-hunit-adapter converts existing HUnit test suites into tasty test suites
- tasty-discover automatically discovers your tests.
- tasty-expected-failure provides test markers for when you expect failures or wish to ignore tests.
Options allow one to customize the run-time behavior of the test suite, such as:
- mode of operation (run tests, list tests, run tests quietly etc.)
- which tests are run (see «Patterns» below)
- parameters of individual providers (like depth of search for SmallCheck)
There are two main ways to set options:
When using the standard console runner, the options can be passed on the
command line or via environment variables. To see the available options, run
your test suite with the
--help flag. The output will look something like this
(depending on which ingredients and providers the test suite uses):
% ./test --help Mmm... tasty test suite Usage: test [-p|--pattern ARG] [-t|--timeout ARG] [-l|--list-tests] [-j|--num-threads ARG] [-q|--quiet] [--hide-successes] [--color ARG] [--quickcheck-tests ARG] [--quickcheck-replay ARG] [--quickcheck-show-replay ARG] [--quickcheck-max-size ARG] [--quickcheck-max-ratio ARG] [--quickcheck-verbose] [--smallcheck-depth ARG] Available options: -h,--help Show this help text -p,--pattern ARG Select only tests that match pattern -t,--timeout ARG Timeout for individual tests (suffixes: ms,s,m,h; default: s) -l,--list-tests Do not run the tests; just print their names -j,--num-threads ARG Number of threads to use for tests execution -q,--quiet Do not produce any output; indicate success only by the exit code --hide-successes Do not print tests that passed successfully --color ARG When to use colored output. Options are 'never', 'always' and 'auto' (default: 'auto') --quickcheck-tests ARG Number of test cases for QuickCheck to generate --quickcheck-replay ARG Replay token to use for replaying a previous test run --quickcheck-show-replay ARG Show a replay token for replaying tests --quickcheck-max-size ARG Size of the biggest test cases quickcheck generates --quickcheck-max-ratio ARG Maximum number of discared tests per successful test before giving up --quickcheck-verbose Show the generated test cases --smallcheck-depth ARG Depth to use for smallcheck tests
Every option can be passed via environment. To obtain the environment variable
name from the option name, replace hyphens
- with underscores
all letters, and prepend
TASTY_. For example, the environment equivalent of
TASTY_SMALLCHECK_DEPTH. To turn on a switch (such as
TASTY_HIDE_SUCCESSES), set the variable to
If you're using a non-console runner, please refer to its documentation to find out how to configure options during the run time.
You can also specify options in the test suite itself, using
localOption. It can be applied not only to the whole test tree, but also to
individual tests or subgroups, so that different tests can be run with
It is possible to combine run-time and compile-time options, too, by using
adjustOption. For example, make the overall testing depth configurable
during the run time, but increase or decrease it slightly for individual
This method currently doesn't work for ingredient options, such as
--num-threads. You can set them by setting the corresponding environment
variable before calling
import Test.Tasty import System.Environment main = do setEnv "TASTY_NUM_THREADS" "1" defaultMain _
It is possible to restrict the set of executed tests using the
option. The syntax of patterns is the same as for test-framework, namely:
- An optional prefixed bang
!negates the pattern.
- If the pattern ends with a slash, it is removed for the purpose of
the following description, but it would only find a match with a
test group. In other words,
foo/will match a group called
fooand any tests underneath it, but will not match a regular test
- If the pattern does not contain a slash
/, the framework checks for a match against any single component of the path.
- Otherwise, the pattern is treated as a glob, where:
- The wildcard
*matches anything within a single path component (i.e.
- Two wildcards
**matches anything (i.e.
- Anything else matches exactly that text in the path (i.e.
foowould only match a component of the test path called
foo(or a substring of that form).
- The wildcard
group/test1 but not
group/subgroup/test1, whereas both examples would be matched by
group/**1. A leading slash matches the beginning of the test path; for
test1 but not
Running tests in parallel
In order to run tests in parallel, you have to do the following:
- Compile (or, more precisely, link) your test program with the
- Launch the program with
+RTS -N -RTS.
To apply timeout to individual tests, use the
option, or set the option in your test suite using the
Timeouts can be fractional, and can be optionally followed by a suffix
m (minutes), or
h (hours). When there's no
suffix, seconds are assumed.
sets a 30 seconds timeout for each individual test.
Options controlling console output
The following options control behavior of the standard console interface:
- Run the tests but don't output anything. The result is indicated only by the exit code, which is 1 if at least one test has failed, and 0 if all tests have passed. Execution stops when the first failure is detected, so not all tests are necessarily run. This may be useful for various batch systems, such as commit hooks.
- Report only the tests that has failed. Especially useful when the number of tests is large.
- Don't run the tests; only list their names, in the format accepted by
- Whether to produce colorful output. Accepted values:
automeans that colors will only be enabled when output goes to a terminal and is the default value.
It is possible to add custom options, too.
To do that,
- Define a datatype to represent the option, and make it an instance of
- Register the options with the
- To query the option value, use
See the Custom options in Tasty article for some examples.
Project organization and integration with Cabal
There may be several ways to organize your project. What follows is not Tasty's requirements but my recommendations.
Tests for a library
Place your test suite sources in a dedicated subdirectory (called
here) instead of putting them among the main library sources.
The directory structure will be as follows:
my-project/ my-project.cabal src/ ... tests/ test.hs Mod1.hs Mod2.hs ...
test.hs is where your
main function is defined. The tests may be
test.hs or spread across multiple modules (
...) which are then imported by
Add the following section to the cabal file (
test-suite test default-language: Haskell2010 type: exitcode-stdio-1.0 hs-source-dirs: tests main-is: test.hs build-depends: base >= 4 && < 5 , tasty >= 0.7 -- insert the current version here , my-project -- depend on the library we're testing , ...
Tests for a program
All the above applies, except you can't depend on the library if there's no library. You have two options:
- Re-organize the project into a library and a program, so that both the program and the test suite depend on this new library. The library can be declared in the same cabal file.
- Add your program sources directory to the
Hs-source-dirs. Note that this will lead to double compilation (once for the program and once for the test suite).
How do I make some tests execute after others?
Blog posts and other publications related to tasty. If you wrote or just found something not mentioned here, send a pull request!
- Holy Haskell Project Starter
- First time testing, also with FP Complete (tasty has been added to stackage since then)
- 24 Days of Hackage: tasty
- Resources in Tasty
- Custom options in Tasty
- Resources in Tasty (update)
- Announcing tasty-rerun
- Code testing in Haskell revisited (with Tasty)
Tasty is heavily influenced by test-framework.
The problems with test-framework are:
- Poor code style (some lines of the code wouldn't even fit in a twitter message!)
- Poor architecture — e.g. relying on laziness for IO and control flow. The
whole story with
ImprovingIOis really obscure.
- Non-extensible options. For example, when I integrated SmallCheck with
test-framework (in the form of the
test-framework-smallcheckpackage), I still had to submit patches to the main package to make SmallCheck depth customizable by the user.
- The project is effectively unmaintained.
So I decided to recreate everything that I liked in test-framework from scratch in this package.
Roman Cheplyaka is the primary maintainer.
Oliver Charles is the backup maintainer. Please get in touch with him if the primary maintainer cannot be reached.
Make filtering tests (
-p) work faster
Fix a critical bug in the quiet mode (
the exit status could be wrong or the test suite could hang.
Fix compatibility with the latest
composeReporters, a function to run multiple reporter ingredients
Fix compatibility with
regex-tdfa, which got a new maintainer.
IsTest’s specification with regard to exceptions
Use monotonic clock when measuring durations.
- Improve the docs
- Fix compatibility with GHC HEAD
- Prevent parsing non-positive number of threads via program options (#104)
- Buffer output to avoid slowdowns when printing test results (#101)
- Default to using the maximum number of available cores for test execution
Don't output ANSI codes for the Emacs terminal emulator
Better handle the situation when there are no ingredients to run
Split the changelog into per-project changelogs
Update to optparse-applicative 0.11
- Add the
- Introduce the
- Change the types of
TestReporterto accept the total run time
consoleTestReporternow displays the timings
- Introduce the
Upgrade to optparse-applicative-0.10.
Be careful not to export the
Show (a -> b) instance, see
Hide cursor when running tests
Fix for GHC 7.9
Remove the old 'colors' flag description from the cabal file
Make ansi-terminal an unconditional dependency
Test.Tasty.Ingredientsis now exposed
Test.Tasty.Ingredients.Basicis added, which exports the ingredients defined in the
tastypackage. These exports should now be used instead of ones exported from
Resulttype is now structured a bit differently. Providers now should use
testFailedfunctions instead of constructing
- Add «quiet mode» (see README)
- Add «hide successes» mode (see README)
- Add short command-line options:
- Add timeout support
AppMonoidis renamed to
Traversalfor consistency with the 'reducers' package. Another similar wrapper,
Ap, is introduced.
- Fix a resources bug (resources were not released if the test suite was interrupted)
- The type of
launchTestTreeis changed. It now takes a continuation as an argument. This is necessary to fix the bug mentioned above.
flagCLParserto be used as the
optionCLParserimplementation for boolean options.
- Add the ability to pass options via environment
regex-posix(which is a native implementation, and as such is more portable)
foldTestTreenow takes the algebra in the form of a record rather than multiple arguments, to minimize breakage when new nodes are added or existing ones change
withResourcenow passes the IO action to get the resource to the inner test tree
- Better handling of exceptions that arise during resource creation or disposal
- Expose the
Depend on ansi-terminal >= 0.6.1. This fixes some issues with colors on Windows.
- Make it clear that only GHC 7.4+ is supported
Add a capability to acquire and release resources. See the «Resources» section
For the end users, the API is backwards-compatible.
Test runners may have to be adjusted — there is a new constructor of
and a new argument of
Print the failure description in red
Fix a bug (#25)
The big change in this release is introduction of ingredients, which is a replacement for runners. But unless you have a custom runner, this is unlikely to affect you much.
Ingredient data type has replaced the
The following functions have been renamed and possibly changed their types:
Added in this release:
- functions operating on ingredients
listingTestsingredient and its option,
NumThreads is no longer a core option, but is automatically included in the
test reporting ingredients (see its haddock).
- Proper reporting of (some) non-terminating tests (#15)
- Upgrade to optparse-applicative 0.6
- Restrict dependency versions
- Fix a bug where non-terminating test would lead to a deadlock (#15)
- Add an
Set lower bound on optparse-applicative dependency version