doctest
Test interactive Haskell examples
https://github.com/sol/doctest#readme
Version on this page: | 0.22.6 |
LTS Haskell 22.36: | 0.22.6 |
Stackage Nightly 2024-10-05: | 0.22.6 |
Latest on Hackage: | 0.22.10 |
doctest-0.22.6@sha256:95ed169c6aaec963f6761a31611bbbe60a63128aec24ed54b2fb52af2fe9dc74,8407
Module documentation for 0.22.6
Doctest: Test interactive Haskell examples
doctest
is a tool that checks
examples
and
properties
in Haddock comments.
It is similar in spirit to the popular Python module with the same name.
- Getting started
- Running doctest for a Cabal package
- Writing examples and properties
- Limitations
- Doctest in the wild
- Development
- Contributors
Getting started
Installation
doctest
is available from
Hackage.
Install it with:
cabal update && cabal install --ignore-project doctest
Make sure that Cabal’s installdir
is on your PATH
.
On Linux / macOS / BSD:
# requires cabal-install version 3.12, or later
export PATH="$(cabal -v0 path --installdir):$PATH"
or
export PATH="$HOME/.local/bin:$PATH"
On Windows with PowerShell:
# requires cabal-install version 3.12, or later
$Env:PATH = "$(cabal -v0 path --installdir)" + ";" + $Env:PATH
A basic example
Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.
-- src/Fib.hs
module Fib where
-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
(A comment line starting with >>>
denotes an expression.
All comment lines following an expression denote the result of that expression.
Result is defined by what a
REPL (e.g. ghci)
prints to stdout
and stderr
when evaluating that expression.)
With doctest
you can check whether the implementation satisfies the given
examples:
doctest src/Fib.hs
Running doctest
for a Cabal package
The easiest way to run doctest
for a Cabal package is via cabal repl --with-compiler=doctest
.
This doesn’t make a big difference for a simple package, but in more involved
situations cabal
will make sure that all dependencies are available and it
will pass any required GHC options to doctest
.
A simple .cabal
file for Fib
looks like this:
-- fib.cabal
cabal-version: 1.12
name: fib
version: 0.0.0
build-type: Simple
library
build-depends: base == 4.*
hs-source-dirs: src
exposed-modules: Fib
default-language: Haskell2010
With a .cabal
file in place, it is possible to run doctest
via cabal repl
:
$ cabal repl --with-compiler=doctest
...
Examples: 2 Tried: 2 Errors: 0 Failures: 0
Notes:
-
If you use properties you need to pass
--build-depends=QuickCheck
and--build-depends=template-haskell
tocabal repl
. -
You likely want to reset the warning strategy for
cabal repl
with--repl-options='-w -Wdefault'
. -
doctest
always uses the version of GHC it was compiled with. Reinstallingdoctest
withcabal install doctest --overwrite-policy=always
before each invocation ensures that it uses the same version of GHC as is on thePATH
. -
Technically,
cabal build
is not necessary.cabal repl --with-compiler=doctest
will build any dependencies as needed. However, it’s more robust to runcabal build
first (specifically it is not a good idea to buildghc-paths
with--with-compiler=doctest
).
So a more robust way to call doctest
is as follows:
cabal install doctest --ignore-project --overwrite-policy=always && cabal build && cabal repl --build-depends=QuickCheck --build-depends=template-haskell --with-compiler=doctest --repl-options='-w -Wdefault'
(This is what you want to use on CI.)
Passing doctest
options to cabal repl
You can pass doctest
options like --fast
, --preserve-it
and --verbose
to
cabal repl
via --repl-options
.
Example:
$ cabal repl --with-compiler=doctest --repl-options=--verbose
### Started execution at src/Fib.hs:7.
### example:
fib 10
### Successful!
### Started execution at src/Fib.hs:10.
### example:
fib 5
### Successful!
# Final summary:
Examples: 2 Tried: 2 Errors: 0 Failures: 0
Cabal integration
NOTE: This feature is experimental.
NOTE: This feature requires cabal-install
version 3.12 or later.
$ cabal install --ignore-project doctest --flag cabal-doctest
$ cabal doctest
Examples: 2 Tried: 2 Errors: 0 Failures: 0
$ cabal doctest -w ghc-8.6.5
Examples: 2 Tried: 2 Errors: 0 Failures: 0
$ cabal doctest --repl-options=--verbose
### Started execution at src/Fib.hs:7.
### example:
fib 10
### Successful!
### Started execution at src/Fib.hs:10.
### example:
fib 5
### Successful!
# Final summary:
Examples: 2 Tried: 2 Errors: 0 Failures: 0
$ cabal doctest --build-depends transformers
Examples: 2 Tried: 2 Errors: 0 Failures: 0
Writing examples and properties
Example groups
Examples from a single Haddock comment are grouped together and share the same scope. E.g. the following works:
-- |
-- >>> let x = 23
-- >>> x + 42
-- 65
If an example fails, subsequent examples from the same group are skipped. E.g. for
-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print n
print n
is skipped, because let n = x + y
fails (as y
is not in scope).
A note on performance
By default, doctest
calls :reload
between each group to clear GHCi’s scope
of any local definitions. This ensures that previous examples cannot influence
later ones. However, it can lead to performance penalties if you are using
doctest
in a project with many modules. One possible remedy is to pass the
--fast
flag to doctest
, which disables calling :reload
between groups.
If doctest
s are running too slowly, you might consider using --fast
.
(With the caveat that the order in which groups appear now matters!)
However, note that due to a
bug on GHC 8.2.1 or later,
the performance of --fast
suffers significantly when combined with the
--preserve-it
flag (which keeps the value of GHCi’s it
value between
examples).
Setup code
You can put setup code in a named chunk with the name $setup
.
The setup code is run before each example group. If the setup code produces
any errors/failures, all tests from that module are skipped.
Here is an example:
module Foo where
import Bar.Baz
-- $setup
-- >>> let x = 23 :: Int
-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42
Note that you should not place setup code inbetween the module header (module ... where
) and import declarations. GHC will not be able to parse it (issue
#167). It is best to place setup
code right after import declarations, but due to its declarative nature you can
place it anywhere inbetween top level declarations as well.
Multi-line input
GHCi supports commands which span multiple lines, and the same syntax works for doctest:
-- |
-- >>> :{
-- let
-- x = 1
-- y = 2
-- in x + y + multiline
-- :}
-- 6
multiline = 3
Note that >>>
can be left off for the lines following the first: this is so that
haddock does not strip leading whitespace. The expected output has whitespace
stripped relative to the :}.
Some peculiarities on the ghci side mean that whitespace at the very start is lost.
This breaks the example broken
, since the x and y aren’t aligned from ghci’s
perspective. A workaround is to avoid leading space, or add a newline such
that the indentation does not matter:
{- | >>> :{
let x = 1
y = 2
in x + y + works
:}
6
-}
works = 3
{- | >>> :{
let x = 1
y = 2
in x + y + broken
:}
3
-}
broken = 3
Multi-line output
If there are no blank lines in the output, multiple lines are handled automatically.
-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!
If however the output contains blank lines, they must be noted
explicitly with <BLANKLINE>
. For example,
import Data.List ( intercalate )
-- | Double-space a paragraph.
--
-- Examples:
--
-- >>> let s1 = "\"Every one of whom?\""
-- >>> let s2 = "\"Every one of whom do you think?\""
-- >>> let s3 = "\"I haven't any idea.\""
-- >>> let paragraph = unlines [s1,s2,s3]
-- >>> putStrLn $ doubleSpace paragraph
-- "Every one of whom?"
-- <BLANKLINE>
-- "Every one of whom do you think?"
-- <BLANKLINE>
-- "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . lines
Matching arbitrary output
Any lines containing only three dots (...
) will match one or more lines with
arbitrary content. For instance,
-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- baz
If a line contains three dots and additional content, the three dots will match anything within that line:
-- |
-- >>> putStrLn "foo bar baz"
-- foo ... baz
QuickCheck properties
Haddock has markup support for properties. Doctest can verify properties with QuickCheck. A simple property looks like this:
-- |
-- prop> \xs -> sort xs == (sort . sort) (xs :: [Int])
The lambda abstraction is optional and can be omitted:
-- |
-- prop> sort xs == (sort . sort) (xs :: [Int])
A complete example that uses setup code is below:
module Fib where
-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary
-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
If you see an error like the following, ensure that
QuickCheck is visible to
doctest
(e.g. by passing --build-depends=QuickCheck
to cabal repl
).
<interactive>:39:3:
Not in scope: ‘polyQuickCheck’
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
<interactive>:39:3:
GHC stage restriction:
‘polyQuickCheck’ is used in a top-level splice or annotation,
and must be imported, not defined locally
In the expression: polyQuickCheck (mkName "doctest_prop")
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
Hiding examples from Haddock
You can put examples into named chunks, and not refer to them in the export list. That way they will not be part of the generated Haddock documentation, but Doctest will still find them.
-- $
-- >>> 1 + 1
-- 2
Using GHC extensions
There’s two sets of GHC extensions involved when running Doctest:
-
The set of GHC extensions that are active when compiling the module code (excluding the doctest examples). The easiest way to specify these extensions is through LANGUAGE pragmas in your source files.
-
The set of GHC extensions that are active when executing the Doctest examples. (These are not influenced by the LANGUAGE pragmas in the file.) The recommended way to enable extensions for Doctest examples is to switch them on like this:
-- |
-- >>> :seti -XTupleSections
-- >>> fst' $ (1,) 2
-- 1
fst' :: (a, b) -> a
fst' = fst
Alternatively you can pass any GHC options to Doctest, e.g.:
doctest -XCPP Foo.hs
These options will affect both the loading of the module and the execution of the Doctest examples.
If you want to omit the information which language extensions are enabled from the Doctest examples you can use the method described in Hiding examples from Haddock, e.g.:
-- $
-- >>> :seti -XTupleSections
Limitations
-
Doctests only works on platforms that have support for GHC’s
--interactive
mode (ghci
). -
Due to a GHC bug, running
:set -XTemplateHaskell
withinghci
may unload any modules that were specified on the command-line.To address this
doctest >= 0.19.0
does two things:- Doctest always enables
-XTemplateHaskell
. So it is safe to use Template Haskell in examples without enabling the extension explicitly. - Doctest filters out
-XTemplateHaskell
from single-line:set
-statements. So it is still safe to include:set -XTemplateHaskell
in examples for documentation purposes. It may just not work as intended inghci
due to that GHC bug.
Doctest does not filter out
-XTemplateHaskell
from multi-line:set
-statements. So if you e.g. use>>> :{ :set -XTemplateHaskell :}
then you are on your own.
Note that all platforms that support
--interactive
also support-XTemplateHaskell
. So this approach does not reduce Doctest’s platform support. - Doctest always enables
-
Modules that are rejected by
haddock
will not work withdoctest
. This can mean thatdoctest
fails on input that is accepted by GHC (e.g. #251). -
Doctest works best with UTF-8. If your locale is e.g.
LC_ALL=C
, you may want to invokedoctest
withLC_ALL=C.UTF-8
.
Doctest in the wild
You can find real world examples of Doctest
being used below:
Development
Discuss your ideas first, ideally by opening an issue on GitHub.
Add tests for new features, and make sure that the test suite passes with your changes.
cabal build && cabal exec $(cabal list-bin spec)
Contributors
- Simon Hengel
- quasicomputational
- Kazu Yamamoto
- Andreas Abel
- Michael Snoyman
- Michael Orlitzky
- Sakari Jokinen
- Adam Vogt
- Ryan Scott
- Oleg Grenrus
- Sönke Hahn
- Edward Kmett
- Elliot Marsden
- Greg Pfeil
- Ignat Insarov
- Julian K. Arni
- Takano Akio
- Joachim Breitner
- Alan Zimmerman
- Alexander Bernauer
- Alexandre Esteves
- Anders Persson
- Ankit Ahuja
- Artyom Kazak
- Gabor Greif
- Guillaume Bouchard
- Hiroki Hattori
- Jens Petersen
- John Chee
- João Cristóvão
- Leon Schoorl
- Levent Erkok
- Luke Murphy
- Matvey Aksenov
- Mitchell Rosen
- Nick Smallbone
- Nikos Baxevanis
- Tamar Christina
- Veronika Romashkina
For up-to-date list, query
git shortlog -s
Changes
Changes in 0.22.5
- Add (experimental)
cabal-doctest
executable. This is guarded behind a flag for now, usecabal install doctest -f cabal-doctest
to install it.
Changes in 0.22.4
- Use
-Wno-unused-packages
for GHC8.10
/9.0
/9.2
Changes in 0.22.3
- Use
-Wno-unused-packages
when extracting comments
Changes in 0.22.2
- GHC 9.8 compatibility
Changes in 0.22.1
- Add
Test.DocTest.Internal.Run.doctestWithRepl
Changes in 0.22.0
- Export more internals
Changes in 0.21.1
- GHC 9.6 compatibility.
Changes in 0.21.0
- Accept
--fast
,--preserve-it
and--verbose
via--repl-options
Changes in 0.20.1
- GHC 9.4 compatibility. (#382)
Changes in 0.20.0
- Allow doctest to be invoked via
cabal repl --with-compiler=doctest
- Include
ghc --info
output in--info
- Make
--info
output formatting consistent with GHC
Changes in 0.19.0
- Better support for
cabal v2-*
Changes in 0.18.2
- GHC 9.2 compatibility. (#305, thanks to Ryan Scott and Matthew Pickering)
Changes in 0.18.1
- GHC 9.0 compatibility. (#275)
Changes in 0.18
- Don’t use unqualified references to
stderr
orstdout
which may collide with definitions in user code. (#201) - Remove support for cabal-install sandboxes. They have been obsoleted in practice by Nix-style builds in cabal-install (i.e., the
v2-*
commands) and stack.
Changes in 0.17
- #266:
- doctest now annotates its internal marker string as a
String
, to prevent misbehaviour inOverloadedStrings
environments. This has a theoretical chance of breakage; if you’re affected, please open an issue. evalEcho
no longer preservesit
.
- doctest now annotates its internal marker string as a
Changes in 0.16.3
- Add a cursor to highlight the differing portion between the expected and actual output. (#249)
- GHC 8.10 compatibility. (#247, #257)
Changes in 0.16.2
- Add doctest’s necessary-for-operation options to GHC’s command line at the end, so that they over-ride anything provided by the user. (#233)
- Allow GHC 8.8.
Changes in 0.16.1
- Fix loading plugins in doctests. (#224)
- Require QuickCheck 2.13.1 or newer.
- Remove dependency on
with-location
Changes in 0.16.0.1
- Bump bounds to allow GHC 8.6. (#210)
Changes in 0.16.0
- Output format has changed to (hopefully) be more machine consumable. (#200)
Changes in 0.15.0
- Add
--verbose
for printing each test as it is run
Changes in 0.14.1
- Add test assets to source tarball (see #189)
Changes in 0.14.0
- GHC 8.4 compatibility.
Changes in 0.13.0
- Add
--preserve-it
for allowing theit
variable to be preserved between examples
Changes in 0.12.0
- Preserve the ‘it’ variable between examples
Changes in 0.11.4
- Add
--fast
, which disables running:reload
between example groups
Changes in 0.11.3
- Add
--info
- Add
--no-magic
Changes in 0.11.2
- Make
...
match zero lines
Changes in 0.11.1
- Fix an issue with Unicode output on Windows (see #149)
Changes in 0.11.0
- Support for GHC 8.0.1-rc2
Changes in 0.10.1
- Automatically expand directories into contained Haskell source files (thanks @snoyberg)
- Add cabal_macros.h and autogen dir by default (thanks @snoyberg)
Changes in 0.10.0
- Support HASKELL_PACKAGE_SANDBOXES (thanks @snoyberg)
Changes in 0.9.13
- Add ellipsis as wildcard
Changes in 0.9.12
- Add support for GHC 7.10
Changes in 0.9.11
- Defaults ambiguous type variables to Integer (#74)
Changes in 0.9.10
- Add support for the upcoming GHC 7.8 release
Changes in 0.9.9
- Add support for multi-line statements
Changes in 0.9.8
- Support for GHC HEAD (7.7)
Changes in 0.9.7
- Ignore trailing whitespace when matching example output
Changes in 0.9.6
- Fail gracefully if GHCi is not supported (#46)
Changes in 0.9.5
- Fix a GHC panic with GHC 7.6.1 (#41)
Changes in 0.9.4
- Respect HASKELL_PACKAGE_SANDBOX (#39)
- Print path to ghc on –version
Changes in 0.9.3
- Properly handle additional object files (#38)
Changes in 0.9.2
- Add support for QuickCheck properties
Changes in 0.9.1
- Fix an issue with GHC 7.6.1 and type families
Changes in 0.9.0
- Add support for setup code (see README).
- There is no distinction between example/interaction anymore. Each expression is counted as an example in the summary.
Changes in 0.8.0
- Doctest now directly accepts arbitrary GHC options, prefixing GHC options with –optghc is no longer necessary
Changes in 0.7.0
- Print source location for failing tests
- Output less clutter on failing examples
- Expose Doctest’s functionality through a very simplistic API, which can be used for cabal integration
Changes in 0.6.1
- Fix a parser bug with CR+LF line endings
Changes in 0.6.0
- Support for ghc-7.4
- Doctest now comes with it’s own parser and does not depend on Haddock anymore
Changes in 0.5.2
- Proper handling of singular/plural when printing stats
- Improve handling of invalid command line options
Changes in 0.5.1
- Adapted for ghc-7.2
Changes in 0.5.0
- Print number of interactions to stderr before running tests
- Exit with exitFailure on failed tests
- Improve documentation
- Give a useful error message if ghc is not executable