MIT licensed by Simon Hengel
Maintained by [email protected], Andreas Abel
This version can be pinned in stack with:doctest-0.18.2@sha256:489a6e8d3d687845798a73f7b53b43ebec1a8368d7cd6a959a91080ed253bcd7,7241

Module documentation for 0.18.2

Doctest: Test interactive Haskell examples

doctest is a small program, that checks examples in Haddock comments. It is similar to the popular Python module with the same name.

Installation

doctest is available from Hackage. Install it, by typing:

cabal install doctest

Make sure that Cabal’s bindir is on your PATH.

On Linux:

export PATH="$HOME/.cabal/bin:$PATH"

On Mac OS X:

export PATH="$HOME/Library/Haskell/bin:$PATH"

On Windows:

set PATH="%AppData%\cabal\bin\;%PATH%"

For more information, see the section on paths in the Cabal User Guide.

Usage

Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.

module Fib where

-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

(A comment line starting with >>> denotes an expression. All comment lines following an expression denote the result of that expression. Result is defined by what a REPL (e.g. ghci) prints to stdout and stderr when evaluating that expression.)

With doctest you may check whether the implementation satisfies the given examples, by typing:

doctest Fib.hs

You may produce Haddock documentation for that module with:

haddock -h Fib.hs -o doc/

doctest will fail on comments that haddock also doesn’t like. Sometimes (e.g., #251), this means that doctest will fail on input that GHC accepts.

doctest likes UTF-8. If you are running it with, e.g., LC_ALL=C, you may need to invoke doctest with LC_ALL=C.UTF-8.

Example groups

Examples from a single Haddock comment are grouped together and share the same scope. E.g. the following works:

-- |
-- >>> let x = 23
-- >>> x + 42
-- 65

If an example fails, subsequent examples from the same group are skipped. E.g. for

-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print n

print n is not tried, because let n = x + y fails (y is not in scope!).

A note on performance

By default, doctest calls :reload between each group to clear GHCi’s scope of any local definitions. This ensures that previous examples cannot influence later ones. However, it can lead to performance penalties if you are using doctest in a project with many modules. One possible remedy is to pass the --fast flag to doctest, which disables calling :reload between groups. If doctests are running too slowly, you might consider using --fast. (With the caveat that the order in which groups appear now matters!)

However, note that due to a bug on GHC 8.2.1 or later, the performance of --fast suffers significantly when combined with the --preserve-it flag (which keeps the value of GHCi’s it value between examples).

Setup code

You can put setup code in a named chunk with the name $setup. The setup code is run before each example group. If the setup code produces any errors/failures, all tests from that module are skipped.

Here is an example:

module Foo where

import Bar.Baz

-- $setup
-- >>> let x = 23 :: Int

-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42

Note that you should not place setup code inbetween the module header (module ... where) and import declarations. GHC will not be able to parse it (issue #167). It is best to place setup code right after import declarations, but due to its declarative nature you can place it anywhere inbetween top level declarations as well.

Multi-line input

GHCi supports commands which span multiple lines, and the same syntax works for doctest:

-- |
-- >>> :{
--  let
--    x = 1
--    y = 2
--  in x + y + multiline
-- :}
-- 6
multiline = 3

Note that >>> can be left off for the lines following the first: this is so that haddock does not strip leading whitespace. The expected output has whitespace stripped relative to the :}.

Some peculiarities on the ghci side mean that whitespace at the very start is lost. This breaks the example broken, since the x and y aren’t aligned from ghci’s perspective. A workaround is to avoid leading space, or add a newline such that the indentation does not matter:

{- | >>> :{
let x = 1
    y = 2
  in x + y + works
:}
6
-}
works = 3

{- | >>> :{
 let x = 1
     y = 2
  in x + y + broken
:}
3
-}
broken = 3

Multi-line output

If there are no blank lines in the output, multiple lines are handled automatically.

-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!

If however the output contains blank lines, they must be noted explicitly with <BLANKLINE>. For example,

import Data.List ( intercalate )

-- | Double-space a paragraph.
--
--   Examples:
--
--   >>> let s1 = "\"Every one of whom?\""
--   >>> let s2 = "\"Every one of whom do you think?\""
--   >>> let s3 = "\"I haven't any idea.\""
--   >>> let paragraph = unlines [s1,s2,s3]
--   >>> putStrLn $ doubleSpace paragraph
--   "Every one of whom?"
--   <BLANKLINE>
--   "Every one of whom do you think?"
--   <BLANKLINE>
--   "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . lines

Matching arbitrary output

Any lines containing only three dots (...) will match one or more lines with arbitrary content. For instance,

-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- baz

If a line contains three dots and additional content, the three dots will match anything within that line:

-- |
-- >>> putStrLn "foo bar baz"
-- foo ... baz

QuickCheck properties

Haddock (since version 2.13.0) has markup support for properties. Doctest can verify properties with QuickCheck. A simple property looks like this:

-- |
-- prop> \xs -> sort xs == (sort . sort) (xs :: [Int])

The lambda abstraction is optional and can be omitted:

-- |
-- prop> sort xs == (sort . sort) (xs :: [Int])

A complete example that uses setup code is below:

module Fib where

-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary

-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

If you see an error like the following, ensure that QuickCheck is a dependency of the test-suite or executable running doctest.

<interactive>:39:3:
    Not in scope: ‘polyQuickCheck’
    In the splice: $(polyQuickCheck (mkName "doctest_prop"))

<interactive>:39:3:
    GHC stage restriction:
      ‘polyQuickCheck’ is used in a top-level splice or annotation,
      and must be imported, not defined locally
    In the expression: polyQuickCheck (mkName "doctest_prop")
    In the splice: $(polyQuickCheck (mkName "doctest_prop"))

Hiding examples from Haddock

You can put examples into named chunks, and not refer to them in the export list. That way they will not be part of the generated Haddock documentation, but Doctest will still find them.

-- $
-- >>> 1 + 1
-- 2

Using GHC extensions

There’s two sets of GHC extensions involved when running Doctest:

  1. The set of GHC extensions that are active when compiling the module code (excluding the doctest examples). The easiest way to specify these extensions is through LANGUAGE pragmas in your source files. (Doctest will not look at your cabal file.)
  2. The set of GHC extensions that are active when executing the Doctest examples. (These are not influenced by the LANGUAGE pragmas in the file.) The recommended way to enable extensions for Doctest examples is to switch them on like this:
-- |
-- >>> :set -XTupleSections
-- >>> fst' $ (1,) 2
-- 1
fst' :: (a, b) -> a
fst' = fst

Alternatively you can pass any GHC options to Doctest, e.g.:

doctest -XCPP Foo.hs

These options will affect both the loading of the module and the execution of the Doctest examples.

If you want to omit the information which language extensions are enabled from the Doctest examples you can use the method described in Hiding examples from Haddock, e.g.:

-- $
-- >>> :set -XTupleSections

Cabal integration

Doctest provides both, an executable and a library. The library exposes a function doctest of type:

doctest :: [String] -> IO ()

Doctest’s own main is simply:

main = getArgs >>= doctest

Consequently, it is possible to create a custom executable for a project, by passing all command-line arguments that are required for that project to doctest. A simple example looks like this:

-- file doctests.hs
import Test.DocTest
main = doctest ["-isrc", "src/Main.hs"]

And a corresponding Cabal test suite section like this:

test-suite doctests
  type:          exitcode-stdio-1.0
  ghc-options:   -threaded
  main-is:       doctests.hs
  build-depends: base, doctest >= 0.8

Doctest in the wild

You can find real world examples of Doctest being used below:

Doctest extensions

Development Build Status

Join in at #hspec on freenode.

Discuss your ideas first, ideally by opening an issue on GitHub.

Add tests for new features, and make sure that the test suite passes with your changes.

cabal configure --enable-tests && cabal build && cabal exec cabal test

Contributors

  • Adam Vogt
  • Alan Zimmerman
  • Alexander Bernauer
  • Alexandre Esteves
  • Anders Persson
  • Andreas Abel
  • Ankit Ahuja
  • Artyom Kazak
  • Edward Kmett
  • Gabor Greif
  • Hiroki Hattori
  • Ignat Insarov
  • Jens Petersen
  • Joachim Breitner
  • John Chee
  • João Cristóvão
  • Julian Arni
  • Kazu Yamamoto
  • Leon Schoorl
  • Levent Erkok
  • Luke Murphy
  • Matvey Aksenov
  • Michael Orlitzky
  • Michael Snoyman
  • Mitchell Rosen
  • Nick Smallbone
  • Nikos Baxevanis
  • Oleg Grenrus
  • quasicomputational
  • Ryan Scott
  • Sakari Jokinen
  • Simon Hengel
  • Sönke Hahn
  • Takano Akio
  • Tamar Christina
  • Veronika Romashkina

For up-to-date list, query

git shortlog -s

Changes

Changes in 0.18.2

  • GHC 9.2 compatibility. (#305, thanks to Ryan Scott and Matthew Pickering)

Changes in 0.18.1

  • GHC 9.0 compatibility. (#275)

Changes in 0.18

  • Don’t use unqualified references to stderr or stdout which may collide with definitions in user code. (#201)
  • Remove support for cabal-install sandboxes. They have been obsoleted in practice by Nix-style builds in cabal-install (i.e., the v2-* commands) and stack.

Changes in 0.17

  • #266:
    • doctest now annotates its internal marker string as a String, to prevent misbehaviour in OverloadedStrings environments. This has a theoretical chance of breakage; if you’re affected, please open an issue.
    • evalEcho no longer preserves it.

Changes in 0.16.3

  • Add a cursor to highlight the differing portion between the expected and actual output. (#249)
  • GHC 8.10 compatibility. (#247, #257)

Changes in 0.16.2

  • Add doctest’s necessary-for-operation options to GHC’s command line at the end, so that they over-ride anything provided by the user. (#233)
  • Allow GHC 8.8.

Changes in 0.16.1

  • Fix loading plugins in doctests. (#224)
  • Require QuickCheck 2.13.1 or newer.
  • Remove dependency on with-location

Changes in 0.16.0.1

  • Bump bounds to allow GHC 8.6. (#210)

Changes in 0.16.0

  • Output format has changed to (hopefully) be more machine consumable. (#200)

Changes in 0.15.0

  • Add --verbose for printing each test as it is run

Changes in 0.14.1

  • Add test assets to source tarball (see #189)

Changes in 0.14.0

  • GHC 8.4 compatibility.

Changes in 0.13.0

  • Add --preserve-it for allowing the it variable to be preserved between examples

Changes in 0.12.0

  • Preserve the ‘it’ variable between examples

Changes in 0.11.4

  • Add --fast, which disables running :reload between example groups

Changes in 0.11.3

  • Add --info
  • Add --no-magic

Changes in 0.11.2

  • Make ... match zero lines

Changes in 0.11.1

  • Fix an issue with Unicode output on Windows (see #149)

Changes in 0.11.0

  • Support for GHC 8.0.1-rc2

Changes in 0.10.1

  • Automatically expand directories into contained Haskell source files (thanks @snoyberg)
  • Add cabal_macros.h and autogen dir by default (thanks @snoyberg)

Changes in 0.10.0

  • Support HASKELL_PACKAGE_SANDBOXES (thanks @snoyberg)

Changes in 0.9.13

  • Add ellipsis as wildcard

Changes in 0.9.12

  • Add support for GHC 7.10

Changes in 0.9.11

  • Defaults ambiguous type variables to Integer (#74)

Changes in 0.9.10

  • Add support for the upcoming GHC 7.8 release

Changes in 0.9.9

  • Add support for multi-line statements

Changes in 0.9.8

  • Support for GHC HEAD (7.7)

Changes in 0.9.7

  • Ignore trailing whitespace when matching example output

Changes in 0.9.6

  • Fail gracefully if GHCi is not supported (#46)

Changes in 0.9.5

  • Fix a GHC panic with GHC 7.6.1 (#41)

Changes in 0.9.4

  • Respect HASKELL_PACKAGE_SANDBOX (#39)
  • Print path to ghc on –version

Changes in 0.9.3

  • Properly handle additional object files (#38)

Changes in 0.9.2

  • Add support for QuickCheck properties

Changes in 0.9.1

  • Fix an issue with GHC 7.6.1 and type families

Changes in 0.9.0

  • Add support for setup code (see README).
  • There is no distinction between example/interaction anymore. Each expression is counted as an example in the summary.

Changes in 0.8.0

  • Doctest now directly accepts arbitrary GHC options, prefixing GHC options with –optghc is no longer necessary

Changes in 0.7.0

  • Print source location for failing tests
  • Output less clutter on failing examples
  • Expose Doctest’s functionality through a very simplistic API, which can be used for cabal integration

Changes in 0.6.1

  • Fix a parser bug with CR+LF line endings

Changes in 0.6.0

  • Support for ghc-7.4
  • Doctest now comes with it’s own parser and does not depend on Haddock anymore

Changes in 0.5.2

  • Proper handling of singular/plural when printing stats
  • Improve handling of invalid command line options

Changes in 0.5.1

  • Adapted for ghc-7.2

Changes in 0.5.0

  • Print number of interactions to stderr before running tests
  • Exit with exitFailure on failed tests
  • Improve documentation
  • Give a useful error message if ghc is not executable