shelltestrunner

Easy, repeatable testing of CLI programs/commands https://github.com/simonmichael/shelltestrunner

Version on this page:1.3.5
LTS Haskell 6.35:1.3.5
Stackage Nightly 2018-02-24:1.9
Latest on Hackage:1.9

See all snapshots shelltestrunner appears in

GPL licensed and maintained by Simon Michael

Module documentation for 1.3.5

There are no documented modules for this package.

Easy, repeatable testing of CLI programs/commands

Install | Usage | Options | Test formats | Support/Contribute | Credits

shelltestrunner (executable: shelltest) is a portable command-line tool for testing command-line programs, or general shell commands, released under GPLv3+. It reads simple test specifications defining a command to run, some input, and the expected output, stderr, and exit status. It can run tests in parallel, selectively, with a timeout, in color, etc. Projects using it include hledger, Agda, and berp.

Install

There may be a new-enough packaged version on your platform. Eg:

||| |----------------|--------------------------------------- | Debian/Ubuntu: | apt install shelltestrunner | Gentoo: | emerge shelltestrunner

Or, build the latest release on any major platform:

||| |----------------|--------------------------------------- | stack: | get stack, stack install shelltestrunner-1.9 | cabal: | cabal update; cabal install shelltestrunner-1.9

Usage

Here's a minimal test file containing one shell test: <!-- keep synced with tests/examples -->

# A comment. Testing bash's builtin "echo" command (if /bin/sh is bash)
echo
>>>= 0

They're called "shell test" because any shell (/bin/sh on POSIX, CMD on Windows) command line can be tested. Each test begins with the command to test, followed by optional stdin input, expected stdout and/or stderr output, and ends with the expected exit status. Here's another file containing two tests:

# Test that the "cat" program copies its input to stdout, 
# nothing appears on stderr, and exit status is 0.
cat
<<<
foo
>>>
foo
>>>2
>>>= 0

# Test that cat prints an error containing "unrecognized option" or
# "illegal option" and exits with non-zero status if given a bad flag.
cat --no-such-flag
>>>2 /(unrecognized|illegal) option/
>>>= !0

To run these tests:

$ shelltest echo.test cat.test
:echo.test: [OK]
:cat.test:1: [OK]
:cat.test:2: [OK]

         Test Cases  Total      
 Passed  3           3          
 Failed  0           0          
 Total   3           3          

That's the basics! There are also some alternate test formats you'll read about below.

Options

$ shelltest --help
shelltest 1.9

shelltest [OPTIONS] [TESTFILES|TESTDIRS]

Common flags:
  -l --list             List all parsed tests and stop
  -a --all              Don't truncate output, even if large
  -c --color            Show colored output if your terminal supports it
  -d --diff             Show expected output mismatches in diff format
  -p --precise          Show expected/actual output precisely (eg whitespace)
  -h --hide-successes   Show only test failures
     --xmlout=FILE      Specify file to store test results in xml format.
  -D --defmacro=D=DEF   Specify a macro that is evaluated by preprocessor
                        before the test files are parsed. D stands for macro
                        definition that is replaced with the value of DEF.
  -i --include=PAT      Include tests whose name contains this glob pattern
  -x --exclude=STR      Exclude test files whose path contains STR
     --execdir          Run tests from within the test file's directory
     --extension=EXT    File suffix of test files (default: .test)
  -w --with=EXECUTABLE  Replace the first word of (unindented) test commands
  -o --timeout=SECS     Number of seconds a test may run (default: no limit)
  -j --threads=N        Number of threads for running tests (default: 1)
     --debug            Show debug info, for troubleshooting
     --debug-parse      Show test file parsing info and stop
  -? --help             Display help message
  -V --version          Print version information
     --numeric-version  Print just the version number

shelltest accepts one or more test file or directory arguments. A directory means all files below it named *.test (customisable with --extension).

Test commands are run with /bin/sh on POSIX systems and with CMD on Windows. By default, they are run in the directory in which you ran shelltest; with --execdir they will run in each test file's directory instead.

--include selects only tests whose name (file name plus intra-file sequence number) matches a .gitignore-style pattern, while --exclude skips tests based on their file path. These can be used eg to focus on a particular test, or to skip tests intended for a different platform.

-D/--defmacro defines a macro that is replaced by preprocessor before any tests are parsed and run.

-w/--with replaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Commands which have been prefixed by an extra space will not be affected by this option.

--hide-successes gives quieter output, reporting only failed tests.

Long flags can be abbreviated to a unique prefix.

For example, the command:

$ shelltest tests -i args -c -j8 -o1 -DCONF_FILE=test/myconf.cfq --hide
  • runs the tests defined in any *.test file in or below the tests/ directory
  • whose names contain "args"
  • in colour if possible
  • with up to 8 tests running in parallel
  • allowing no more than 1 second for each test
  • replacing the text "CONF_FILE" in all tests with "test/myconf.cfq"
  • reporting only the failures.

Test formats

shelltestrunner 1.9 adds some experimental new test file formats, described below. These need more real-world testing and may evolve further, but they will remain supported or will have a migration path.

| Format name | Description | Delimiters, in order | |-------------|------------------------------------------------------------------------------|----------------------------| | format 1 | command first, exit status is required | (none) <<< >>> >>>2 >>>= | | format 2 | input first, can be reused by multiple tests, some delimiters can be omitted | <<< $$$ >>> >>>2 >>>= | | format 3 | like format 2, but with shorter delimiters | < $ > >2 >= |

shelltestrunner tries to parse each file first with format 2, then format 3, then format 1. All tests within a file should use the same format. I suggest choosing format 3 (short delimiters), switching to format 2 when longer delimiters are needed.

Format 1

Test files contain one or more individual tests, each consisting of a one-line shell command, optional input, expected standard output and/or error output, and a (required) exit status.

# COMMENTS OR BLANK LINES
COMMAND LINE
<<<
INPUT
>>>
EXPECTED OUTPUT (OR >>> /REGEXP/)
>>>2
EXPECTED STDERR (OR >>>2 /REGEXP/)
>>>= EXPECTED EXIT STATUS (OR >>>= /REGEXP/)

When not specified, stdout/stderr are ignored. A space before the command protects it from -w/--with.

Examples: above, shelltestrunner, hledger, berp, cblrepo.

Format 2

(shelltestrunner 1.9+) This improves on format 1 in two ways: it allows tests to reuse the same input, and it allows delimiters to often be omitted.

Test files contain one or more test groups. A test group consists of some optional standard input and one or more tests. Each test is a one-line shell command followed by optional expected standard output, error output and/or numeric exit status, separated by delimiters.

# COMMENTS OR BLANK LINES
<<<
INPUT
$$$ COMMAND LINE
>>>
EXPECTED OUTPUT (OR >>> /REGEX/)
>>>2
EXPECTED STDERR (OR >>>2 /REGEX/)
>>>= EXPECTED EXIT STATUS (OR >>>= /REGEX/ OR >>>=)
# COMMENTS OR BLANK LINES
ADDITIONAL TESTS FOR THIS INPUT
ADDITIONAL TEST GROUPS WITH DIFFERENT INPUT

All test parts are optional except the command line. If not specified, stdout and stderr are expected to be empty and exit status is expected to be zero.

Two spaces between $$$ and the command protects it from -w/--with.

The <<< delimiter is optional for the first input in a file. Without it, input begins at the first non-blank/comment line. Input ends at the $$$ delimiter. You can't put a comment before the first $$$.

The >>> delimiter is optional except when matching via regex. Expected output/stderr extends to the next >>>2 or >>>= if present, or to the last non-blank/comment line before the next <<< or $$$ or file end. /REGEX/ regular expression patterns may be used instead of specifying the expected output in full. The regex syntax is regex-tdfa's, plus you can put ! before /REGEX/ to negate the match.

The exit status is a number, normally 0 for a successful exit. This too can be prefixed with ! to negate the match, or you can use a /REGEX/ pattern. A >>>= with nothing after it ignores the exit status.

Examples: <!-- keep synced with tests/examples -->

All delimiters explicit:

# cat copies its input to stdout
<<<
foo
$$$ cat
>>>
foo

# or, given a bad flag, prints a platform-specific error and exits with non-zero status
$$$ cat --no-such-flag
>>>2 /(unrecognized|illegal) option/
>>>= !0

# echo ignores the input and prints a newline.
# We need the >>>= (or a >>>2) to delimit the whitespace which
# would otherwise be ignored.
$$$ echo
>>>

>>>=

Non-required <<< and >>> delimiters omitted:

foo
$$$ cat
foo

$$$ cat --no-such-flag
>>>2 /(unrecognized|illegal) option/
>>>= !0

$$$ echo

>>>=

Format 3

(shelltestrunner 1.9+) The same as format 2, but with more convenient short delimiters: < $ > >2 >=.

# COMMENTS OR BLANK LINES
<
INPUT
$ COMMAND LINE
>
EXPECTED OUTPUT (OR > /REGEX/)
>2
EXPECTED STDERR (OR >2 /REGEX/)
>= EXPECTED EXIT STATUS (OR >= /REGEX/ OR >=)
# COMMENTS OR BLANK LINES
ADDITIONAL TESTS FOR THIS INPUT
ADDITIONAL TEST GROUPS WITH DIFFERENT INPUT

Examples: <!-- keep synced with tests/examples -->

All delimiters explicit:

# cat copies its input to stdout
<
foo
$ cat
>
foo

# or, given a bad flag, prints a platform-specific error and exits with non-zero status
$ cat --no-such-flag
>2 /(unrecognized|illegal) option/
>= !0

# echo ignores the input and prints a newline.
# We use an explicit >= (or >2) to delimit the whitespace which
# would otherwise be ignored.
$ echo
>

>=

Non-required < and > delimiters omitted:

foo
$ cat
foo

$ cat --no-such-flag
>2 /(unrecognized|illegal) option/
>= !0

$ echo
>

>2

shelltestrunner

Support/Contribute

||| |----------------------|--------------------------------------------------| | Released version: | http://hackage.haskell.org/package/shelltestrunner | Changelog: | http://hackage.haskell.org/package/shelltestrunner/changelog | Code | https://github.com/simonmichael/shelltestrunner | Issues | https://github.com/simonmichael/shelltestrunner/issues | IRC | http://webchat.freenode.net?channels=shelltestrunner <!-- or /msg sm -->

2012 user survey.

Feedback, testing, code, documentation, packaging, blogging, and funding are most welcome.

Credits

Simon Michael wrote shelltestrunner, inspired by John Wiegley's tests for Ledger.

Code contributors include: Taavi Väljaots, John Macfarlane, Andrés Sicard-Ramírez, Iustin Pop, Trygve Laugstøl, Bernie Pope, Sergei Trofimovich, John Chee.

shelltestrunner depends on several fine libraries, in particular Max Bolingbroke's test-framework, and of course on the Glorious Haskell Compiler.

The Blade Runner font is by Phil Steinschneider.

Changes

1.9 (2018/1/14)

* two new test file formats have been added, allowing input re-use and lighter syntax
* new -l/--list flag lists the tests found
* new -D/--defmacro option allows text substitution (Taavi Valjaots)
* new --xmlout option saves test results as xml (Taavi Valjaots)
* tests with Windows line endings now also work on unix (Taavi Valjaots)
* shelltestrunner's tests should now pass on Windows (Taavi Valjaots)
* flags formerly passed through to test-framework are now built in
* >>>= with nothing after it now matches any exit status
* failure messages now show the test command (John Chee)
* include shelltestrunner's tests in cabal sdist archive (Iustin Pop)
* build with latest deps and stackage resolvers
* shelltestrunner's code and home page have moved to github

1.3.5 (2015/3/30)

* fix Applicative warning with ghc 7.10
* allow utf8-string <1.1

1.3.4 (2014/5/28)

* drop cabal-file-th, support GHC 7.8.2

1.3.3 (2014/5/25)

* allow process 1.2, regex-tdfa-1.2
* add a hackage-compatible changelog

1.3.2 (2013/11/13)

* increase upper bound on Diff package

1.3.1 (2012/12/28)

* fix cabal file typo breaking the build

1.3 (2012/12/28)

* support latest Diff, cmdargs, test-framework; tested with GHC 7.6.1 (Magnus Therning)

* fix unicode handling on GHC >= 7.2

1.2.1 (2012/3/12)

* use the more up-to-date filemanip package for easier Debian packaging

1.2 (2012/2/26)

* support latest cmdargs, test-framework, and GHC 7.4
* more readable non-quoted failure output by default; for quoted output, use -p/--precise
* the --all, --diff and --precise options now interact well

1.1 (2011/8/25)

* bump process dependency to allow building with GHC 7.2.1
* new -a/--all flag shows all failure output without truncating

1.0 (2011/7/23)

* New home page/docs
* The >>>= field is now required; you may need to add it to your existing tests
* Input and expected output can now contain lines beginning with #
* Multiple tests in a file may now have whitespace between them
* The -i/--implicit option has been dropped
* New -d/--diff option shows test failures as a unified diff when possible, including line numbers to help locate the problem
* New -x/--exclude option skips certain test files (eg platform-specific ones)
* Passing arguments through to test-framework is now more robust
* Fixed: parsing could fail when input contained left angle brackets
* Fixed: some test files generated an extra blank test at the end

0.9 (2010/9/3)

* show plain non-ansi output by default, add --color option
* better handling of non-ascii test data. We assume that non-ascii file
paths, command-line arguments etc. are UTF-8 encoded on unix systems
(cf http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html),
and that GHC 6.12 or greater is used. Then:
- non-ascii test file paths should render correctly, eg in failure messages
- non-ascii test commands should run correctly
- non-ascii expected output should match correctly
- non-ascii regular expressions should match correctly. (Caveat: not
thoroughly tested, this may break certain regexps, )
* use regex-tdfa instead of pcre-light for better windows compatibility
To avoid a memory leak in current regex-tdfa, only regular expressions
up to 300 characters in size are supported. Also, DOTALL is no longer
enabled and probably fewer regexp constructs are supported. There are
still issues on windows/wine but in theory this will help.
* tighten up dependencies

0.8 (2010/4/9)

* rename executable to shelltest. The package might also be renamed at some point.
* better built-in help
* shell tests now include a full command line, making them more readable
and self-contained. The --with option can be used to replace the first
word with something else, unless the test command line begins with a
space.
* we also accept directory arguments, searching for test files below
them, with two new options:
--execdir execute tested command in same directory as test file
--extension=EXT file extension of test files (default=.test)

0.7 (2010/3/5)

* more robust parsing
- --debug-parse parses test files and stops
- regexps now support escaped forward slash (\/)
- bad regexps now fail at startup
- command-line arguments are required in a test, and may be blank
- a >>>= is no longer required to separate multiple tests in a file
- comments can be appended to delimiter lines
- comments can appear at end of file
- files need not have a final newline
- files containing nothing, all comments, or valid tests are allowed; anything else is rejected
- somewhat better errors
- allow indented input
* support negative (-) and negatively-matched (!) numeric exit codes
* let . in regexps match newline
* warn but continue when a test file fails to parse
* output cleanups, trim large output
* more flexible --implicit flag
* switch to the more robust and faster pcre-light regexp lib

0.6 (2009/7/15)

* allow multiple tests per file, handle bad executable better

0.5 (2009/7/14)

* show failure output in proper order

0.4 (2009/7/14)

* run commands in a more robust way to avoid hangs
This fixes hanging when a command generates large output, and hopefully
all other deadlocks. The output is consumed strictly. Thanks to Ganesh
Sittampalam for his help with this.
* --implicit-tests flag providing implicit tests for omitted fields
* --debug flag
* regular expression matching
* disallow interspersed foreign options which confused parseargs
* change comment character to #

0.3 (2009/7/11)

* misc. bugfixes/improvements

0.2 (2009/7/10)

* bugfix, build with -threaded

0.1 (2009/7/10)

* shelltestrunner, a generic shell command stdout/stderr/exit status tester
comments powered byDisqus