megaparsec
Monadic parser combinators
https://github.com/mrkkrp/megaparsec
LTS Haskell 22.37: | 9.5.0@rev:4 |
Stackage Nightly 2024-10-10: | 9.6.1 |
Latest on Hackage: | 9.6.1 |
megaparsec-9.6.1@sha256:8d8f8ee5aca5d5c16aa4219afd13687ceab8be640f40ba179359f2b42a628241,3323
Module documentation for 9.6.1
- Text
Megaparsec
- Features
- Documentation
- Tutorials
- Performance
- Comparison with other solutions
- Related packages
- Prominent projects that use Megaparsec
- Links to announcements and blog posts
- Contribution
- License
This is an industrial-strength monadic parser combinator library. Megaparsec is a feature-rich package that tries to find a nice balance between speed, flexibility, and quality of parse errors.
Features
The project provides flexible solutions to satisfy common parsing needs. The section describes them shortly. If you’re looking for comprehensive documentation, see the section about documentation.
Core features
The package is built around MonadParsec
, an MTL-style monad transformer.
Most features work with all instances of MonadParsec
. One can achieve
various effects combining monad transformers, i.e. building a monadic stack.
Since the common monad transformers like WriterT
, StateT
, ReaderT
and
others are instances of the MonadParsec
type class, one can also wrap
ParsecT
in these monads, achieving, for example, backtracking state.
On the other hand ParsecT
is an instance of many type classes as well. The
most useful ones are Monad
, Applicative
, Alternative
, and
MonadParsec
.
Megaparsec includes all functionality that is typically available in Parsec-like libraries and also features some special combinators:
parseError
allows us to end parsing and report an arbitrary parse error.withRecovery
can be used to recover from parse errors “on-the-fly” and continue parsing. Once parsing is finished, several parse errors may be reported or ignored altogether.observing
makes it possible to “observe” parse errors without ending parsing.
In addition to that, Megaparsec features high-performance combinators similar to those found in Attoparsec:
tokens
makes it easy to parse several tokens in a row (string
andstring'
are built on top of this primitive). This is about 100 times faster than matching a string token by token.tokens
returns “chunk” of original input, meaning that if you parseText
, it’ll returnText
without repacking.takeWhile
andtakeWhile1
are about 150 times faster than approaches involvingmany
,manyTill
and other similar combinators.takeP
allows us to grab n tokens from the stream and returns them as a “chunk” of the stream.
Megaparsec is about as fast as Attoparsec if you write your parser carefully (see also the section about performance).
The library can currently work with the following types of input stream out-of-the-box:
String = [Char]
ByteString
(strict and lazy)Text
(strict and lazy)
It’s also possible to make it work with custom token streams by making them
an instance of the Stream
type class.
Error messages
-
Megaparsec has typed error messages and the ability to signal custom parse errors that better suit the user’s domain of interest.
-
Since version 8, the location of parse errors can independent of current offset in the input stream. It is useful when you want a parse error to point to a particular position after performing some checks.
-
Instead of a single parse error Megaparsec produces so-called
ParseErrorBundle
data type that helps to manage multi-error messages and pretty-print them. Since version 8, reporting multiple parse errors at once has become easier.
External lexers
Megaparsec works well with streams of tokens produced by tools like Alex.
The design of the Stream
type class has been changed significantly in the
recent versions, but user can still work with custom streams of tokens.
Character and binary parsing
Megaparsec has decent support for Unicode-aware character parsing. Functions
for character parsing live in the Text.Megaparsec.Char
module.
Similarly, there is Text.Megaparsec.Byte
module for parsing
streams of bytes.
Lexer
Text.Megaparsec.Char.Lexer
is a module that should help
you write your lexer. If you have used Parsec
in the past, this module
“fixes” its particularly inflexible Text.Parsec.Token
.
Text.Megaparsec.Char.Lexer
is intended to be imported
using a qualified import, it’s not included in Text.Megaparsec
. The
module doesn’t impose how you should write your parser, but certain
approaches may be more elegant than others. An especially important theme is
parsing of white space, comments, and indentation.
The design of the module allows one quickly solve simple tasks and doesn’t get in the way when the need to implement something less standard arises.
Text.Megaparsec.Byte.Lexer
is also available for users
who wish to parse binary data.
Documentation
Megaparsec is well-documented. See the current version of Megaparsec documentation on Hackage.
Tutorials
You can find the most complete Megaparsec tutorial here. It should provide sufficient guidance to help you start with your parsing tasks.
Performance
Despite being flexible, Megaparsec is also fast. Here is how Megaparsec compares to Attoparsec (the fastest widely used parsing library in the Haskell ecosystem):
Test case | Execution time | Allocated | Max residency |
---|---|---|---|
CSV (Attoparsec) | 76.50 μs | 397,784 | 10,544 |
CSV (Megaparsec) | 64.69 μs | 352,408 | 9,104 |
Log (Attoparsec) | 302.8 μs | 1,150,032 | 10,912 |
Log (Megaparsec) | 337.8 μs | 1,246,496 | 10,912 |
JSON (Attoparsec) | 18.20 μs | 128,368 | 9,032 |
JSON (Megaparsec) | 25.45 μs | 203,824 | 9,176 |
You can run the benchmarks yourself by executing:
$ nix-build -A benches.parsers-bench
$ cd result/bench
$ ./bench-memory
$ ./bench-speed
More information about benchmarking and development can be found here.
Comparison with other solutions
There are quite a few libraries that can be used for parsing in Haskell, let’s compare Megaparsec with some of them.
Megaparsec vs Attoparsec
Attoparsec is another prominent Haskell library for parsing. Although both libraries deal with parsing, it’s usually easy to decide which you will need in particular project:
-
Attoparsec is sometimes faster but not that feature-rich. It should be used when you want to process large amounts of data where performance matters more than quality of error messages.
-
Megaparsec is good for parsing of source code or other human-readable texts. It has better error messages and it’s implemented as a monad transformer.
So, if you work with something human-readable where the size of input data is moderate, it makes sense to go with Megaparsec, otherwise Attoparsec may be a better choice.
Megaparsec vs Parsec
Since Megaparsec is a fork of Parsec, we are bound to list the main differences between the two libraries:
-
Better error messages. Megaparsec has typed error messages and custom error messages, it can also report multiple parse errors at once.
-
Megaparsec can show the line on which parse error happened as part of parse error. This makes it a lot easier to figure out where the error happened.
-
Some quirks and bugs of Parsec are fixed.
-
Better support for Unicode parsing in
Text.Megaparsec.Char
. -
Megaparsec has more powerful combinators and can parse languages where indentation matters.
-
Better documentation.
-
Megaparsec can recover from parse errors “on the fly” and continue parsing.
-
Megaparsec allows us to conditionally process parse errors inside a running parser. In particular, it’s possible to define regions in which parse errors, should they happen, will get a “context tag”, e.g. we could build a context stack like “in function definition foo”, “in expression x”, etc.
-
Megaparsec is faster and supports efficient operations
tokens
,takeWhileP
,takeWhile1P
,takeP
, like Attoparsec.
If you want to see a detailed change log, CHANGELOG.md
may be helpful.
Also see this original announcement for another
comparison.
Megaparsec vs Trifecta
Trifecta is another Haskell library featuring good error messages. These are the common reasons why Trifecta may be problematic to use:
-
Complicated, doesn’t have any tutorials available, and documentation doesn’t help much.
-
Trifecta can parse
String
andByteString
natively, but notText
. -
Depends on
lens
, which is a very heavy dependency. If you’re not intolens
, you may not like the API.
Idris has switched from Trifecta to Megaparsec which allowed it to have better error messages and fewer dependencies.
Megaparsec vs Earley
Earley is a newer library that allows us to safely parse context-free grammars (CFG). Megaparsec is a lower-level library compared to Earley, but there are still enough reasons to choose it:
-
Megaparsec is faster.
-
Your grammar may be not context-free or you may want introduce some sort of state to the parsing process. Almost all non-trivial parsers require state. Even if your grammar is context-free, state may allow for additional niceties. Earley does not support that.
-
Megaparsec’s error messages are more flexible allowing to include arbitrary data in them, return multiple error messages, mark regions that affect any error that happens in those regions, etc.
In other words, Megaparsec is less safe but also more powerful.
Related packages
The following packages are designed to be used with Megaparsec (open a PR if you want to add something to the list):
hspec-megaparsec
—utilities for testing Megaparsec parsers with with Hspec.replace-megaparsec
—Stream editing and find-and-replace with Megaparsec.cassava-megaparsec
—Megaparsec parser of CSV files that plays nicely with Cassava.tagsoup-megaparsec
—a library for easily using TagSoup as a token type in Megaparsec.parser-combinators
—provides permutation and expression parsers previously bundled with Megaparsec.faster-megaparsec
—speeds up parsing by trying a simpleMonadParsec
instance and falls back toParsecT
to report errors.
Prominent projects that use Megaparsec
Some prominent projects that use Megaparsec:
- Idris—a general-purpose functional programming language with dependent types
- Dhall—an advanced configuration language
- hnix—re-implementation of the Nix language in Haskell
- Hledger—an accounting tool
- MMark—strict markdown processor for writers
Links to announcements and blog posts
Here are some blog posts mainly announcing new features of the project and describing what sort of things are now possible:
- Megaparsec 8
- Megaparsec 7
- Evolution of error messages
- A major upgrade to Megaparsec: more speed, more power
- Latest additions to Megaparsec
- Announcing Megaparsec 5
- Megaparsec 4 and 5
- The original Megaparsec 4.0.0 announcement
Contribution
Issues (bugs, feature requests or otherwise feedback) may be reported in the GitHub issue tracker for this project.
Pull requests are also welcome. If you would like to contribute to the project, you may find this document helpful.
License
Copyright © 2015–present Megaparsec contributors
Copyright © 2007 Paolo Martini
Copyright © 1999–2000 Daan Leijen
Distributed under FreeBSD license.
Changes
Megaparsec follows SemVer.
Megaparsec 9.6.1
- Exposed
Text.Megaparsec.State
, so that the new functions (initialState
andinitialPosState
) can be actually imported from it. PR 549.
Megaparsec 9.6.0
- Added the functions
initialState
andinitialPosState
toText.Megaparsec.State
. Issue 449.
Megaparsec 9.5.0
-
Dropped a number of redundant constraints here and there. PR 523.
-
Added a
MonadWriter
instance forParsecT
. PR 534.
Megaparsec 9.4.1
- Removed
Monad m
constraints in several places where they were introduced in 9.4.0. Issue 532.
Megaparsec 9.4.0
-
dbg
now prints hints among other debug information. PR 530. -
Hints are no longer lost in certain methods of MTL instances for
ParsecT
. Issue 528. -
Added a new method to the
MonadParsec
type class—mkParsec
. This can be used to construct “new primitives” with arbitrary behavior at the expense of having to dive into Megaparsec’s internals. PR 514.
Megaparsec 9.3.1
- Fixed a bug related to processing of tabs when error messages are rendered. Issue 524.
Megaparsec 9.3.0
-
Now
label
can override more than one group of hints in the parser it wraps. Issue 482. -
takeP n
now returns the empty chunk of the input stream whenn
is negative, similar to whenn == 0
. Issue 497. -
Added the
MonadParsecDbg
type class inText.Megaparsec.Debug
. The type class allows us to usedbg
in MTL monad transformers. Issue 488. -
Introduced the
ShareInput
andNoShareInput
newtype wrappers inText.Megaparsec.Stream
in order to allow the user to choose how the input should be sliced and shared during the parsing. Issue 492.
Megaparsec 9.2.2
- Fixed a space leak in the implementations of the
reachOffset
andreachOffsetNoLine
methods ofTraversableStream
. Issue 486.
Megaparsec 9.2.1
- Builds with
mtl-2.3
andtransformers-0.6
.
Megaparsec 9.2.0
- Added parsers for binary representations (little/big endian) of numbers in
Text.Megaparsec.Byte.Binary
.
Megaparsec 9.1.0
-
Added
dbg'
inText.Megaparsec.Debug
for debugging parsers that have unshowable return values. -
Documentation improvements.
Megaparsec 9.0.1
- Added Safe Haskell support.
Megaparsec 9.0.0
-
Split the
Stream
type class. The methodsshowTokens
andtokensLength
have been put into a separate type classVisualStream
, whilereachOffset
andreachOffsetNoLine
are now inTraversableStream
. This should make definingStream
instances for custom streams easier. -
Defined
Stream
instances for lists andSeq
s. -
Added the functions
hspace
andhspace1
to theText.Megaparsec.Char
andText.Megaparsec.Byte
modules.
Megaparsec 8.0.0
-
The methods
failure
andfancyFailure
ofMonadParsec
are now ordinary functions and live inText.Megaparsec
. They are defined in terms of the newparseError
method ofMonadParsec
. This method allows us to signal parse errors at a given offset without manipulating parser state manually. -
Megaparsec now supports registration of “delayed” parse errors. On lower level we added a new field called
stateParseErrors
to theState
record. The type also had to change fromState s
toState s e
. This field contains the list of registeredParseErrors
that do not end parsing immediately but still will cause failure in the end if the list is not empty. Users are expected to register parse errors using the three functions:registerParseError
,registerFailure
, andregisterFancyFailure
. These functions are analogous to those without theregister
prefix, except that they have “delayed” effect. -
Added the
tokensLength
method to theStream
type class to improve support for custom input streams. -
Added the
setErrorOffset
function to set offset ofParseError
s. -
Changed type signatures of
reachOffset
andreachOffsetNoLine
methods of theStream
type class. Instead of three-tuplereachOffset
now returns two-tuple becauseSourcePos
is already contained in the returnedPosState
record. -
Generalized
decimal
,binary
,octal
, andhexadecimal
parsers in lexer modules so that theyNum
instead of justIntegral
. -
Dropped support for GHC 8.2.x and older.
Megaparsec 7.0.5
-
Dropped support for GHC 7.10.
-
Adapted the code to
MonadFail
changes inbase-4.13
. -
Separated the test suite into its own package. The reason is that we can avoid circular dependency on
hspec-megaparsec
and thus avoid keeping copies of its source files in our test suite, as we had to do before. Another benefit is that we can export some auxiliary functions inmegaparsec-tests
which can be used by other test suites, for example in theparser-combinators-tests
package.Version of
megaparsec-tests
will be kept in sync with versions ofmegaparsec
from now on.
Megaparsec 7.0.4
- Numerous documentation corrections.
Megaparsec 7.0.3
- Fixed the build with
mtl
older than2.2.2
.
Megaparsec 7.0.2
-
Fixed the property test for
char'
which was failing in the case when there is a character with different upper and title cases. -
More descriptive error messages when
elabel
orulabel
fromText.Megaparsec.Error.Builder
are used with empty strings. -
Typo fixes in the docs.
Megaparsec 7.0.1
- Fixed a bug in
errorBundlePretty
. Previously the question sign?
was erroneously inserted before offending line in 2nd and later parse errors.
Megaparsec 7.0.0
General
-
Dropped the
Text.Megaparsec.Perm
module. UseControl.Applicative.Permutations
fromparser-combinators
instead. -
Dropped the
Text.Megaparsec.Expr
module. UseControl.Monad.Combinators.Expr
fromparser-combinators
instead. -
The debugging function
dbg
has been moved fromText.Megaparsec
to its own moduleText.Megaparsec.Debug
. -
Dropped support for GHC 7.8.
Combinators
-
Moved some general combinators from
Text.Megaparsec.Char
andText.Megaparsec.Byte
toText.Megaparsec
, renaming some of them for clarity.Practical consequences:
-
Now there is the
single
combinator that is a generalization ofchar
for arbitrary streams.Text.Megaparsec.Char
andText.Megaparsec.Byte
still containchar
as type-constrained versions ofsingle
. -
Similarly, now there is the
chunk
combinator that is a generalization ofstring
for arbitrary streams. Thestring
combinator is still re-exported fromText.Megaparsec.Char
andText.Megaparsec.Byte
for compatibility. -
satisfy
does not depend on type of token, and so it now lives inText.Megaparsec
. -
anyChar
was renamed toanySingle
and moved toText.Megaparsec
. -
notChar
was renamed toanySingleBut
and moved toText.Megaparsec
. -
oneOf
andnoneOf
were moved toText.Megaparsec
.
-
-
Simplified the type of the
token
primitive. It now takes just a matching functionToken s -> Maybe a
as the first argument and the collection of expected itemsSet (ErrorItem (Token s))
as the second argument. This makes sense because the collection of expected items cannot depend on what we see in the input stream. -
The
label
primitive now doesn’t prepend the phrase “the rest of” to the label when its inner parser produces hints after consuming input. In that caselabel
has no effect. -
Fixed the
Text.Megaparsec.Char.Lexer.charLiteral
so it can accept longer escape sequences (max length is now 10). -
Added the
binDigitChar
functions inText.Megaparsec.Byte
andText.Megaparsec.Char
. -
Added the
binary
functions inText.Megaparsec.Byte.Lexer
andText.Megaparsec.Char.Lexer
. -
Improved case-insensitive character matching in the cases when e.g.
isLower
andisUpper
both returnFalse
. Functions affected:Text.Megaparsec.Char.char'
. -
Renamed
getPosition
togetSourcePos
. -
Renamed
getTokensProcessed
togetOffset
,setTokensProcessed
tosetOffset
. -
Dropped
getTabWidth
andsetTabWidth
because tab width is irrelevant to parsing process now, it’s only relevant for pretty-printing of parse errors, which is handled separately. -
Added and
withParsecT
inText.Megaparsec.Internal
to allow changing the type of the custom data component in parse errors.
Parser state and input stream
-
Dropped stacks of source positions. Accordingly, the functions
pushPosition
andpopPosition
fromText.Megaparsec
andsourcePosStackPretty
fromText.Megaparsec.Error
were removed. The reason for this simplification is that I could not find any code that uses the feature and it makes manipulation of source positions hairy. -
Introduced
PosState
for calculatingSourcePos
from offsets and getting offending line for displaying on pretty-printing of parse errors. It’s now contained in bothState
andParseErrorBundle
. -
Dropped
positionAt1
,positionAtN
,advance1
, andadvanceN
methods fromStream
. They are no longer necessary becausereachOffset
(and its specialized versionreachOffsetNoLine
) takes care ofSourcePos
calculation.
Parse errors
-
ParseError
now contains raw offset in input stream instead ofSourcePos
.errorPos
was dropped fromText.Megaparsec.Error
. -
ParseError
is now parametrized over stream types
instead of token typet
. -
Introduced
ParseErrorBundle
which contains one or moreParseError
equipped with all information that is necessary to pretty-print them together with offending lines from the input stream. Functions likerunParser
now returnParseErrorBundle
instead of plainParseError
.By default there will be only one
ParseError
in such a bundle, but it’s possible to add more parse errors to a bundle manually. During pretty-printing, the input stream will be traversed only once. -
The primary function for pretty-printing of parse errors—
errorBundlePretty
always prints offending lines now.parseErrorPretty
is still there, but it probably won’t see a lot of use from now on.parseErrorPretty'
andparseErrorPretty_
were removed.parseTest'
was removed becauseparseTest
always prints offending lines now. -
Added
attachSourcePos
function inText.Megaparsec.Error
. -
The
ShowToken
type class has been removed and its methodshowTokens
now lives in theStream
type class. -
The
LineToken
type class is no longer necessary because the new methodreachOffset
of the type classStream
does its job. -
In
Text.Megaparsec.Error
the following functions were added:mapParseError
,errorOffset
. -
Implemented continuous highlighting in parse errors. For this we added the
errorComponentLen
method to theShowErrorComponent
type class.
Parse error builder
-
The functions
err
anderrFancy
now accept offsets at which the parse errors are expected to have happened, i.e.Int
s. ThusposI
andposN
are no longer necessary and were removed. -
ET
is now parametrized over the type of streams
instead of token typet
. -
Combinators like
utoks
andetoks
now accept chunks of input stream directly, i.e.Tokens s
instead of[Token s]
which should be more natural and convenient.
Megaparsec 6.5.0
- Added
Text.Megaparsec.Internal
, which exposes some internal data structures and data constructor ofParsecT
.
Megaparsec 6.4.1
scientific
now correctly backtracks after attempting to parse fractional and exponent parts of a number.float
correctly backtracks after attempting to parse optional exponent part (when it comes after fractional part, otherwise it’s obligatory).
Megaparsec 6.4.0
-
Text.Megaparsec
now re-exportsControl.Monad.Combinators
instead ofControl.Applicative.Combinators
fromparser-combinators
because the monadic counterparts of the familiar combinators are more efficient and not as leaky.This may cause minor breakage in certain cases:
-
You import
Control.Applicative
and in that case there will be a name conflict betweenControl.Applicative.many
andControl.Monad.Combinator.many
now (the same forsome
). -
You define a polymorphic helper in terms of combinator(s) from
Control.Applicative.Combinators
and useApplicative
orAlternative
constraint. In this case you’ll have to adjust the constraint to beMonad
orMonadPlus
respectively.
Also note that the new
Control.Monad.Combinators
module we re-export now re-exportsempty
fromControl.Applicative
. -
-
Fix the
atEnd
parser. It now does not produce hints, so when you use it, it won’t contribute to the “expecting end of input” component of parse error.
Megaparsec 6.3.0
-
Added an
IsString
instance forParsecT
. Now it is possible to write"abc"
rather thanstring "abc"
. -
Added the
customFailure
combinator, which is a special case offancyFailure
. -
Made implementation of
sconcat
andmconcat
ofParsecT
more efficient.
Megaparsec 6.2.0
float
inText.Megaparsec.Char.Lexer
andText.Megaparsec.Byte.Lexer
now does not accept plain integers. This is the behavior we had in version 5 of the library.
Megaparsec 6.1.1
- Fixed the bug when
tokens
usedcok
continuation even when matching an empty chunk. Now it correctly useseok
in this case.
Megaparsec 6.1.0
-
Improved rendering of offending line in
parseErrorPretty'
in the presence of tab characters. -
Added
parseErrorPretty_
, which is just likeparseErrorPretty'
but allows to specify tab width to use. -
Adjusted hint generation so when we backtrack a consuming parser with
try
, we do not create hints from its parse error (because it’s further in input stream!). This was a quite subtle bug that stayed unnoticed for several years apparently.
Megaparsec 6.0.2
- Allow
parser-combinators-0.2.0
.
Megaparsec 6.0.1
-
Fixed a typo in
README.md
. -
Added some text that clarifies how to parametrize the
ParseError
type.
Megaparsec 6.0.0
General
-
Re-organized the module hierarchy. Some modules such as
Text.Megaparsec.Prim
do not exist anymore. Stream definitions were moved toText.Megaparsec.Stream
. Generic combinators are now re-exported from theControl.Applicative.Combinators
from the packageparser-combinators
. Just importText.Megaparsec
and you should be OK. AddText.Megaparsec.Char
if you are working with a stream ofChar
s orText.Megaparsec.Byte
if you intend to parse binary data, then add qualified modules you need (permutation parsing, lexing, expression parsing, etc.).Text.Megaparsec.Lexer
was renamed toText.Megaparsec.Char.Lexer
because many functions in it has theToken s ~ Char
constraint. There is alsoText.Megaparsec.Byte.Lexer
now, although it has fewer functions. -
Dropped per-stream modules, the
Parser
type synonym is to be defined manually by user. -
Added a
MonadFix
instance forParsecT
. -
More lightweight dependency tree, dropped
exceptions
andQuickCheck
dependencies. -
Added dependency on
case-insensitive
.
Source positions
-
Now
Pos
contains anInt
inside, notWord
. -
Dropped
unsafePos
and changed type ofmkPos
so it throws from pure code if its argument is not a positiveInt
. -
Added
pos1
constant that represents thePos
with value 1 inside. -
Made
InvalidPosException
contain the invalidInt
value that was passed tomkPos
.
Parse errors
-
Changed the definition of
ParseError
to have separate data constructors for “trivial” errors (unexpected/expected tokens) and “fancy” errors (everything else). -
Removed the
ErrorComponent
type class, addedErrorFancy
instead.ErrorFancy
is a sum type which can representfail
messages, incorrect indentation, and custom data (we useVoid
for that by default to “disable” it). This is better than the typeclass-based approach because every instance ofErrorComponent
needed to have constructors forfail
and indentation massages anyway, leading to duplication of code (for example for parse error component rendering). -
Added
Functor
instances forErrorItem
andErrorFancy
. -
Added the function
errorPos
to get error positions fromParseError
(previously it was a record selector inParseError
). -
Control characters in parse error are displayed in a readable form even when they are part of strings, for example:
{<newline>
({
followed by the newline character). Previously control characters were rendered in readable form only as standalone tokens. -
Added
Text.Megaparsec.Error.Builder
module to help constructParseError
s easily. It is useful for testing and debugging. Previously we had something like that in thehspec-megaparsec
package, but it does not hurt to ship it with the library. -
Added
parseErrorPretty'
allowing to display offending line in parse errors. -
Added
LineToken
type class for tokens that support operations necessary for selecting and displaying relevant line of input (used inparseErrorPretty'
). -
Added
parseTest'
function that is just likeparseTest
, but also prints offending line in parse errors. This is powered by the newparseErrorPretty'
.
Stream
- Introduced the new
Text.Megaparsec.Stream
module that is the home ofStream
type class. In version 6, the type class has been extended significantly to improve performance and make some combinators more general.
Combinators
-
Changed signatures of
failure
andtoken
, they only can signal trivial errors now. -
Added a new method of
MonadParsec
type class calledfancyFailure
for signalling non-trivial failures. Signatures of some functions (failure
,token
) have been changed accordingly. -
Added
takeWhileP
,takeWhile1P
andtakeP
toMonadParsec
. -
Added
takeRest
non-primitive combinator to consume the rest of input. -
Added
atEnd
which returnsTrue
when end of input has been reached. -
Dropped
oneOf'
andnoneOf'
fromText.Megaparsec.Char
. These were seldom (if ever) used and are easily re-implemented. -
Added
notChar
inText.Megaparsec.Char
. -
Added
space1
inText.Megaparsec.Char
. This parser is likespace
but requires at least one space character to be present to succeed. -
Added new module
Text.Megaparsec.Byte
, which is similar toText.Megaparsec.Char
, but for token streams of the typeWord8
instead ofChar
. -
integer
was dropped fromText.Megaparsec.Char.Lexer
. Usedecimal
instead. -
number
was dropped fromText.Megaparsec.Char.Lexer
. Usescientific
instead. -
decimal
,octal
, andhexadecimal
are now polymorphic in their return type and can be used to parse any instance ofIntegral
. -
float
is now polymorphic in its return type and can be used to parse any instance ofRealFloat
. -
Added new module
Text.Megaparsec.Byte.Lexer
, which provides some functions (white space and numeric helpers) fromText.Megaparsec.Char.Lexer
for streams with token typeWord8
.
Megaparsec 5.3.1
-
Various updates to the docs.
-
Allowed
QuickCheck-2.10
.
Megaparsec 5.3.0
-
Added the
match
combinator that allows to get collection of consumed tokens along with result of parsing. -
Added the
region
combinator which allows to process parse errors happening when its argument parser is run. -
Added the
getNextTokenPosition
, which returns position where the next token in the stream begins. -
Defined
Semigroup
andMonoid
instances ofParsecT
. -
Dropped support for GHC 7.6.
-
Added an
ErrorComponent
instance for()
.
Megaparsec 5.2.0
-
Added
MonadParsec
instance forRWST
. -
Allowed
many
to run parsers that do not consume input. Previously this signalled anerror
which was ugly. Of course, in most cases givingmany
a parser that do not consume input will lead to non-termination bugs, but there are legal cases when this should be allowed. The test suite now contains an example of this. Non-termination issues is something inherited from the power Megaparsec gives (with more power comes more responsibility), so thaterror
case inmany
really does not solve the problem, it was just a little ah-hoc guard we got from Parsec’s past. -
The criterion benchmark was completely re-written and a new weigh benchmark to analyze memory consumption was added.
-
Performance improvements:
count
(marginal improvement, simpler implementation),count'
(considerable improvement), andmany
(marginal improvement, simpler implementation). -
Added
stateTokensProcessed
field to parser state and helper functionsgetTokensProcessed
andsetTokensProcessed
. The field contains number of processed tokens so far. This allows, for example, create wrappers that return just parsed fragment of input stream alongside with result of parsing. (It was possible before, but very inefficient because it required traversing entire input stream twice.) -
IndentNone
option ofindentBlock
now picks whitespace after it like its sistersIndentMany
andIndentSome
do, see #161. -
Fixed a couple of quite subtle bugs in
indentBlock
introduced by changing behaviour ofskipLineComment
in version 5.1.0. See #178 for more information.
Megaparsec 5.1.2
-
Stopped using property tests with
dbg
helper to avoid flood of debugging info when test suite is run. -
Fixed the build with
QuickCheck
versions older than 2.9.0.
Megaparsec 5.1.1
- Exported the
observing
primitive fromText.Megaparsec
.
Megaparsec 5.1.0
-
Defined
displayException
forParseError
, so exceptions are displayed in human-friendly form now. This works with GHC 7.10 and later. -
Line comments parsed by
skipLineComment
now may end at the end of input and do not necessarily require a newline to be parsed correctly. See #119. -
Exposed
parseErrorTextPretty
function inText.Megaparsec.Error
to allow to renderParseError
s without stack of source positions. -
Eliminated the
old-tests
test suite — Parsec legacy. The cases that are not already obviously covered in the main test suite were included into it. -
Added
Arbitrary
instances for the following data types:Pos
,SourcePos
,ErrorItem
,Dec
,ParseError
andState
. This should make testing easier without the need to add orphan instances every time. The drawback is that we start to depend onQuickCheck
, but that’s a fair price. -
The test suite now uses the combination of Hspec and the
hpesc-megaparsec
package, which also improved the latter (that package is the recommended way to test Megaparsec parsers). -
The
try
combinator now truly backtracks parser state when its argument parser fails (either consuming input or not). Most users will never notice the difference though. See #142. -
Added the
dbg
function that should be helpful for debugging. -
Added
observing
primitive combinator that allows to “observe” parse errors without ending parsing (they are returned inLeft
, while normal results are wrapped inRight
). -
Further documentation improvements.
Megaparsec 5.0.1
-
Derived
NFData
instances forPos
,InvalidPosException
,SourcePos
,ErrorItem
,Dec
,ParseError
, andState
. -
Derived
Data
instance forParseError
,Data
andTypeable
instances forSourcePos
andState
. -
Minor documentation improvements.
Megaparsec 5.0.0
General changes
-
Removed
parseFromFile
andStorableStream
type-class that was necessary for it. The reason for removal is that reading from file and then parsing its contents is trivial for every instance ofStream
and this function provides no way to use newer methods for running a parser, such asrunParser'
. So, simply put, it adds little value and was included in 4.x versions for compatibility reasons. -
Moved position-advancing function from arguments of
token
andtokens
functions toStream
type class (namedupdatePos
). The new function allows to handle custom streams of tokens where every token contains information about its position in stream better (for example when stream of tokens is produced with happy/alex). -
Support for include files (stack of positions instead of flat position) added. The new functions
pushPosition
andpopPosition
can be used to move “vertically” in the stack of positions.getPosition
andsetPosition
still work on top (“current file”) level, but user can get full stack viagetParserState
if necessary. Note thatParseError
and pretty-printing for it also support the new feature. -
Added type function
Token
associated withStream
type class. The function returns type of token corresponding to specific token stream. -
Type
ParsecT
(and also type synonymParsec
) are now parametrized over type of custom component in parse errors. -
Parameters of
MonadParsec
type class are:e
— type of custom component in parse errors,s
— type of input stream, andm
— type of underlying monad. -
Type of
failure
primitive combinator was changed, now it accepts three arguments: set of unexpected items, set of expected items, and set of custom data. -
Type of
token
primitive combinator was changed, now in case of failure a triple-tuple is returned with elements corresponding to arguments offailure
primitive. Thetoken
primitive can also be optionally given an argument of token type to use in error messages (as expected item) in case of end of input. -
unexpected
combinator now accepts argument of typeErrorItem
instead of plainString
. -
General performance improvements and improvements in speed of some combinators,
manyTill
in particular.
Error messages
-
The module
Text.Megaparsec.Pos
was completely rewritten. The new module usesPos
data type with smart constructors to ensure that things like line and column number can be only positive.SourcePos
on the other hand does not require smart constructors anymore and its constructors are exported.Show
andRead
instances ofSourcePos
are derived and pretty-printing is done with help ofsourcePosPretty
function. -
The module
Text.Megaparsec.Error
was completely rewritten. A number of new types and type-classes are introduced:ErrorItem
,Dec
,ErrorComponent
, andShowErrorComponent
.ParseError
does not need smart constructors anymore and its constructor and field selectors are exported. It uses sets (from thecontainers
package) instead of sorted lists to enumerate unexpected and expected items. The new definition is also parametrized over token type and custom data type which can be passed around as part of parse error. Default “custom data” component isDec
, which see. All in all, we have completely well-typed and extensible error messages now.Show
andRead
instances ofParseError
are derived and pretty-printing is done with help ofparseErrorPretty
. -
The module
Text.Megaparsec.ShowToken
was eliminated and type classShowToken
was moved toText.Megaparsec.Error
. The only method of that class in now namedshowTokens
and it works on streams of tokens, where single tokes are represented byNonEmpty
list with single element.
Built-in combinators
- Combinators
oneOf
,oneOf'
,noneOf
, andnoneOf'
now accept any instance ofFoldable
, not onlyString
.
Lexer
-
Error messages about incorrect indentation levels were greatly improved. Now every such message contains information about desired ordering between “reference” indentation level and actual indentation level as well as values of these levels. The information is stored in
ParseError
in well-typed form and can be pretty-printed when necessary. As part of this improvement, type ofindentGuard
was changed. -
incorrectIndent
combinator is introduced inText.Megaparsec.Lexer
module. It allows to fail with detailed information regarding incorrect indentation. -
Introduced
scientific
parser that can parse arbitrary big numbers without error or memory overflow.float
still returnsDouble
, but it’s defined in terms ofscientific
now. SinceScientific
type can reliably represent integer values as well as floating point values,number
now returnsScientific
instead ofEither Integer Double
(Integer
orDouble
can be extracted fromScientific
value anyway). This in turn makessigned
parser more natural and general, because we do not need ad-hocSigned
type class anymore. -
Added
skipBlockCommentNested
function that should help parse possibly nested block comments. -
Added
lineFold
function that helps parse line folds.
Megaparsec 4.4.0
-
Now state returned on failure is the exact state of parser at the moment when it failed, which makes incremental parsing feature much better and opens possibilities for features like “on-the-fly” recovering from parse errors.
-
The
count
combinator now works withApplicative
instances (previously it worked only with instances ofAlternative
). It’s now also faster. -
tokens
and parsers built upon it (such asstring
andstring'
) backtrack automatically on failure now, that is, when they fail, they never consume any input. This is done to make their consumption model match how error messages are reported (which becomes an important thing as user gets more control with primitives likewithRecovery
). This means, in particular, that it’s no longer necessary to usetry
withtokens
-based parsers. This new feature does not affect performance in any way. -
New primitive parser
withRecovery
added. The parser allows to recover from parse errors “on-the-fly” and continue parsing. Once parsing is finished, several parse errors may be reported or ignored altogether. -
eitherP
combinator added. -
Removed
Enum
instance ofMessage
type. This was Parsec’s legacy that we should eliminate now.Message
does not constitute enumeration,toEnum
was never properly defined for it. The idea to usefromEnum
to determine type ofMessage
is also ugly, for this purpose new functionsisUnexpected
,isExpected
, andisMessage
are defined inText.Megaparsec.Error
. -
Minor tweak in signature of
MonadParsec
type class. Collection of constraints changed fromAlternative m, Monad m, Stream s t
toAlternative m, MonadPlus m, Stream s t
. This is done to make it easier to write more abstract code with older GHC where such primitives asguard
are defined for instances ofMonadPlus
, notAlternative
.
Megaparsec 4.3.0
-
Canonicalized
Applicative
/Monad
instances. Thanks to Herbert Valerio Riedel. -
Custom messages in
ParseError
are printed each on its own line. -
Now accumulated hints are not used with
ParseError
records that have only custom messages in them (created withMessage
constructor, as opposed toUnexpected
orExpected
). This strips “expected” line from custom error messages where it’s unlikely to be relevant anyway. -
Added higher-level combinators for indentation-sensitive grammars:
indentLevel
,nonIndented
, andindentBlock
.
Megaparsec 4.2.0
-
Made
newPos
constructor and other functions inText.Megaparsec.Pos
smarter. Now it’s impossible to createSourcePos
with non-positive line number or column number. Unfortunately we cannot useNumeric.Natural
because we need to support older versions ofbase
. -
ParseError
is now a monoid.mergeError
is used asmappend
. -
Added functions
addErrorMessages
andnewErrorMessages
to add several messages to existing error and to construct error with several attached messages respectively. -
parseFromFile
now lives inText.Megaparsec.Prim
. Previously we had 5 nearly identical definitions of the function, varying only in type-specificreadFile
function. Now the problem is solved by introduction ofStorableStream
type class. All supported stream types are instances of the class out of box and thus we have polymorphic version ofparseFromFile
. -
ParseError
is now instance ofException
(andTypeable
). -
Introduced
runParser'
andrunParserT'
functions that take and return parser state. This makes it possible to partially parse input, resume parsing, specify non-standard initial textual position, etc. -
Introduced
failure
function that allows to fail with arbitrary collection of messages.unexpected
is now defined in terms offailure
. One consequence of this design decision is thatfailure
is now method ofMonadParsec
, whileunexpected
is not. -
Removed deprecated combinators from
Text.Megaparsec.Combinator
:chainl
chainl1
chainr
chainr1
-
number
parser inText.Megaparsec.Lexer
now can be used withsigned
combinator to parse either signedInteger
or signedDouble
.
Megaparsec 4.1.1
-
Fixed bug in implementation of
sepEndBy
andsepEndBy1
and removed deprecation notes for these functions. -
Added tests for
sepEndBy
andsepEndBy1
.
Megaparsec 4.1.0
-
Relaxed dependency on
base
, so that minimal required version ofbase
is now 4.6.0.0. This allows Megaparsec to compile with GHC 7.6.x. -
Text.Megaparsec
andText.Megaparsec.Prim
do not export data typesConsumed
andReply
anymore because they are rather low-level implementation details that should not be visible to end-user. -
Representation of file name and textual position in error messages was made conventional.
-
Fixed some typos is documentation and other materials.
Megaparsec 4.0.0
General changes
-
Renamed
many1
→some
as well as other parsers that hadmany1
part in their names. -
The following functions are now re-exported from
Control.Applicative
:(<|>)
,many
,some
,optional
. See #9. -
Introduced type class
MonadParsec
in the style of MTL monad transformers. Eliminated built-in user state since it was not flexible enough and can be emulated via stack of monads. Now all tools in Megaparsec work with any instance ofMonadParsec
, not only withParsecT
. -
Added new function
parseMaybe
for lightweight parsing where error messages (and thus file name) are not important and entire input should be parsed. For example it can be used when parsing of single number according to specification of its format is desired. -
Fixed bug with
notFollowedBy
always succeeded with parsers that don’t consume input, see #6. -
Flipped order of arguments in the primitive combinator
label
, see #21. -
Renamed
tokenPrim
→token
, removed oldtoken
, becausetokenPrim
is more general and originaltoken
is little used. -
Made
token
parser more powerful, now its second argument can returnEither [Message] a
instead ofMaybe a
, so it can influence error message when parsing of token fails. See #29. -
Added new primitive combinator
hidden p
which hides “expected” tokens in error message when parserp
fails. -
Tab width is not hard-coded anymore. It can be manipulated via
getTabWidth
andsetTabWidth
. Default tab-width isdefaultTabWidth
, which is 8.
Error messages
-
Introduced type class
ShowToken
and improved representation of characters and strings in error messages, see #12. -
Greatly improved quality of error messages. Fixed entire
Text.Megaparsec.Error
module, see #14 for more information. Made possible normal analysis of error messages without “render and re-parse” approach that previous maintainers had to practice to write even simplest tests, see moduleUtils.hs
inold-tests
for example. -
Reduced number of
Message
constructors (now there are onlyUnexpected
,Expected
, andMessage
). Empty “magic” message strings are ignored now, all the library now uses explicit error messages. -
Introduced hint system that greatly improves quality of error messages and made code of
Text.Megaparsec.Prim
a lot clearer.
Built-in combinators
-
All built-in combinators in
Text.Megaparsec.Combinator
now work with any instance ofAlternative
(some of them even withApplicative
). -
Added more powerful
count'
parser. This parser can be told to parse fromm
ton
occurrences of some thing.count
is defined in terms ofcount'
. -
Removed
optionMaybe
parser, becauseoptional
fromControl.Applicative
does the same thing. -
Added combinator
someTill
. -
These combinators are considered deprecated and will be removed in future:
chainl
chainl1
chainr
chainr1
sepEndBy
sepEndBy1
Character parsing
-
Renamed some parsers:
alphaNum
→alphaNumChar
digit
→digitChar
endOfLine
→eol
hexDigit
→hexDigitChar
letter
→letterChar
lower
→lowerChar
octDigit
→octDigitChar
space
→spaceChar
spaces
→space
upper
→upperChar
-
Added new character parsers in
Text.Megaparsec.Char
:asciiChar
charCategory
controlChar
latin1Char
markChar
numberChar
printChar
punctuationChar
separatorChar
symbolChar
-
Descriptions of old parsers have been updated to accent some Unicode-specific moments. For example, old description of
letter
stated that it parses letters from “a” to “z” and from “A” to “Z”. This is wrong, since it usedData.Char.isAlpha
predicate internally and thus parsed many more characters (letters of non-Latin languages, for example). -
Added combinators
char'
,oneOf'
,noneOf'
, andstring'
which are case-insensitive variants ofchar
,oneOf
,noneOf
, andstring
respectively.
Lexer
-
Rewritten parsing of numbers, fixed #2 and #3 (in old Parsec project these are number 35 and 39 respectively), added per bug tests.
-
Since Haskell report doesn’t say anything about sign,
integer
andfloat
now parse numbers without sign. -
Removed
natural
parser, it’s equal to newinteger
now. -
Renamed
naturalOrFloat
→number
— this doesn’t parse sign too. -
Added new combinator
signed
to parse all sorts of signed numbers.
-
-
Transformed
Text.Parsec.Token
intoText.Megaparsec.Lexer
. Little of Parsec’s code remains in the new lexer module. New module doesn’t impose any assumptions on user and should be vastly more useful and general. Hairy stuff from original Parsec didn’t get here, for example built-in Haskell functions are used to parse escape sequences and the like instead of trying to re-implement the whole thing.
Other
-
Renamed the following functions:
permute
→makePermParser
buildExpressionParser
→makeExprParser
-
Added comprehensive QuickCheck test suite.
-
Added benchmarks.
Parsec 3.1.9
-
Many and various updates to documentation and package description (including the homepage links).
-
Add an
Eq
instance forParseError
. -
Fixed a regression from 3.1.6:
runP
is again exported from moduleText.Parsec
.
Parsec 3.1.8
- Fix a regression from 3.1.6 related to exports from the main module.
Parsec 3.1.7
-
Fix a regression from 3.1.6 related to the reported position of error messages. See bug #9 for details.
-
Reset the current error position on success of
lookAhead
.
Parsec 3.1.6
-
Export
Text
instances fromText.Parsec
. -
Make
Text.Parsec
exports more visible. -
Re-arrange
Text.Parsec
exports. -
Add functions
crlf
andendOfLine
toText.Parsec.Char
for handling input streams that do not have normalized line terminators. -
Fix off-by-one error in
Token.charControl
.
Parsec 3.1.4 & 3.1.5
- Bump dependency on
text
.
Parsec 3.1.3
- Fix a regression introduced in 3.1.2 related to positions reported by error messages.