Module documentation for 5.2.0
- Comparison with other solutions
- Related packages
- Links to announcements
This is an industrial-strength monadic parser combinator library. Megaparsec is a fork of Parsec library originally written by Daan Leijen.
This project provides flexible solutions to satisfy common parsing needs. The section describes them shortly. If you're looking for comprehensive documentation, see the section about documentation.
The package is built around
MonadParsec, a MTL-style monad
transformer. All tools and features work with any instance of
MonadParsec. You can achieve various effects combining monad transformers,
i.e. building monad stack. Since most common monad transformers like
ReaderT and others are instances of
you can wrap
ParsecT in these monads, achieving, for example,
On the other hand
ParsecT is instance of many type classes as well. The
most useful ones are
(its functions are included in
Text.Megaparsec) contains traditional,
general combinators that work with any instance of
Alternative and some
even with instances of
Alternative should be obvious, so
let's enumerate methods of
MonadParsec type class. The class represents
core, basic functions of Megaparsec parsing. The rest of library is built
via combination of these primitives:
failureallows to fail with arbitrary collection of messages.
labelallows to add a “label” to any parser, so when it fails the user will see the label in the error message where “expected” items are enumerated.
hiddenhides any parser from error messages altogether, this is officially recommended way to hide things, prefer it to the
tryenables backtracking in parsing.
lookAheadallows to parse something without consuming input.
notFollowedBysucceeds when its argument fails, it does not consume input.
withRecoveryallows to recover from parse errors “on-the-fly” and continue parsing. Once parsing is finished, several parse errors may be reported or ignored altogether.
observingallows to “observe” parse errors without ending parsing (they are returned in
Left, while normal results are wrapped in
eofonly succeeds at the end of input.
tokenis used to parse single token.
tokensmakes it easy to parse several tokens in a row.
getParserStatereturns full parser state.
updateParserStateapplies given function on parser state.
This list of core functions is longer than in some other libraries. Our goal was efficient and readable implementation of functionality provided by every such primitive, not minimal number of them. You can read the comprehensive description of every primitive function in Megaparsec documentation.
Megaparsec can currently work with the following types of input stream out-of-box:
ByteString(strict and lazy)
Text(strict and lazy)
Megaparsec 5 introduces well-typed error messages and ability to use custom data types to adjust the library to your domain of interest. No need to keep your info as shapeless bunch of strings anymore.
The default error component (
Dec) has constructors corresponding to
function and indentation-related error messages. It is a decent option that
should work out-of-box for most parsing needs, while you are free to use
your own custom error component when necessary with little effort.
This new design allowed Megaparsec 5 to have much more helpful error messages for indentation-sensitive parsing instead of plain “incorrect indentation” phrase.
Alex and Happy support
Megaparsec works well with streams of tokens produced by tools like
Alex/Happy. Megaparsec 5 adds
updatePos method to
Stream type class that
gives you full control over textual positions that are used to report token
positions in error messages. You can update current position on per
character basis or extract it from token — all cases are covered.
Megaparsec has decent support for Unicode-aware character parsing. Functions
for character parsing live in
Text.Megaparsec.Char (they all are
Text.Megaparsec). The functions can be divided into several
Simple parsers — parsers that parse certain character or several characters of the same kind. This includes
Parsers corresponding to categories of characters parse single character that belongs to certain category of characters, for example:
digitChar, and others.
General parsers that allow you to parse a single character you specify or one of given characters, or any character except for given ones, or character satisfying given predicate. Case-insensitive versions of the parsers are available.
Parsers for sequences of characters parse strings. These are more efficient and provide better error messages than other approaches most programmers can come up with. Case-sensitive
stringparser is available as well as case-insensitive
For those who are interested in parsing of permutation phrases, there is
Text.Megaparsec.Perm. You have to import the module explicitly, it's not
included in the
Megaparsec has a solution for parsing of expressions. Take a look at
Text.Megaparsec.Expr. You have to import the module explicitly, it's not
included in the
Given a table of operators that describes their fixity and precedence, you can construct a parser that will parse any expression involving the operators. See documentation for comprehensive description of how it works.
is a module that should help you write your lexer. If you have used
in the past, this module “fixes” its particularly inflexible
Text.Megaparsec.Lexer is intended to be imported qualified, it's not
Text.Megaparsec. The module doesn't impose how you should
write your parser, but certain approaches may be more elegant than
others. An especially important theme is parsing of white space, comments,
The design of the module allows you quickly solve simple tasks and doesn't get in your way when you want to implement something less standard.
Since Megaparsec 5, all tools for indentation-sensitive parsing are
Text.Megaparsec.Lexer module — no third party packages
Megaparsec is well-documented. All functions and data-types are thoroughly described. We pay attention to avoid outdated info or unclear phrases in our documentation. See the current version of Megaparsec documentation on Hackage for yourself.
You can visit site of the project which has several tutorials that should help you to start with your parsing tasks. The site also has instructions and tips for Parsec users who decide to switch. If you want to improve an existing tutorial or add your own, open a PR against this repo.
Despite being quite flexible, Megaparsec is also faster than Parsec. The repository includes benchmarks that can be easily used to compare Megaparsec and Parsec. In most cases Megaparsec is faster, sometimes dramatically faster. If you happen to have some other benchmarks, I would appreciate if you add Megaparsec to them and let me know how it performs.
If you think your Megaparsec parser is not efficient enough, take a look at these instructions.
Comparison with other solutions
There are quite a few libraries that can be used for parsing in Haskell, let's compare Megaparsec with some of them.
Megaparsec vs Attoparsec
Attoparsec is another prominent Haskell library for parsing. Although the both libraries deal with parsing, it's usually easy to decide which you will need in particular project:
Attoparsec is much faster but not that feature-rich. It should be used when you want to process large amounts of data where performance matters more than quality of error messages.
Megaparsec is good for parsing of source code or other human-readable texts. It has better error messages and it's implemented as monad transformer.
So, if you work with something human-readable where size of input data is usually not huge, just go with Megaparsec, otherwise Attoparsec may be a better choice.
Megaparsec vs Parsec
Since Megaparsec is a fork of Parsec, it's necessary to list main differences between the two libraries:
Better error messages. We test our error messages using dense QuickCheck tests. Good error messages are just as important for us as correct return values of our parsers. Megaparsec will be especially useful if you write compiler or interpreter for some language.
Some quirks and “buggy features” (as well as plain bugs) of original Parsec are fixed. There is no undocumented surprising stuff in Megaparsec.
Better support for Unicode parsing in
Megaparsec has more powerful combinators and can parse languages where indentation matters.
Comprehensive QuickCheck test suite covering nearly 100% of our code.
We have benchmarks to detect performance regressions.
Better documentation, with 100% of functions covered, without typos and obsolete information, with working examples. Megaparsec's documentation is well-structured and doesn't contain things useless to end users.
Megaparsec's code is clearer and doesn't contain “magic” found in original Parsec.
Megaparsec has well-typed error messages and custom error messages.
Megaparsec can recover from parse errors “on the fly” and continue parsing.
Megaparsec is faster.
Megaparsec is better supported.
If you want to see a detailed change log,
CHANGELOG.md may be helpful.
Also see this original announcement for another
To be honest Parsec's development has seemingly stagnated. It has no test suite (only three per-bug tests), and all its releases beginning from version 3.1.2 (according or its change log) were about introducing and fixing regressions. Parsec is old and somewhat famous in Haskell community, so we understand there will be some kind of inertia, but we advise you use Megaparsec from now on because it solves many problems of original Parsec project. If you think you still have a reason to use original Parsec, open an issue.
Megaparsec vs Trifecta
Trifecta is another Haskell library featuring good error messages. Like some other projects of Edward Kmett, it's probably good, but also poorly documented, arcane, and has unfixed bugs and flaws that Edward is too busy to fix (simply a fact, no offense intended). Other reasons one may question choice of Trifecta is his/her parsing library:
Complicated, doesn't have any tutorials available, and documentation doesn't help at all.
Trifecta can parse
ByteStringnatively, but not
Trifecta's error messages may be different with their own features, but certainly not as flexible as Megaparsec's error messages in the latest versions.
lens. This means you'll pull in half of Hackage as transitive dependencies. Also if you're not into
lensand would like to keep your code “vanilla”, you may not like the API.
Megaparsec vs Earley
Earley is a newer library that allows to safely (it your code compiles, then it probably works) parse context-free grammars (CFG). Megaparsec is a lower-level library compared to Earley, but there are still enough reasons to choose it over Earley:
Megaparsec is faster.
Your grammar may be not context free or you may want introduce some sort of state to the parsing process. Almost all non-trivial parsers require something of this sort. Even if your grammar is context-free, state may allow to add some additional niceties. Earley does not support that.
Megaparsec's error messages are more flexible allowing to include arbitrary data in them, return multiple error messages, mark regions that affect any error that happens in those regions, etc.
The approach Earley uses differs from conventional monadic parsing. If you work not alone, chances people you work with, especially beginners will be much more productive with libraries taking more traditional path to parsing like Megaparsec.
IOW, Megaparsec is less safe but also much more powerful.
Megaparsec vs Parsers
There is Parsers package, which is great. You can use it with Megaparsec or Parsec, but consider the following:
It depends on Attoparsec, Parsec, and Trifecta, which means you always grab half of Hackage as transitive dependencies by using it. This is ridiculous, by the way, because this package is supposed to be useful for parser builders, so they can write basic core functionality and get the rest “for free”.
It currently has a bug in definition of
lookAheadfor various monad transformers like
StateT, etc. which is visible when you create backtracking state via monad stack, not via built-in features.
We intended to use Parsers library in Megaparsec at some point, but aside from already mentioned flaws the library has different conventions for naming of things, different set of “core” functions, etc., different approach to lexer. So it didn't happen, Megaparsec has minimal dependencies, it is feature-rich and self-contained.
The following packages are designed to be used with Megaparsec:
Links to announcements
Here are some blog posts mainly announcing new features of the project and describing what sort of things are now possible:
- The original Megaparsec 4.0.0 announcement
- Megaparsec 4 and 5
- Announcing Megaparsec 5
- Latest additions to Megaparsec
The project was started and is currently maintained by Mark Karpov. You can
find complete list of contributors in
AUTHORS.md file in official
repository of the project. Thanks to all the people who propose features and
ideas, although they are not in
AUTHORS.md, without them Megaparsec would
not be that good.
Issues (bugs, feature requests or otherwise feedback) may be reported in the GitHub issue tracker for this project.
Pull requests are also welcome (and yes, they will get attention and will be merged quickly if they are good, we are progressive folks).
If you want to write a tutorial to be hosted on Megaparsec's site, open an issue or pull request here.
Copyright © 2015–2017 Megaparsec contributors
Copyright © 2007 Paolo Martini
Copyright © 1999–2000 Daan Leijen
Distributed under FreeBSD license.
matchcombinator that allows to get collection of consumed tokens along with result of parsing.
regioncombinator which allows to process parse errors happening when its argument parser is run.
getNextTokenPosition, which returns position where the next token in the stream begins.
Dropped support for GHC 7.6.
manyto run parsers that do not consume input. Previously this signalled an
errorwhich was ugly. Of course, in most cases giving
manya parser that do not consume input will lead to non-termination bugs, but there are legal cases when this should be allowed. The test suite now contains an example of this. Non-termination issues is something inherited from the power Megaparsec gives (with more power comes more responsibility), so that
manyreally does not solve the problem, it was just a little ah-hoc guard we got from Parsec's past.
The criterion benchmark was completely re-written and a new weigh benchmark to analyze memory consumption was added.
count(marginal improvement, simpler implementation),
count'(considerable improvement), and
many(marginal improvement, simpler implementation).
stateTokensProcessedfield to parser state and helper functions
setTokensProcessed. The field contains number of processed tokens so far. This allows, for example, create wrappers that return just parsed fragment of input stream alongside with result of parsing. (It was possible before, but very inefficient because it required traversing entire input stream twice.)
indentBlocknow picks whitespace after it like its sisters
IndentSomedo, see #161.
Fixed a couple of quite subtle bugs in
indentBlockintroduced by changing behaviour of
skipLineCommentin version 5.1.0. See #178 for more information.
Stopped using property tests with
dbghelper to avoid flood of debugging info when test suite is run.
Fixed the build with
QuickCheckversions older than 2.9.0.
ParseError, so exceptions are displayed in human-friendly form now. This works with GHC 7.10 and later.
Line comments parsed by
skipLineCommentnow may end at the end of input and do not necessarily require a newline to be parsed correctly. See #119.
Text.Megaparsec.Errorto allow to render
ParseErrors without stack of source positions.
old-teststest suite — Parsec legacy. The cases that are not already obviously covered in the main test suite were included into it.
Arbitraryinstances for the following data types:
State. This should make testing easier without the need to add orphan instances every time. The drawback is that we start to depend on
QuickCheck, but that's a fair price.
The test suite now uses the combination of Hspec and the
hpesc-megaparsecpackage, which also improved the latter (that package is the recommended way to test Megaparsec parsers).
trycombinator now truly backtracks parser state when its argument parser fails (either consuming input or not). Most users will never notice the difference though. See #142.
dbgfunction that should be helpful for debugging.
observingprimitive combinator that allows to “observe” parse errors without ending parsing (they are returned in
Left, while normal results are wrapped in
Further documentation improvements.
Minor documentation improvements.
StorableStreamtype-class that was necessary for it. The reason for removal is that reading from file and then parsing its contents is trivial for every instance of
Streamand this function provides no way to use newer methods for running a parser, such as
runParser'. So, simply put, it adds little value and was included in 4.x versions for compatibility reasons.
Moved position-advancing function from arguments of
Streamtype class (named
updatePos). The new function allows to handle custom streams of tokens where every token contains information about its position in stream better (for example when stream of tokens is produced with happy/alex).
Support for include files (stack of positions instead of flat position) added. The new functions
popPositioncan be used to move “vertically” in the stack of positions.
setPositionstill work on top (“current file”) level, but user can get full stack via
getParserStateif necessary. Note that
ParseErrorand pretty-printing for it also support the new feature.
Added type function
Streamtype class. The function returns type of token corresponding to specific token stream.
ParsecT(and also type synonym
Parsec) are now parametrized over type of custom component in parse errors.
MonadParsectype class are:
e— type of custom component in parse errors,
s— type of input stream, and
m— type of underlying monad.
failureprimitive combinator was changed, now it accepts three arguments: set of unexpected items, set of expected items, and set of custom data.
tokenprimitive combinator was changed, now in case of failure a triple-tuple is returned with elements corresponding to arguments of
tokenprimitive can also be optionally given an argument of token type to use in error messages (as expected item) in case of end of input.
unexpectedcombinator now accepts argument of type
ErrorIteminstead of plain
General performance improvements and improvements in speed of some combinators,
Text.Megaparsec.Poswas completely rewritten. The new module uses
Posdata type with smart constructors to ensure that things like line and column number can be only positive.
SourcePoson the other hand does not require smart constructors anymore and its constructors are exported.
SourcePosare derived and pretty-printing is done with help of
Text.Megaparsec.Errorwas completely rewritten. A number of new types and type-classes are introduced:
ParseErrordoes not need smart constructors anymore and its constructor and field selectors are exported. It uses sets (from the
containerspackage) instead of sorted lists to enumerate unexpected and expected items. The new definition is also parametrized over token type and custom data type which can be passed around as part of parse error. Default “custom data” component is
Dec, which see. All in all, we have completely well-typed and extensible error messages now.
ParseErrorare derived and pretty-printing is done with help of
Text.Megaparsec.ShowTokenwas eliminated and type class
ShowTokenwas moved to
Text.Megaparsec.Error. The only method of that class in now named
showTokensand it works on streams of tokens, where single tokes are represented by
NonEmptylist with single element.
noneOf'now accept any instance of
Foldable, not only
Error messages about incorrect indentation levels were greatly improved. Now every such message contains information about desired ordering between “reference” indentation level and actual indentation level as well as values of these levels. The information is stored in
ParseErrorin well-typed form and can be pretty-printed when necessary. As part of this improvement, type of
incorrectIndentcombinator is introduced in
Text.Megaparsec.Lexermodule. It allows to fail with detailed information regarding incorrect indentation.
scientificparser that can parse arbitrary big numbers without error or memory overflow.
Double, but it's defined in terms of
Scientifictype can reliably represent integer values as well as floating point values,
Either Integer Double(
Doublecan be extracted from
Scientificvalue anyway). This in turn makes
signedparser more natural and general, because we do not need ad-hoc
Signedtype class anymore.
skipBlockCommentNestedfunction that should help parse possibly nested block comments.
lineFoldfunction that helps parse line folds.
Now state returned on failure is the exact state of parser at the moment when it failed, which makes incremental parsing feature much better and opens possibilities for features like “on-the-fly” recovering from parse errors.
countcombinator now works with
Applicativeinstances (previously it worked only with instances of
Alternative). It's now also faster.
tokensand parsers built upon it (such as
string') backtrack automatically on failure now, that is, when they fail, they never consume any input. This is done to make their consumption model match how error messages are reported (which becomes an important thing as user gets more control with primitives like
withRecovery). This means, in particular, that it's no longer necessary to use
tokens-based parsers. This new feature does not affect performance in any way.
New primitive parser
withRecoveryadded. The parser allows to recover from parse errors “on-the-fly” and continue parsing. Once parsing is finished, several parse errors may be reported or ignored altogether.
Messagetype. This was Parsec's legacy that we should eliminate now.
Messagedoes not constitute enumeration,
toEnumwas never properly defined for it. The idea to use
fromEnumto determine type of
Messageis also ugly, for this purpose new functions
isMessageare defined in
Minor tweak in signature of
MonadParsectype class. Collection of constraints changed from
Alternative m, Monad m, Stream s tto
Alternative m, MonadPlus m, Stream s t. This is done to make it easier to write more abstract code with older GHC where such primitives as
guardare defined for instances of
Monadinstances. Thanks to Herbert Valerio Riedel.
Custom messages in
ParseErrorare printed each on its own line.
Now accumulated hints are not used with
ParseErrorrecords that have only custom messages in them (created with
Messageconstructor, as opposed to
Expected). This strips “expected” line from custom error messages where it's unlikely to be relevant anyway.
Added higher-level combinators for indentation-sensitive grammars:
newPosconstructor and other functions in
Text.Megaparsec.Possmarter. Now it's impossible to create
SourcePoswith non-positive line number or column number. Unfortunately we cannot use
Numeric.Naturalbecause we need to support older versions of
ParseErroris now a monoid.
mergeErroris used as
newErrorMessagesto add several messages to existing error and to construct error with several attached messages respectively.
parseFromFilenow lives in
Text.Megaparsec.Prim. Previously we had 5 nearly identical definitions of the function, varying only in type-specific
readFilefunction. Now the problem is solved by introduction of
StorableStreamtype class. All supported stream types are instances of the class out of box and thus we have polymorphic version of
ParseErroris now instance of
runParserT'functions that take and return parser state. This makes it possible to partially parse input, resume parsing, specify non-standard initial textual position, etc.
failurefunction that allows to fail with arbitrary collection of messages.
unexpectedis now defined in terms of
failure. One consequence of this design decision is that
failureis now method of
Removed deprecated combinators from
Text.Megaparsec.Lexernow can be used with
signedcombinator to parse either signed
Fixed bug in implementation of
sepEndBy1and removed deprecation notes for these functions.
Added tests for
Relaxed dependency on
base, so that minimal required version of
baseis now 220.127.116.11. This allows Megaparsec to compile with GHC 7.6.x.
Text.Megaparsec.Primdo not export data types
Replyanymore because they are rather low-level implementation details that should not be visible to end-user.
Representation of file name and textual position in error messages was made conventional.
Fixed some typos is documentation and other materials.
someas well as other parsers that had
many1part in their names.
The following functions are now re-exported from
optional. See #9.
Introduced type class
MonadParsecin the style of MTL monad transformers. Eliminated built-in user state since it was not flexible enough and can be emulated via stack of monads. Now all tools in Megaparsec work with any instance of
MonadParsec, not only with
Added new function
parseMaybefor lightweight parsing where error messages (and thus file name) are not important and entire input should be parsed. For example it can be used when parsing of single number according to specification of its format is desired.
Fixed bug with
notFollowedByalways succeeded with parsers that don't consume input, see #6.
Flipped order of arguments in the primitive combinator
label, see #21.
token, removed old
tokenPrimis more general and original
tokenis little used.
tokenparser more powerful, now its second argument can return
Either [Message] ainstead of
Maybe a, so it can influence error message when parsing of token fails. See #29.
Added new primitive combinator
hidden pwhich hides “expected” tokens in error message when parser
Tab width is not hard-coded anymore. It can be manipulated via
setTabWidth. Default tab-width is
defaultTabWidth, which is 8.
Introduced type class
ShowTokenand improved representation of characters and strings in error messages, see #12.
Greatly improved quality of error messages. Fixed entire
Text.Megaparsec.Errormodule, see #14 for more information. Made possible normal analysis of error messages without “render and re-parse” approach that previous maintainers had to practice to write even simplest tests, see module
Reduced number of
Messageconstructors (now there are only
Message). Empty “magic” message strings are ignored now, all the library now uses explicit error messages.
Introduced hint system that greatly improves quality of error messages and made code of
Text.Megaparsec.Prima lot clearer.
All built-in combinators in
Text.Megaparsec.Combinatornow work with any instance of
Alternative(some of them even with
Added more powerful
count'parser. This parser can be told to parse from
noccurrences of some thing.
countis defined in terms of
Control.Applicativedoes the same thing.
These combinators are considered deprecated and will be removed in future:
Renamed some parsers:
Added new character parsers in
Descriptions of old parsers have been updated to accent some Unicode-specific moments. For example, old description of
letterstated that it parses letters from “a” to “z” and from “A” to “Z”. This is wrong, since it used
Data.Char.isAlphapredicate internally and thus parsed many more characters (letters of non-Latin languages, for example).
string'which are case-insensitive variants of
Rewritten parsing of numbers, fixed #2 and #3 (in old Parsec project these are number 35 and 39 respectively), added per bug tests.
Since Haskell report doesn't say anything about sign,
floatnow parse numbers without sign.
naturalparser, it's equal to new
number— this doesn't parse sign too.
Added new combinator
signedto parse all sorts of signed numbers.
Text.Megaparsec.Lexer. Little of Parsec's code remains in the new lexer module. New module doesn't impose any assumptions on user and should be vastly more useful and general. Hairy stuff from original Parsec didn't get here, for example built-in Haskell functions are used to parse escape sequences and the like instead of trying to re-implement the whole thing.
Renamed the following functions:
Added comprehensive QuickCheck test suite.
Many and various updates to documentation and package description (including the homepage links).
Fixed a regression from 3.1.6:
runPis again exported from module
Fix a regression from 3.1.6 related to exports from the main module.
Fix a regression from 3.1.6 related to the reported position of error messages. See bug #9 for details.
Reset the current error position on success of
Text.Parsecexports more visible.
Text.Parsec.Charfor handling input streams that do not have normalized line terminators.
Fix off-by-one error in
Parsec 3.1.4 & 3.1.5
Bump dependency on
- Fix a regression introduced in 3.1.2 related to positions reported by error messages.