GPL-2.0-only licensed by John MacFarlane
Maintained by [email protected]
This version can be pinned in stack with:skylighting-0.7.7@sha256:0339943c68b761f45cd046b32975224744ebfdcbf5dafd55e0364f412f490b7f,10194

Module documentation for 0.7.7

skylighting

Package License Status
skylighting license skylighting build status
skylighting-core license skylighting-core build status

A Haskell syntax highlighting library with tokenizers derived from KDE XML syntax highlighting descriptions.

A command-line highlighter program, skylighting, is also provided.

This project is divided up into two packages:

  • skylighting-core: this provides KDE XML parsing, data types, and output formatters. This includes the core functionality of the skylighting project licensed under the BSD3 license, along with the KDE XML files, some of which are licensed under the LGPL or GPL. This package does not provide any built-in parsers corresponding to the XML descriptions, however. For that, use skylighting.
  • skylighting: this exposes the skylighting-core API but also provides bundled Haskell parser modules derived from the XML descriptions in the core package. This package is entirely licensed under the GPL.

Motivation

This library is the successor to highlighting-kate, which had some problems that were difficult to resolve given its architecture.

In highlighting-kate, the XML syntax descriptions were converted into individual parsec parsers, which were then compiled. This made it difficult to handle IncludeRules properly without circular imports. There was also no way to load a syntax description dynamically.

Skylighting, by contrast, parses the XML syntax descriptions into Haskell data structures, which are then interpreted by a “tokenize” function. IncludeRules can now be handled properly, and users can add new syntax descriptions dynamically. It is also now possible to convert .theme files directly into styles.

Skylighting is also faster than highlighting-kate, by a factor of 3 in some cases.

Installing

To install the latest release from Hackage, do

stack install skylighting

or

cabal install skylighting

If you want the command-line tool, set the executable flag using --flag "skylighting:executable" in stack or -fexecutable in cabal.

The release tarball for the skylighting package includes generated files not present in this repository. Building from this repository is a two-step process. In the first step we build the skylighting-core package, which provides a program, skylighting-extract, which reads XML syntax highlighting definitions from the xml directory and writes Haskell source files. In the second we actually build the skylighting package.

Using cabal:

cd skylighting-core
cabal install -fexecutable
cd ../skylighting
skylighting-extract ../skylighting-core/xml/*.xml
cabal install -fexecutable

Using stack:

stack install --flag skylighting-core:executable skylighting-core
cd skylighting
skylighting-extract ../skylighting-core/xml/*.xml
cd ..
stack install --flag skylighting:executable

Command-line tool

A command-line executable, skylighting, is installed if the executable cabal flag is set in building.

For help, skylighting --help.

Adding new syntaxes

To compile with additional syntaxes, simply add the syntax definition (XML) file to the xml directory of the skylighting-core package and repeat the bootstrap build described above.

Note that both the library and the executable can dynamically load syntax definitions, so you may not need to compile them in. If you prefer this approach, you can use the skylighting-core package directly which provides the XML files without the generated code produced by the bootstrap process described above.

License

The skylighting package is licensed under the GPL because some of the XML syntax descriptions from which its tokenizers are generated are GPL-licensed. However, the skylighting-core package, which provides the core types and functions of this project is licensed under the BSD3 license and bundles the GPL-licensed XML files separately.

References

Kate syntax highlighting documentation: https://docs.kde.org/stable5/en/applications/katepart/highlight.html

Kate highlighting definitions: https://github.com/KDE/syntax-highlighting/tree/master/data/syntax

Changes

Revision history for skylighting

0.7.7 – 2019-02-08

  • Depend on skylighting-core 0.7.7.x.

0.7.6 – 2019-02-08

  • Depend on skylighting-core 0.7.6.x.

0.7.5 – 2018-12-01

  • Depend on skylighting-core 0.7.5.x.
  • Add J, sml, typescript.

0.7.4 – 2018-10-08

  • Depend on skylighting-core 0.7.4.x.

0.7.3 – 2018-08-27

  • Add Sylighting.Syntax.Default and ‘default’ language. This just highlights everything as normal text.
  • Require base >= 4.8. Drop support for ghc 7.8.

0.7.2 – 2018-06-08

  • Require skylighting-core >= 0.7.2.

0.7.1 – 2018-03-15

  • Require skylighting-core >= 0.7.1.

0.7.0.2 – 2018-03-06

  • Require skylighting-core >= 0.7.0.2.

0.7.0.1 – 2018-03-03

  • Updated changelog.md.

0.7 – 2018-03-03

  • Split skylighting into skylighting-core (BSD3) and skylighting (GPL) (#37, Jonathan Daugherty). skylighting-core (BSD3 licensed) includes the core library functionality and the source of the XML syntax, but not the compiled Syntax modules, and skylighting (GPL2 licensed) includes Syntax modules compiled from the XML syntax files.

  • We discontinue support for GHC before 7.10 (because we need to support the reexport-modules Cabal directive.)

  • Print message about loaded syntax defs to stderr (#36).

  • New module Skylighting.Loader, exporting loadSyntaxFromFile and loadSyntaxesFromDir (Jonathan Daugherty).

  • Adjust tests to load xml definitions dynamically.

0.6 – 2018-01-18

  • Add ANSI color output (Alexander Ronald Altman, #22).

    • New package dependencies: ansi-terminal and colour.
    • New module: Skylighting.Format.ANSI, exporting formatANSI, (also reexported from the module Skylighting).
    • In Skylighting.Types, new types ANSIColorLevel and Xterm256ColorCode, and a new field ansiColorLevel in FormatOptions.
    • Main skylighting executable now supports ANSI output (use ansi as argument to --format), which is now the default output format. A new flag --color-level has been added to select the number of colors (with options 16, 256, true, and the default auto).
  • Reword error for missing contexts in IncludeRules.

  • Use ordNub instead of nub for missingIncludes.

0.5.1 – 2018-01-07

  • Fixed tokenizer exceptions (#31). Previously throwError did not actually generate Left results in tokenize, because of the way the MonadPlus instance worked. Rewrote the TokenizerM monad from scratch so that exceptions would not be skipped.

  • Improved error message about undefined contexts.

  • Makefile: use cabal for bootstrap target.

  • README: remove stack bootstrap instructions. You can no longer bootstrap with stack, because it insists on building ALL the executables the first time, and of course that can’t be done with this library.

  • Use quickcheck for fuzz test.

0.5.0.1 – 2017-12-18

  • Small improvements to fuzz tests in test suite. We now ensure that we print the random text on test failure. Also, we now run the test with many smaller samples rather than one big one.

  • Add aeson lower bound (because of toEncoding) (#28).

0.5 – 2017-12-10

  • Fix line spacing and overflowing content in generated HTML (David Baynard, #25, see jgm/pandoc#4128).

    • Fix empty line height, explicitly
    • Ensure long lines scroll on screen
    • Only apply colour to the outer div
    • Don’t reset line number colour
    • Fix borders on empty code lines
    • Collapse divs correctly.
  • Changes to Style types and JSON instances. Previously we could not successfully round-trip through JSON.

    • tokenStyles is now a map rather than an association list.
    • We now use line-number-color instead of line-numbers at the top level in the JSON instances, falling back to line-numbers in editor-colors, for KDE theme compatibility.
    • We use line-number-background-color at the top level, falling back to the text background color.
    • We use text-color at the top level, falling back to the text-color of the Normal token style if it exists, for KDE compatibility.
    • We use background-color at the top level, falling back to the background-color in editor-colors, for KDE compatibility.
    • A round-trip JSON test has been added.

0.4.4.1 – 2017-11-27

  • HTML formatting: fix color, bgcolor when numbering enabled. Previously, the code got the color and background color the numbers were supposed to get. See jgm/pandoc #4103.
  • Updated syntax descriptions for bash, clojure, commonlisp, diff, dockerfile, doxygen, doxygenlua, fsharp, hamlet, haskell, haxe, java, javascript, julia, latex, literate-curry, literate-haskell, makefile, mediawiki, monobasic, ocaml, prolog, r, relaxng, scala, sci, sql-mysql, sql-postgresql, sql, xslt.
  • test program: use –accept instead of –regen.

0.4.4 – 2017-11-21

  • HTML formatter: always use an a element (rather than a div) for source lines. The divs were invalid, because code elements must contain phrasing content. Previously we used a elements when line anchors were called for, but there is no clear reason not to use them always.
  • skylighting binary: add doctype to generated HTML.

0.4.3.2 – 2017-11-04

  • Fixed regression in data-line-number attributes in HTML output. We should not include the lineIdPrefix in front of this.

0.4.3.1 – 2017-11-03

  • Fixed typo in css (Artymort).
  • Fixed travis build times.

0.4.3 — 2017-11-02

  • FormatOptions: added lineIdPrefix (jgm/pandoc#4031). This is needed because a web page may contain several code samples, and we need to make sure that the ids on lines are unique.
  • HTML formatter: use lineIdPrefix in ids for lines.
  • HTML formatter: Don’t put href attributes on divs.

0.4.2 — 2017-10-26

  • HTML output: remove outer div. This prevented margin collapsing between the pre and surrounding block elements, and often gave us excess white space around code blocks. See jgm/pandoc#3996.

0.4.1 — 2017-09-28

  • Updated XML definitions from KDE repository. Changed: actionscript, ada, agda, alert, alert_indent, asn1, awk, bash, boo, c, changelog, clojure, cmake, coldfusion, commonlisp, cpp, cs, css, curry, d, dockerfile, dot, doxygen, doxygenlua, eiffel, email, erlang, fortran, fsharp, gcc, haskell, haxe, isocpp, java, javascript, jsp, julia, lilypond, lua, m4, makefile, matlab, maxima, mips, modelines, modula-2, monobasic, objectivec, objectivecpp, ocaml, octave, opencl, pascal, perl, php, pike, postscript, prolog, purebasic, python, r, relaxng, relaxngcompact, rest, rhtml, ruby, rust, scala, scheme, sci, sql-mysql, sql-postgresql, sql, tcl, tcsh, verilog, vhdl, xml, xslt, xul, zsh.
  • Added support for powershell, using definition from KDE repository.
  • Stop terminating long builds on TravisCI (Kyle Ondy).

0.4 — 2017-09-15

  • Removed ToJSON/FromJSON instances for KeywordAttr, WordSet, Matcher, Context, ContextSwitch, Rule, Syntax (added in 0.3.5). Creating these increased the memory requirements for compiling skylighting to a degree not justified by the usefulness of these instances.

0.3.5 — 2017-09-14

  • Added ToJSON/FromJSON instances for all basic types.
  • Added some background colors to ‘kate’ style, matching default.theme, and made FromJSON for Style sensitive to background-color.

0.3.4.1 – 2017-09-09

  • HTML formatting: do not use div elements for source lines in inline output.

0.3.4 – 2017-09-09

  • HTML formatting changes (David Baynard). Note that these changes may require changes in hard-coded CSS that is used with skylighting’s HTML output.

    • Wrap lines of source code in a div with display set to inline-block. The divs make per-line processing easier. They cannot be set as display: block as that introduces extra new lines when copying and pasting.
    • Render line numbers in html using CSS pseudo elements rather than a table. The line numbers are always produced, as data-line-number attributes, and css to display them as ::before elements are always produced. The option to switch on line numbering only toggles a class; this means it is possible to toggle line numbering without re-running skylighting.
    • If the linkAnchors option is set, wrap with an a element rather than a div, set so that clicking the line number (and only the line number) will jump to that line.
    • Code wraps by default when html is printed, with wrapped lines indented.

0.3.3.1 – 2017-06-26

  • Updated xml syntax definitions and clojure test.
  • Improved ‘make update-xml’ target.
  • Updated version bound for criterion.

0.3.3 – 2017-04-22

  • Revert change from 0.3.2; now entities from a DTD are resolved if the DTD element is included. This is needed for agda.xml and some others. If you want to use a custom xml definition without needing language.dtd, just remove the reference to language.dtd in your xml file.
  • Added case keyword for ats (Hanwen (Steinway) Wu).

0.3.2 – 2017-04-01

  • Parse xml definitions without resolving entities from language.dtd. This allows xml definitions to be used even when language.dtd isn’t available. Existing definitions don’t rely on the two or three entities definied in language.dtd, so this is harmless.
  • Small optimizations.

0.3.1 – 2017-02-28

  • Use handwritten parser for Float instead of regex. Fixes jgm/highlighting-kate#57 (again).
  • Added float parsing tests.
  • Use parsec parsers, not regex, for CStringChar and CChar.
  • Rewrote Int, Hex, Oct, CStringChar, CCHar parsers with parsec instead of regex. This speeds things up.
  • Don’t include leading + in Int, Hex, Oct parsers. That gives bad results for things like i+1.

0.3 – 2017-02-19

  • Store literal binary-encoded representations of the Syntax structure in Skylighting.Syntax.*.hs, instead of the structure itself. This works around a problem ghc has compiling modules with large structures. This indirect method produces faster compile times and avoids massive memory usage by ghc (especially in profiling builds). For background see http://stackoverflow.com/questions/16348340/. Closes #7.
  • Types: Internals of ‘WordSet’ are no longer exposed.

0.2 – 2017-02-19

  • Restore Data, Typeable, Generic instances for basic types. To do this, we remove reCompiled field from RE (API change). Instead, we have the tokenizer compile regexes the first time it encounters them, memoizing the results. Performance is not significantly worse.
  • Skylighting.Syntax.*: use string representation of the Syntax, which is then ‘read’, rather than including the code for the data structure directly (#7). This indirect method produces faster compile times and avoids massive memory usage by ghc (especially in profiling builds). For background see http://stackoverflow.com/questions/16348340/compiling-very-large-constants-with-ghc
  • Use -fprof-auto-exported for profiling builds.
  • Added benchmark for xml syntax definition parsing.
  • Patched perl.xml (improperly escaped regex) (#8). This fixes a perl regex compilation error for regex patterns with backslash as delimiter, e.g. m\a.
  • Add a flag to build with system pcre (Felix Yan).

0.1.1.5 – 2017-02-03

  • Avoid depending on a PrintfArg instance for Text (#5). This isn’t provided in some older versions of the text library. This change should allow the package to build on older platforms.

0.1.1.4 – 2017-01-31

  • Properly escape characters in subDynamic. This fixes a problem that caused failures with

    echo -e "s\0b\0c" | skylighting -s perl
    
  • More efficient rewrite of char escaping for regexes.

0.1.1.3 – 2017-01-29

  • Increase test timeout to 6s.
  • Avoid double {{}} in latex macros.
  • Fixed problem compiling Format.LaTeX on older ghc versions (7.8.3) that can’t take a Text as PrintfArg.

0.1.1.2 – 2017-01-28

  • Added much more extensive testing to ensure that tokenizers don’t hang or drop input.
  • Fixed some issues with line-end and fallthrough context handling.
  • Fixed a bug in WordDetect handling which caused it to drop input (#2).

0.1.1.1 – 2017-01-21

  • Optimized. Speed is now comparable to highlighting-kate and often better.
  • Added benchmarks.

0.1.1 – 2017-01-19

  • Added breezeDark style, from breeze-dark.theme.
  • Fixed performance bug in regex application (#1). This gives a significant speedup, especially for inputs with long lines.

0.1.0.1 – 2016-12-23

  • Fixed bug in LaTeX renderer (backslash in wrong position).

0.1.0.0 – 2016-12-23

  • Initial release