Fast Unicode 12.1.0 normalization in Haskell (NFC, NFKC, NFD, NFKD).
What is normalization?
Unicode characters with adornments (e.g. Á) can be represented in two different forms, as a single composed character (U+00C1 = Á) or as multiple decomposed characters (U+0041(A) U+0301( ́ ) = Á). They are differently encoded byte sequences but for humans they have exactly the same visual appearance.
A regular byte comparison may tell that two strings are different even though
they might be equivalent. We need to convert both the strings in a
normalized form using the Unicode
Character Database before we can
compare them for equivalence. For example:
>> import Data.Text.Normalize >> normalize NFC "\193" == normalize NFC "\65\769" True
Please use https://github.com/harendra-kumar/unicode-transforms to raise issues, or send pull requests.
- Update to Unicode version 12.1.0
- Update Quickcheck dependency version bounds
- Test with GHC 8.6.5
- Update dependency version bounds
- Test with GHC 8.6.2
- GHC 8.4.1 support
- GHC 8.2.1 support
- Work around a GHC/LLVM issue for ARM
- Update dependency versions
- Support Unicode version 9.0
- Improve speed and resource hog during compilation
- Support Unicode version 8.0
- Switch to pure Haskell implementation
- Initial release based on utf8proc C implementation