katip-elasticsearch is a scribe for the Katip logging framework that sends structured logs to ElasticSearch.
Built in bounded buffering.
Configurable pool of logging workers to help with high write volume.
Optional field type annotation to avoid mistyping values.
Optional automatic date sharding, so logs can be filed into monthly, weekly, daily, hourly, every minute indices. You can even specify your own index routing logic. This pattern can be seen in the ELK stack as a way of keeping indexes reasonably sized and easy to optimize, rotate, and manage.
Customizable retry policy for temporary outages and errors.
Automatic index and mapping setup.
- Loosen katip dep
- Loosen deps
- Update template for ESv1 to not analyze certain fields like host and
namespace. These are not fields that benefit from tokenization. If
you’re on ESv1, after running
mkEsScribe, the logs template will be updated to stop analyzing those fields. That means the next index that is cut (e.g. tomorrow’s logs if you’re using daily index sharding) will no longer analyze some fields.
- Allow newer bloodhound, aeson, async.
- Fix bug where index was created in
mkEsScribewhen it would not be used due to index sharding.
- Update some index settings if index already exists and sharding is not used.
- For ES V5 and higher, stop using the deprecated (and in 6.x, removed)
stringtype for index templates, instead using
keywordas appropriate. This makes
katip-elasticsearchcompatible with ES 6.x.
- Allow http-types 0.12
- Fix benchmark and test builds for stackage nightly
- Add repository/homepage info to cabal file
- Bump dependencies to allow GHC 8.2.1
- Update to bloodhound >= 0.13.0.0. This version adds support for both ElasticSearch versions 1 and 5. Previously, we implicitly supported one and maybe would work on 5. The types in
EsScribeCfghad to change to be able to specify which version was being targeted.
- Improved documentation.
- Widen dependency on katip
- Loosen deps on aeson to allow 188.8.131.52
- Loosen deps on bloodhound, aeson, and http-client.
- Added zero padding to date-based indices. This shouldn’t negatively
impact most users but to be safe, this was put behind a breaking
version number. Previously, you may see indices
log-index-prefix-2016-1-2. That index will now be
log-index-prefix-2016-01-02, so at worst when you deploy, the day of the change will have 2 indices: one zero-padded and one not. If you are using custom index sharding, you will not be affected by this. The reasoning here is that most existing elasticsearch tools such as
curatorexpects date indices to be zero padded. By switching to the standard, users can easily get log rotation and other features from
curatorrather than having to roll their own or add them to Katip.
- Drop direct dependency on random, upgrade uuid to >= 1.3.12 for safer id generation. Previously, UUID was using randomIO, which uses the system clock as a seed. So if multiple nodes happened to start at the same time, they would produce conflicting UUID sequences.
Default index sharding policy to daily. Previously it was no sharding. The reasoning here is that no sharding creates very large indices which become very difficult to manage in production. Rotating data out on a time basis is very slow compared to deleting date-based indices.
Upgrade note: if you were using the defaults before and switch to daily, rather than having the index name of
my-index, you’ll start seeing
my-index-2016-3-14. The good news is that whatever you’re using to use to search against your logs (such as kibana) will support index patterns, so just use the pattern of
my-index*to get everything. Eventually if you have a retention period, you can manually delete the
my-indexindex without disruption.
- Set upper bounds for a few dependencies.
- Initial release