co-log-concurrent is an asynchronous backend for the co-log library.
The core idea of co-log-concurrent is that you may easily make your logs asynchronous by adding a few
lines of code to the program that is using
co-log. But we design this library in a way that you can easily inject it into your logs pipeline, in case if you need some particular functionality, and the library does not prevent you from doing so.
When do you need co-log-concurrent?
In some applications storing logs may become a bottleneck, it happens because if logs are synchronous, then before writing a log thread must take a lock on the resource. It means that we serialize the processing of the messages and introduce additional contention. Asynchronous logging can improve the situation, a thread emits logs and knows that it logs are written at some future point of time.
There are a few kinds of applications that may benefit from asynchronous logs writing: CPU intensive applications Web services In general asynchronous logging has some downsides.
- Unbounded memory usage - if there is no backpressure mechanism the user threads, threads may generate more logs that can we can store at the same amount of time. In such cases messages are accumulated in memory. It extends GC times and memory usage.
- Persistence requirements - sometimes application may want to ensure that we persisted the logs before it moved to the next statement. It is not a case with concurrent log systems in general; some we lose logs even the thread moves forward. It may happen when the application exits before dumping all logs.
- Non-precise logging - sometimes there may be anomalies when storing logs, such as logs reordering or imprecise timestamps.
co-log-concurrent provides a framework that you can use to have precisely the properties you need. But you still need to carefully think if a violation of the properties may harm your application.
How to use?
For a general description of the co-log framework refer to the co-log documentation it’s always up to date with the latest library version. In this tutorial, we concentrate on the co-log-concurrent alone.
You should use simple API if you don’t need anything special and want the library to work. Simple API provides the following defaults: There is a backpressure mechanism: a thread is blocked if there are too many pending messages. Messages event from the different threads is never reordered. Messages may be lost if the program is abnormally killed (e.g. using sigKILL), but the library does it’s best effort to dump all logs in all other cases.
To use simple API, you should wrap your program with
withBackgroundWorker function. It takes the following parameters:
withBackgroundLogger (defCapacity :: Capacity) (logDumper :: LoggerAction IO a) (program :: IO a)
defCapacity is the size of the buffer for not yet stored messages, it acts as a backpressure mechanism protecting your program from memory overflows.
logDumper an action to store logs, you can use simple
Colog.IO.logStringStdout or any other
program is your ordinary program. So the skeleton for your application may look like:
-- usual imports: import Colog.Actions import Colog.Monad import Colog.Concurrent -- import of the library main :: IO () main = withBackgroundLogger defCapacity (pure ()) logByteStringStdout $ \log -> usingLoggerT log $ do logMsg @ByteString "Starting application..." ... logMsg @ByteString "Finishing application..."
This approach is enough for most of the cases, and we try to keep this API fast and safe.
Sometime your application may need some advanced features, in such a case, you need to know how to use
co-log-concurrent for constructing log pipelines. Please refer to the haddocks for that.
Comparison with other libraries may be outdated, as other libraries are improving and may have solved the issued described below.
fast-logger — fast logger implements concurrent approach to storing logs, it’s very effective. It has a buffer for each capability that each thread can append messages to the buffer without locking. However, that approach comes at some costs, in pathological cases runtime may reschedule a thread to another capability and logs from that thread may be reordered. Even if a case is pathological, it happens in real-world applications. Current
co-log-concurrentimplementation guarantees no logs reordering by the cost of using
STMchannels. Another scenario that may have downsides of using
fast-loggeris that it takes over a timer control. Library uses auto-update package that updates timer every second, to reduce syscall count, it means that precision of the timer is 1s, that may not be suitable for some applications.
co-log-concurrentis abstracted over the timer so that user can use any approach that suits it well, including not to give any timestamp to the message at all.
katip — katip uses the approach that is very similar to
co-log-concurrentand has the same runtime properties. However
co-log-concurrentallows to use any messages, timers and doesn’t restrict the user to use particular ones. It makes co-log-concurrent more flexible.
Revision history for co-log-concurent
- Introduce mkCapacity function, as previously it was not possible to create custom capacity.
- Fork background logger now takes an explicit flush action. And reads logs in chunks.
0.4.0.0 – 2020-04-18
- Library extracted from co-log.