hworker
A reliable at-least-once job queue built on top of redis.
LTS Haskell 22.40: | 0.1.0.1@rev:1 |
Stackage Nightly 2023-12-26: | 0.1.0.1@rev:1 |
Latest on Hackage: | 0.1.0.1@rev:1 |
hworker-0.1.0.1@sha256:c5f32a8283ca858e67b2a49a51c41b36aa661228eca1bc6fc4c22293ddbe4b70,1539
Module documentation for 0.1.0.1
- System
About
hworker
is a Redis-backed persistent at-least-once queue library. It
is vaguely inspired by sidekiq
for Ruby. It is intended to be a
simple reliable mechanism for processing background tasks. The jobs
can be created by a Haskell application or any application that can
push JSON data structures of the right shape into a Redis queue. The
application that processes the jobs need not be the same one as the
application that creates them (they just need to be able to talk to
the same Redis server, and use the same serialization to/from JSON).
Stability
This has been running in one application sending email (using
hworker-ses
) for several months. This is relatively low traffic
(transactional messages) most of the time, with spikes of 10k-30k
messages (mailing blasts).
Important Note
The expiration of jobs is really important. It defaults to 120 seconds, which may be short depending on your application (for things like sending emails, it may be fine). The reason why this timeout is important is that if a job ever runs longer than this, the monitor will think that the job failed in some inexplicable way (like the server running the job died) and will add the job back to the queue to be run. Based on the semantics of this job processor, jobs running multiple times is not a failure case, but it’s obviously not something you want to happen, so be sure to set the timeout to something reasonable for your application.
Overview
To define jobs, you define a serialized representation of the job, and
a function that runs the job, which returns a status. The behavior of
uncaught exceptions is defined when you create the worker - it can be
either Failure
or Retry
. Jobs that return Failure
are removed
from the queue, whereas jobs that return Retry
are added again. The
only difference between a Success
and a Failure
is that a
Failure
returns a message that is logged (ie, neither run again).
Example
See the example
directory in the repository.
Semantics
This behavior of this queue processor is at-least-once.
We rely on the defined behavior of Redis for reliability. Once a job
has been queue
d, it is guaranteed to be run eventually, provided
some worker and monitor threads exist. If the worker thread that was
running a given job dies, the job will eventually be retried (if you
do not want this behavior, do not start any monitor threads). Once the
job completes, provided nothing kills the worker thread in the
intervening time, jobs that returned Success
will not be run again,
jobs that return Failure
will have their messages logged and will
not be run again, and jobs that return Retry
will be queued
again. If something kills the worker thread before these
acknowledgements go through, the job will be retried. Exceptions
triggered within the job cannot affect the worker thread - what they
do to the job is defined at startup (they can cause either a Failure
or Retry
).
Any deviations from this behavior are considered bugs that will be fixed.
Redis Operations
Under the hood, we will have the following data structures in redis
(name
is set when you create the hworker
instance):
hworker-jobs-name
: list of json serialized job descriptions
hworker-progress-name
: a hash of jobs that are in progress, mapping to time started
hworker-broken-name
: a hash of jobs to time that couldn’t be deserialized; most likely means you changed the serialization format with jobs still in queue, or you pointed different applications at the same queues.
hworker-failed-queue
: a record of the jobs that failed (limited in size based on config).
In the following pseudo-code, I’m using MULTI
…EXEC
to indicate
atomic blocks of code. These are actually implemented with lua and
EVAL
, but I think it’s easier to read this way. If you want to see
what’s actually happening, just read the code - it’s not very long!
When a worker wants to do work, the following happens:
now = TIME
MULTI
v = RPOP hworker-jobs-name
if v
HSET hworker-progress-name v now
EXEC
v
When it completes the job, it does the following:
v = JOB
HDEL hwork-progress v
If the job returned Retry
, the following occurs:
v = JOB
t = START_TIME
MULTI
LPUSH hwork-jobs v
HDEL hwork-progress t
EXEC
A monitor runs on another thread that will re-run jobs that stay in progress for too long (as that indicates that something unknown went wrong). The operation that it runs periodically is:
keys = HKEYS (or HSCAN) hwork-progress
for keys as v:
started = HGET hwork-progress v
if started < TIME - timeout
MULTI
RPUSH hwork-jobs v
HDEL hwork-progress v
EXEC
Note that what the monitor does and Retry
is slightly different -
the monitor puts jobs on the front of the queue, whereas Retry
puts
them on the back.
Primary Libraries Used
- hedis
- aeson
Contributors
- Daniel Patterson (@dbp - [email protected])
Build Status
Changes
-
0.1.0.1 Daniel Pattersion [email protected] 2015-11-1
Add example in haddocks.
-
0.1.0.0 Daniel Patterson [email protected] 2015-10-25
Initial release.