BSD-3-Clause licensed
Maintained by JP Rupp
This version can be pinned in stack with:rocksdb-haskell-jprupp-2.8.0@sha256:3b864483e8c3b343e52c379f102d89b1bd99e75c34c689889624d0e10ef6eee2,2475

Module documentation for 2.8.0

Used by 2 packages in nightly-2026-11-17(full list with versions):

rocksdb-haskell-jprupp

Hackage CI License: BSD-3-Clause

Haskell bindings to RocksDB, the embedded persistent key-value store. The package on Hackage is rocksdb-haskell-jprupp; the module to import is Database.RocksDB.

The bindings follow the RocksDB C API closely. On top of it they add bracket-style resource management, so that databases, snapshots and iterators are always released, even when an exception is thrown.

Contents

Why these bindings

RocksDB is a storage engine: a library that keeps sorted keys and values in files on local disk, with fast writes, fast point lookups and fast ordered scans. There is no server to run. It is what you reach for when SQLite’s relational model is more than you need, and an in-memory map is not durable or big enough.

These bindings exist because the Haskoin Bitcoin indexer needed them. It keeps the whole Bitcoin block chain indexed by address, transaction and block, which is hundreds of gigabytes, and serves queries from many threads while new blocks are written. That use shaped the library:

  • it must be safe to share one database between many threads;
  • resources must never leak, because the process runs for a long time;
  • memory outside the Haskell heap must be boundable, because RocksDB’s defaults let it grow with the size of the database;
  • the hot paths, such as reading many keys at once, must cost as little as the C API allows.

The library is a fork of leveldb-haskell and of earlier RocksDB bindings, maintained as rocksdb-haskell-jprupp.

Design principles

  • Stay close to the C API. Functions map nearly one to one to RocksDB’s C functions, so RocksDB’s own documentation and tuning guides apply directly. Conveniences such as typed keys live in separate libraries (rocksdb-query). The two exceptions are iterChunk, which walks an iterator, and writePacked, which fills a batch, each in a C function of this package: the cost they remove is that of calling C from Haskell.
  • Scope every resource. Databases, snapshots and iterators are C objects that must be freed once and never touched again. withDB, withSnapshot and withIter free them when their block ends, even on an exception.
  • Plain IO. No monad transformer constraints. Lift into your own monad with liftIO.
  • Bytes in, bytes out. Keys and values are strict ByteStrings, which is what RocksDB stores. How you serialize them is your choice.
  • Don’t crash in C. RocksDB crashes the process if you misuse an iterator. The Haskell side checks first and turns those cases into no-ops or Nothing.
  • Expose what bounds memory. The few settings that turn RocksDB’s memory use into a fixed budget are first-class fields of Config.

Features

Area What you get
Basic access get, put, delete on strict ByteString keys and values
Batches Atomic write of many puts and deletes, across column families
Packed batches A batch of any size handed to RocksDB in a few foreign calls, from your buffers
Merges Change a value without reading it: merge records, and a merge operator in C
Multi-get multiGet reads many keys in one call to RocksDB’s MultiGet
Column families Open a database with several independent key spaces; add new ones on open
Snapshots Consistent point-in-time reads across many queries and iterators
Iterators Ordered scans, forwards and backwards, optionally pinned to a snapshot
Bounds Iterators that stop at a key or at the end of a prefix, without reading past it
Chunks Hundreds of entries from one foreign call, in either direction, within a prefix
Tuning Bloom filters, fixed-length prefix extractor, shared LRU block cache
Limits Memtable and WAL size limits, open file limit
Administration Properties (memory use, statistics, SST files), approximate sizes, destroy, repair

What is not included

  • No transactions (TransactionDB, OptimisticTransactionDB), TTL databases, backups or checkpoints.
  • No merge operators written in Haskell: RocksDB calls a merge operator from its own threads, so it is a C function.
  • No DeleteRange or SingleDelete.
  • No custom comparators: keys always sort by their bytes.

Pull requests that add bindings for these are welcome.

Installation

System library

You need the RocksDB C library and its headers.

# Debian and Ubuntu
sudo apt-get install librocksdb-dev

# Fedora
sudo dnf install rocksdb-devel

# macOS with Homebrew
brew install rocksdb

# Nix
nix-shell -p rocksdb

The library links against -lrocksdb. If RocksDB is installed in a non-standard place, pass its directories with --extra-include-dirs and --extra-lib-dirs (Cabal or Stack).

The headers are needed to build, not only the library: the binding of the Bloom filter policy is compiled against rocksdb/c.h, because RocksDB changed the type of its argument in 6.24 and the header says which one is installed.

Check the library is found

ls /usr/include/rocksdb/c.h        # or $(brew --prefix)/include/rocksdb/c.h
pkg-config --modversion rocksdb    # prints the installed version, if registered

Step by step with Stack

  1. Install Stack and the RocksDB library (above).

  2. Create a project:

    stack new rocks-demo simple
    cd rocks-demo
    
  3. Add rocksdb-haskell-jprupp, bytestring and data-default to build-depends in rocks-demo.cabal.

  4. Put the quick start program in src/Main.hs and run it:

    stack run
    

    It creates a database in /tmp/example and prints what it reads back.

If the build fails with cannot find -lrocksdb or rocksdb/c.h: No such file, the C library is not where the compiler looks. Install the development package, or pass --extra-include-dirs and --extra-lib-dirs.

With Cabal

build-depends:
    base
  , bytestring
  , data-default
  , rocksdb-haskell-jprupp

data-default gives you def for the default Config.

With Stack

The package is in Stackage nightly snapshots. If your resolver does not have it, or has an older version, add it to stack.yaml:

extra-deps:
  - rocksdb-haskell-jprupp-2.8.0

With stack --nix, add RocksDB to the Nix packages:

nix:
  packages:
    - rocksdb

Quick start

{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}

import Data.Default (def)
import Database.RocksDB

main :: IO ()
main = do
  let cfg = def {createIfMissing = True, bloomFilter = True}
  withDB "/tmp/example" cfg $ \db -> do
    put db "hello" "world"
    get db "hello" >>= print -- Just "world"

    -- atomic batch
    write db [Put "a" "1", Put "b" "2", Del "hello"]

    -- many keys at once
    multiGet db ["a", "b", "c"] >>= print -- [Just "1",Just "2",Nothing]

    -- every entry, in key order
    withIter db $ \it -> do
      iterFirst it
      let loop =
            iterEntry it >>= \case
              Nothing -> return ()
              Just kv -> print kv >> iterNext it >> loop
      loop

Tutorial: an event log

This builds a small append-only event log, to show how a RocksDB schema is designed around sorted keys.

1. Design the keys

RocksDB sorts keys by their bytes. Records read together should therefore sort together. Events are read per user and in time order, so the key is the user followed by a big-endian timestamp, which sorts numerically:

{-# LANGUAGE OverloadedStrings #-}

import Data.ByteString (ByteString)
import qualified Data.ByteString as B
import qualified Data.ByteString.Builder as BB
import qualified Data.ByteString.Lazy as BL
import Data.Word (Word64)

eventKey :: ByteString -> Word64 -> ByteString
eventKey user time =
  BL.toStrict . BB.toLazyByteString $
    BB.byteString user <> BB.char7 '/' <> BB.word64BE time

2. Write atomically

An event and a per-user counter must stay consistent, so they go in one batch:

import Database.RocksDB

logEvent :: DB -> ByteString -> Word64 -> ByteString -> Int -> IO ()
logEvent db user time payload count =
  write db
    [ Put (eventKey user time) payload,
      Put ("count/" <> user) (BL.toStrict (BB.toLazyByteString (BB.intDec count)))
    ]

If the program dies halfway, either both writes are there or neither is.

3. Read a range with an iterator

All events of a user sort together, so one seek and a short scan read them:

eventsOf :: DB -> ByteString -> IO [ByteString]
eventsOf db user = withIter db $ \it -> do
  let prefix = user <> "/"
  iterSeek it prefix
  let go acc = do
        mkv <- iterEntry it
        case mkv of
          Just (k, v) | prefix `B.isPrefixOf` k -> iterNext it >> go (v : acc)
          _ -> return (reverse acc)
  go []

A list of thousands of events is read faster in chunks: see Reading many entries in one call.

4. Read consistently with a snapshot

If another thread logs events while you read the counter and then the events, the two can disagree. Read both through a snapshot:

report :: DB -> ByteString -> IO (Maybe ByteString, [ByteString])
report db user = withSnapshot db $ \snap ->
  (,) <$> get snap ("count/" <> user) <*> eventsOf snap user

5. Tune for size

Once the log holds millions of events, open it with a memory budget and a Bloom filter for faster point lookups:

let cfg =
      def
        { createIfMissing = True,
          bloomFilter = True,
          maxFiles = Just 1000,
          blockCacheSize = Just (512 * 1024 * 1024),
          cacheIndexAndFilterBlocks = True
        }

Concepts

The database handle

withDB and withDBCF open a database, pass a DB to your function, and close it when the function returns or throws. The DB must not escape that function. It is safe to use from many threads at the same time.

Keys and values

Keys and values are strict ByteStrings. RocksDB sorts keys by their bytes, which is the order iterators walk them in. To store Haskell types, serialize them yourself, or use rocksdb-query.

Column families

A database can hold several key spaces called column families, each with its own configuration. Functions ending in CF act on a given column family; the others act on the default one. The handles are in columnFamilies db, in the order you listed them when opening.

Snapshots

withSnapshot db f passes f a DB that reads from a frozen view of the database. Reads and iterators through it do not see later writes. Writes through it still go to the live database.

Errors

Failures reported by RocksDB are thrown as IOError (built with userError). The message starts with the operation, such as put: ... or open: ....

Usage guide

Column families

withDBCF "/tmp/example-cf" cfg [("users", cfg), ("orders", cfg)] $ \db -> do
  let [users, orders] = columnFamilies db
  putCF db users "alice" "..."
  getCF db orders "1234" >>= print
  write db [PutCF users "bob" "...", DelCF orders "1234"]

On an empty directory, withDBCF creates the database and all the column families. On an existing database, RocksDB needs every column family that exists to be listed. To add a new one, list it and set createMissingColumnFamilies = True in the database Config.

Snapshots

withSnapshot db $ \snap -> do
  a <- get snap "a"
  b <- get snap "b" -- same point in time as "a"
  withIter snap $ \it -> iterFirst it >> iterEntry it >>= print

When the snapshot’s lifetime does not fit a scope, use createSnapshot and releaseSnapshot. withIterSnap and withIterSnapCF iterate over a raw Snapshot.

Iterators

An iterator starts out unpositioned. Move it with iterFirst, iterLast or iterSeek, then step with iterNext and iterPrev. iterKey, iterValue and iterEntry return Nothing once it moves past either end.

-- every key that starts with "user:"
withIter db $ \it -> do
  iterSeek it "user:"
  let go acc =
        iterEntry it >>= \case
          Just (k, v) | "user:" `B.isPrefixOf` k -> iterNext it >> go ((k, v) : acc)
          _ -> return (reverse acc)
  users <- go []
  iterGetError it >>= mapM_ (fail . show) -- tell the end apart from an error
  return users

createIterator, createIteratorSnap and destroyIterator manage iterators by hand.

Bounded iterators

The scan above stops when it reads a key that does not start with user:. RocksDB has to find that key first, and that can cost far more than the scan itself. A deleted key is not gone: it leaves a marker that stays until compaction drops it, and an iterator steps over every marker on its way to the next key that is alive. If the keys after user: were a queue that has been emptied, the scan reads the marker of every job that ever was in it.

withIterConfig creates an iterator that RocksDB stops at a bound, without looking past it:

-- every key that starts with "user:", and not one step further
withIterConfig db Nothing def {upperBound = prefixUpperBound "user:"} $ \it -> do
  iterSeek it "user:"
  let go acc =
        iterEntry it >>= \case
          Just kv -> iterNext it >> go (kv : acc)
          Nothing -> return (reverse acc)
  go []

The second argument is the column family, or Nothing for the default one. The IterConfig has four fields, all off in def:

Field Type What it does
upperBound Maybe ByteString The iterator stays below this key; the key itself is outside
lowerBound Maybe ByteString The iterator does not go below this key; the key itself is inside
prefixSameAsStart Bool The iterator stays within the prefix of the key it was sent to
totalOrderSeek Bool See every key in order, in a database that has a prefixLength

Which one to use:

  • upperBound, with prefixUpperBound, when each range gets its own iterator. It needs nothing from the configuration of the database, and the range can be any.
  • prefixSameAsStart when one iterator is sent to many ranges, seek after seek, which saves creating an iterator for each: the bound moves with every iterSeek. It needs a prefixLength in the Config, and each range has to be a whole prefix of that length. It also lets RocksDB use its prefix filters to skip the files that have nothing for the prefix. Every key, sought or stored, has to be at least as long as the prefix: see Things to watch out for.
  • totalOrderSeek to walk a database that has a prefixLength from end to end. Without it, an iterator there only promises the keys that share the prefix of the key it was sent to.
  • withIter when what follows the range is alive, or when there is no range. A bound costs a little when the iterator is created, and gains nothing there.

How much it matters, in this package’s own test: one live key, then 20,000 keys that were written and deleted, then another live key. A scan from just after the first one, which finds nothing, in microseconds:

Iterator Time
withIter 2,600
withIterConfig with an upperBound 2.5

createIteratorConfig manages such an iterator by hand. It returns the iterator together with the action that destroys it, which also frees the bounds: RocksDB keeps pointers to them for as long as the iterator lives.

Reading many entries in one call

iterNext may wait for the disk, so it is a foreign call of the kind that lets other Haskell threads run meanwhile. For that, the thread gives up its capability and has to get one back when the call returns. A scan that moves one step at a time pays this for every entry, and under load, when more threads want to run than there are capabilities, it waits each time.

iterChunk walks the iterator in C and returns up to some hundreds of entries from one call, with a flag that tells whether more follow:

-- every key that starts with "user:", in chunks
withIter db $ \it -> do
  iterSeek it "user:"
  let go acc = do
        (entries, more) <- iterChunk it def {chunkPrefix = "user:"}
        let acc' = entries : acc
        if more then go acc' else return (concat (reverse acc'))
  go []

The ChunkConfig says what one call reads:

Field Default What it does
chunkEntries 256 The most entries to read
chunkBytes 1 MiB Stop before keys and values take more; one entry is always read
chunkPrefix empty Stop at the first key that does not start with it
chunkBackwards False Go backwards, as iterPrev does

The iterator is left on the first entry that was not returned, so the next call continues there. A bound of the iterator, from withIterConfig, ends a chunk as it ends a scan by single steps. And iterChunk throws if the iterator stopped at an error, so there is no iterGetError to remember.

For a known number of entries there are short forms, which return just the list: iterEntries it n, iterEntriesBack it n, iterEntriesPrefix it prefix n and iterEntriesPrefixBack it prefix n. They have no limit of bytes, so a list shorter than n means that the scan is over.

-- a page of twenty users, starting after the last one seen
iterSeek it (lastSeen <> "\0")
page <- iterEntriesPrefix it "user:" 20

What it saves, in this package’s benchmark (stack bench): seconds for every thread to read 200,000 entries of 52 bytes, on four capabilities and four cores.

Scan 1 thread 16 threads
Single steps, loop not tail recursive 0.78 3.9
Single steps, loop tail recursive 0.090 0.50
Chunks of 256 entries 0.048 0.25

The first row is a loop that builds its list with (entry :) <$> go. The run-time system walks the stack of a thread at every foreign call that may block, and such a loop leaves a frame there for every entry read so far. Whatever you read with, keep the result in an argument of the loop.

With more cores than capabilities the chunks gain more, because a thread that is inside the call holds no capability: with twenty more cores free, the sixteen threads took 0.12 seconds in chunks and 0.83 in single steps.

Keep to single steps where each move depends on the entry just read, or where one entry is all you want: iterEntry after a seek reads it without moving the iterator.

Writing a large batch in a few calls

write hands a batch to RocksDB one operation at a time, and each is a foreign call of the kind that lets other Haskell threads run meanwhile. For a batch of a few operations that is nothing. For tens of thousands it adds up, and so do the ByteStrings, one for each key and each value.

A packed batch is the same operations written one after another into a few large buffers, the chunks. writePacked gives each chunk to a C function of this package, which adds its operations to the batch, and then writes the batch as write does: atomically, with one write.

-- the same as: write db ops
writePacked db [packOps ops]

packOps packs a list of BatchOp into one chunk. It is the simple way, and it copies every key and value once more. A program that serialises its keys and values itself can skip the list, the ByteStrings and the copy, and write records straight into a buffer of its own:

-- one record at a pointer: room for the header, the key, the value, and
-- then the header, now that the lengths are known
packRecord :: Ptr Word8 -> ColumnFamily -> IO (Ptr Word8)
packRecord ptr cf = do
  let body = ptr `plusPtr` packedHeaderSize
  keyLen <- serialiseKey body
  valueLen <- serialiseValue (body `plusPtr` keyLen)
  pokePackedHeader ptr (Just cf) PackedPut keyLen valueLen
  return (body `plusPtr` (keyLen + valueLen))

Here serialiseKey and serialiseValue stand for whatever writes your bytes at a pointer and says how many. The next record goes where this one ended: records need no alignment and nothing goes between them. A chunk is then a ByteString over the part of the buffer that holds whole records.

What to know:

  • All the chunks are one batch. writePacked db chunks applies the records of every chunk in order, atomically, with one write. Chunks only decide how much RocksDB is handed in one call: a few hundred kilobytes is plenty. A program that packs on several cores gives the chunks of each core, one core after another.
  • A chunk holds whole records. A record cannot start in one chunk and end in the next.
  • A chunk is checked before it is used. One that is cut short, or whose lengths say more than it holds, is refused with an exception, and nothing of the batch is written. The check cannot cover the column family handle of a record.
  • A chunk is only good here and now. It holds column family handles, which are pointers of this process and of the open database.
  • A record can be a merge. PackedMerge adds a merge record, as Merge does in a BatchOp, for a column family that has a merge operator: see Merge records.

The layout of a record, for a program that writes them by other means, is in cbits/rocksdb_haskell.h. It is this package’s own format, and what a chunk becomes is a series of calls to RocksDB’s C API. It is not RocksDB’s internal encoding of a batch, which a binding should not depend on.

What it saves, in this package’s benchmark (stack bench rocksdb-haskell-jprupp:bench:batch): milliseconds to write a batch of 65,000 operations and 6 MB over four column families, on four capabilities and four cores, alone and next to eight threads that compute.

Way to write the batch Alone Next to busy threads
write: a call for each operation 48 420
writePacked, with packOps in the time 28 115
writePacked, chunks packed beforehand 22 28

All three end with the same write, which takes about 20 ms here: the rest is the handing over. Alone, a thread gets its capability back at once after each call, and still pays 400 ns an operation. Next to threads that want the capabilities too, as in a server under load, it waits for one after every call. packOps is work in Haskell, which has to share the capabilities with those threads like any other.

With all 24 hardware threads of the machine free, the batch took write 1,160 ms next to the busy threads, and 31 ms from packed chunks.

Keep to write for small batches: it is as fast there, and simpler.

Merge records: change a value without reading it

A put replaces the value of a key. A program that wants to change a value, to add to a counter or to add an entry to a list, has to read it first. For a batch of thousands of such changes those reads are most of the work, and each may wait for the disk.

A merge record is a third kind of write, next to a put and a delete. It says how the value changes, not what it becomes: “add 5”, “add this entry”. RocksDB stores the record as it is and reads nothing.

The work is done later by the merge operator of the column family: a function that takes the value that the key had and the records written since, oldest first, and gives the value that the key has now. RocksDB calls it when the key is read, and when a compaction brings the records and the value together. After that the key holds a plain value again.

-- a column family of counters, with RocksDB's own operator for them
let counters = def {mergeOperator = Just UInt64Add}
withDBCF "/tmp/example-merge" def {createIfMissing = True} [("hits", counters)] $ \db -> do
  let [hits] = columnFamilies db
      five = BS.pack [5, 0, 0, 0, 0, 0, 0, 0] -- eight bytes, least significant first
  mergeCF db hits "home" five -- no read
  mergeCF db hits "home" five
  getCF db hits "home" >>= print -- Just "\n\NUL\NUL\NUL\NUL\NUL\NUL\NUL": ten
  write db [MergeCF hits "home" five, MergeCF hits "about" five]

What to know:

  • Readers see values. get, multiGet and iterators return merged values, in a snapshot too. For a key with records pending they pay for the merge, and for a look into each table file that holds one of the records.
  • A put or a delete hides the records before it, like any older value. A record written after a delete, or to a key that never had a value, is merged with no value: the operator decides what that means.
  • The operator is set when the database is opened, with the mergeOperator field of the Config of the column family. The one in the Config of the database is for the default column family.
  • UInt64Add is RocksDB’s own: values and records are unsigned 64-bit numbers in eight bytes, least significant first, added modulo 2^64.
  • MergeFunctions makes an operator of a name and two C functions: one that works out a value (the full merge), and one that makes a single record of several (the partial merge), which RocksDB uses to shorten a list of records before it has the value. They are C because RocksDB calls them from its own threads, during reads, flushes and compactions.
/* my_merge.c, listed under c-sources of your package */
char *my_full_merge(void *state, const char *key, size_t key_length,
                    const char *existing_value, size_t existing_value_length,
                    const char *const *operands_list,
                    const size_t *operands_list_length, int num_operands,
                    unsigned char *success, size_t *new_value_length);
char *my_partial_merge(void *state, const char *key, size_t key_length,
                       const char *const *operands_list,
                       const size_t *operands_list_length, int num_operands,
                       unsigned char *success, size_t *new_value_length);
foreign import ccall "&my_full_merge" myFullMerge :: FunPtr FullMerge
foreign import ccall "&my_partial_merge" myPartialMerge :: FunPtr PartialMerge

mine :: Config
mine = def {mergeOperator = Just (MergeFunctions "my-merge" myFullMerge myPartialMerge)}

A function returns the new bytes in memory from malloc, which RocksDB frees, with their length in *new_value_length and 1 in *success. existing_value is NULL when the key has no value. To fail, it stores 0 in both and returns NULL: the read of that key then throws. A partial merge may always fail, and RocksDB then keeps the records as they are. cbits/rocksdb_haskell.h has the two types and the rules in full, and test/cbits/merge_test.c is a small operator to start from.

Every program that opens the database must set the same operator. RocksDB opens a column family that has merge records without one, and this is what version 11.1 was seen to do then:

Where the merge records are What happens without the operator
In the write-ahead log Dropped when the database is opened, without an error: lost
In table files Kept, also through a compaction, but they cannot be read
Read: records and no value get throws; an iterator stops there with an error
Read: records over a value get and multiGet crash the process
Writing a merge record Refused with an error

So when a program starts to write merge records to a database that older versions of it can open, make sure that they no longer can. A new column family is enough: RocksDB refuses to open a database without all its column families, and does so before it reads the write-ahead log.

Properties and maintenance

getProperty db Stats >>= mapM_ BS.putStr
getProperty db BlockCacheUsage
getPropertyCF db cf CurSizeAllMemTables
approximateSize db ("a", "z")

destroy deletes a closed database and repair tries to recover a damaged one. Both take a raw Options pointer, which the public API does not let you build yet, so in practice they are only usable from inside the library.

Configuration

Config has a Default instance: all flags off and all sizes left to RocksDB.

Field Scope Meaning
createIfMissing database Create the database if it does not exist
errorIfExists database Fail if the database already exists
paranoidChecks database Check data aggressively and stop on corruption
createMissingColumnFamilies database Create column families given to withDBCF that are missing
maxFiles database Maximum number of open files (default unlimited)
dbWriteBufferSize database Bytes of all memtables together (default unlimited)
blockCacheSize database Bytes of an LRU block cache shared by all column families
maxTotalWalSize database Bytes of write-ahead log that trigger flushes
maxBackgroundJobs database Threads for flushes and compactions together (RocksDB default 2)
writeBufferSize column family Bytes per memtable (RocksDB default 64 MiB)
bloomFilter column family Full Bloom filter with 10 bits per key
prefixLength column family Length of a fixed key prefix for the prefix extractor
cacheIndexAndFilterBlocks column family Keep index and filter blocks in the block cache
partitionIndexAndFilters column family Write index and filter in 4 KiB pieces, loaded as needed
mergeOperator column family How merge records are combined with a value (default: none)
disableAutoCompactions column family No background compaction: for bulk loads, tests and benchmarks

Database-wide fields are ignored in the configuration of a column family. The column family fields of the database’s own Config are for its default column family.

Choosing a configuration

RocksDB’s defaults are tuned for small databases. Its memory use outside the Haskell heap comes from three places:

  1. Memtables, where recent writes collect before being flushed to disk. Capped per column family by writeBufferSize and in total by dbWriteBufferSize.
  2. Index and filter blocks of every open table file. Without cacheIndexAndFilterBlocks they live outside any cache and grow with the number of files; maxFiles limits how many are open.
  3. The block cache of recently read data. blockCacheSize makes one cache shared by all column families, so memory goes where reads are.

A reasonable starting point for a large database is a block cache of a few hundred megabytes to a few gigabytes, cacheIndexAndFilterBlocks = True, maxFiles of a few thousand, and dbWriteBufferSize of a few hundred megabytes. Enable bloomFilter when most reads are point lookups of keys that may not exist. A filter takes about 1.25 bytes per key, held with the index blocks: in the block cache with cacheIndexAndFilterBlocks, in memory outside it otherwise. A filter belongs to a table file and is written with it, so turning bloomFilter on for an existing database only covers the files written from then on; the others get a filter when compaction rewrites them.

For a large database, setting maxFiles, blockCacheSize, cacheIndexAndFilterBlocks and dbWriteBufferSize puts a ceiling on the memory RocksDB uses outside the Haskell heap. Without them, index and filter blocks of every open file and every memtable can grow without bound.

Module overview

Database.RocksDB is the only exposed module. It re-exports:

Part Contents
Types DB, Config, BatchOp, ColumnFamily, Snapshot, Iterator, Property, Range
Merge operators MergeOperator, FullMerge, PartialMerge
Opening withDB, withDBCF
Reading and writing get, getCF, multiGet, multiGetCF, put, putCF, delete, deleteCF, write
Merging merge, mergeCF, and Merge, MergeCF in a BatchOp
Packed batches writePacked, packOps, PackedOp, packedHeaderSize, pokePackedHeader
Snapshots withSnapshot, createSnapshot, releaseSnapshot
Creating iterators withIter, withIterCF, withIterSnap, withIterSnapCF
Bounded iterators IterConfig, withIterConfig, createIteratorConfig, prefixUpperBound
Moving iterators iterSeek, iterFirst, iterLast, iterNext, iterPrev
Reading from iterators iterKey, iterValue, iterEntry, iterValid, iterGetError
Reading in chunks iterChunk, ChunkConfig
Reading a number of entries iterEntries, iterEntriesBack, iterEntriesPrefix, iterEntriesPrefixBack
Administration getProperty, getPropertyCF, approximateSize, compactRange, destroy, repair
Flushing flush

The FFI bindings (Database.RocksDB.C) and helpers (Database.RocksDB.Internal) are hidden modules. cbits/rocksdb_haskell.c is the one piece of C in the package: the functions behind iterChunk and writePacked, and the one that makes a merge operator of two functions.

Things to watch out for

  • Do not use a DB, snapshot or iterator after the function that gave it to you has returned. The C object is gone and RocksDB will crash.
  • Only one process can open a database at a time. A second open fails with a lock error.
  • Moving an invalid iterator does nothing. Reposition it with iterSeek, iterFirst or iterLast.
  • An iterator that becomes invalid because of corruption looks like one that reached the end. Check iterGetError.
  • A scan that stops by looking at the key it has reached steps over every deleted key that follows its range. Where keys come and go, give the iterator a bound with withIterConfig.
  • In a database with a prefixLength, an iterator that moves past the prefix it was sent to, or that starts with iterFirst or iterLast, may skip keys. Ask for totalOrderSeek, or stay within the prefix with prefixSameAsStart.
  • With prefixSameAsStart, no key may be shorter than the prefixLength: not the keys you seek, and not the keys in the database. RocksDB takes the prefix of a key without checking its length and reads the memory that follows a shorter one, so the iterator stops at an arbitrary place. The binding cannot check this, because an iterator does not know how it was made. Where keys can be shorter, use an upperBound.
  • With prefixSameAsStart, iterFirst and iterLast give the first and the last prefix of the database, and no more: the iterator stays within the prefix of the key it finds.
  • Destroy an iterator from createIteratorConfig with the action that came with it, not with destroyIterator, which would leave its bounds and read options allocated.
  • Calls into RocksDB are safe foreign calls, so they do not block other Haskell threads, but each one costs more than an unsafe call, and much more when threads compete for capabilities. A scan by single steps makes one for every entry: read ranges with iterChunk. A write makes one for every operation: write batches of thousands with writePacked.
  • A chunk for writePacked is checked for its lengths, not for its column family handles. A handle that is not of the open database crashes the process, as it does in PutCF.
  • A loop over an iterator that is not tail recursive gets slower with every entry, because the run-time system walks the stack at each foreign call. Keep what has been read in an argument of the loop.
  • iterChunk and the iterEntries functions leave the iterator on the first entry that they did not return, not on the last one that they did.
  • An iterChunk with a chunkPrefix still moves the iterator to the first key outside the prefix, as a loop written by hand does. Where deleted keys follow the range, the iterator needs a bound as well.
  • A column family with merge records must get its mergeOperator from every program that opens the database. Without it RocksDB drops the records that are still in its write-ahead log, and a read of a key that has a value under merge records crashes the process: see Merge records.
  • A merge operator is called from RocksDB’s threads, several at a time. It must be a C function that keeps no state between calls, and must return memory from malloc.
  • Releases before 2.6.2 built no Bloom filters on RocksDB 6.24 or later, whatever bloomFilter said: they passed the bits per key as an integer to a function that had come to take a double. A database written by one of them has no filters in its table files. After upgrading, files get them as they are written, and the memory the filters need comes with them. The size of the filters of a column family is in the rocksdb.aggregated-table-properties property, as filter block size.
  • The compiler does not check a foreign import against RocksDB’s header. A binding with the wrong type builds and runs, with garbage in place of the argument; the two fixed in 2.6.2 went unnoticed for years.

Development

Stack

git clone https://github.com/jprupp/rocksdb-haskell.git
cd rocksdb-haskell

stack build                        # build the library
stack test                         # run the tests
stack bench                        # time scans and large batches
stack haddock --no-haddock-deps    # build the API documentation

With stack --nix, the nix section of stack.yaml provides RocksDB.

bench/Scan.hs times a scan of 200,000 entries by single steps and in chunks, from one thread and from sixteen on four capabilities. bench/Batch.hs times a batch of 65,000 operations written with write and with writePacked, alone and next to threads that compute. To see what a machine with four cores does, confine them: taskset -c 0-3 stack bench. To run one of the two: stack bench rocksdb-haskell-jprupp:bench:batch.

Nix

nix develop
cabal build all
cabal test all

The shell includes GHC 9.8.4, cabal, haskell-language-server, fourmolu, hlint and hoogle.

package.yaml is the source of the package description. Change it rather than rocksdb-haskell-jprupp.cabal, which hpack generates.

The tests are in test/Spec.hs and use hspec and QuickCheck. Each test runs against a fresh database in a temporary directory, so they need RocksDB installed and nothing else. There are two kinds:

  • Examples pin down one behaviour each: what a function returns, when it throws, what a setting changes in the files RocksDB writes. A test of a setting checks an effect that can be observed, such as the size of the filter blocks, because a wrong binding can do nothing without failing.
  • Properties apply random writes to the database and to a Map, and require that every way of reading the database agrees with the map: point reads, multiGet, iteration in both directions, seeks, snapshots and column families. Chunks are held to what single steps read, with random limits, prefixes and bounds. The keys are short and drawn from the bytes a binding is most likely to mishandle, zero and 255, so that operations often meet on the same key.

destroy and repair have no tests: they take the Options of the C API, which this package gives no way to create.

Contributing

Pull requests are welcome.

  • Format new code with fourmolu, which the Nix shell provides.
  • Add tests for the behaviour you change.
  • When you add or change a binding in Database.RocksDB.C, read the prototype in rocksdb/c.h argument by argument: nothing else checks the Haskell type against it. If the type of an argument differs between versions of RocksDB, import the function with capi.
  • Describe the change in CHANGELOG.md. Versions follow the Haskell Package Versioning Policy.

Related projects

  • rocksdb-query stores and queries Haskell types with these bindings.
  • haskoin-store is a Bitcoin block chain indexer built on them.

This library started as a fork of leveldb-haskell and of earlier RocksDB bindings. See AUTHORS.

License

rocksdb-haskell-jprupp is released under the BSD 3-Clause License.

Changes

2.8.0

  • maxBackgroundJobs, a new field of Config: how many threads RocksDB may use at once to flush memtables and to compact files, where its default is two, one of each. A database that has a backlog of compactions, after a bulk load for instance, works through it with one thread unless it is given more. More threads only help where the disk and the cores have room for them: a compaction that already waits for a slow disk is not made faster by a second one, and both take from the reads.

  • partitionIndexAndFilters, another new field of Config: the index and the filter of each table file are written in pieces of 4 KiB under a small index of their own. Each is otherwise one block that grows with the file, megabytes for a file of 64 MiB with a filter, and a database that has more of them than its block cache holds loads and drops them at nearly every lookup. Measured on a database of 1.6 TB with a cache of 1 GiB: the cache held the index and filter of 400 of 32,000 files and three megabytes of data, and a lookup took 0.8 ms of processor time on an idle disk. With pieces, a lookup loads the two that it needs. It applies to files written from then on; existing files keep their layout until compacted.

  • compactRange writes again every table file of a range of keys, those of the last level included, so that the files a database already has get the layout of its present Config: a Bloom filter, or partitionIndexAndFilters. RocksDB’s own compactions rarely reach the last level, where most of the data is.

  • Many entries of an iterator in one foreign call. iterNext and iterPrev can wait for the disk, so each is a call for which the Haskell thread gives up its capability and has to get one back, and a scan made one for every entry. iterChunk walks the iterator in C, in a small function that now comes with the package (cbits/rocksdb_haskell.c), and returns up to 256 entries or 1 MiB from one call, forwards or backwards, within a prefix if one is given, with a flag that tells whether more follow. A ChunkConfig sets those limits. iterEntries, iterEntriesBack, iterEntriesPrefix and iterEntriesPrefixBack are short forms for a known number of entries. All of them leave the iterator on the first entry that they did not return, respect the bounds of an IterConfig, and throw if the iterator stopped at an error. The single steps are unchanged.

    Seconds for every thread to read 200,000 entries of 52 bytes, on four capabilities and four cores (stack bench):

    Scan 1 thread 16 threads
    Single steps, loop not tail recursive 0.78 3.9
    Single steps, loop tail recursive 0.090 0.50
    Chunks of 256 entries 0.048 0.25

    With all 24 hardware threads of the machine free, the sixteen threads took 4.4, 0.83 and 0.12 seconds: threads inside the call hold no capability, so chunks gain more where there are more cores than capabilities. The first row is a loop that returns its list with (entry :) <$> go; the run-time system walks the stack at every such call, and that loop leaves a frame on it for every entry.

  • A large batch in a few foreign calls. write hands RocksDB the operations of a batch one call at a time, each a call for which the Haskell thread gives up its capability and has to get one back. For the 65,000 keys of a Bitcoin block that was more than half of the time of the batch, and nearly all of it in a program whose other threads were busy. writePacked takes the batch as chunks of packed records, and a second function of cbits/rocksdb_haskell.c adds the operations of a chunk to the batch in one call. The batch is then written as before: atomically, with one write, however many chunks it came in.

    packOps packs a list of BatchOp into a chunk. A program that serialises its keys and values itself can write records straight into its own buffers instead, with no ByteString for each: it leaves packedHeaderSize bytes, serialises the key and the value after them, and fills in the header with pokePackedHeader once the lengths are known. Records are not aligned, so the next one starts where the last one ended.

    A chunk is checked in C before anything of it is used. One that is cut short, that has a length reaching past its end, an operation that does not exist or a delete with a value, makes writePacked throw, with nothing written.

    A record can also ask for a merge (PackedMerge), which is handed to rocksdb_writebatch_merge. The binding still has no way to give a column family a merge operator, so RocksDB refuses such a batch; the format has it so that it need not change later.

    The format is this package’s own. It is not RocksDB’s internal encoding of a batch (rocksdb_writebatch_create_from), which would save a copy and tie the binding to the insides of RocksDB.

    Milliseconds to write a batch of 65,000 operations and 6 MB over four column families, on four capabilities and four cores (stack bench rocksdb-haskell-jprupp:bench:batch):

    Way to write the batch Alone Next to 8 busy threads
    write: a call for each operation 48 420
    writePacked, with packOps in the time 28 115
    writePacked, chunks packed beforehand 22 28

    All three end with the same write, of about 20 ms. Alone, the calls of write cost 400 ns an operation. Next to threads that compute, the thread waits for a capability after each of its 65,000 calls. With all 24 hardware threads of the machine free, that took write 1,160 ms, and packed chunks 31.

  • Merge records and merge operators. A merge record is a third kind of write, next to a put and a delete: it says how a value changes, and RocksDB stores it without reading the value. The merge operator of the column family works out the value when the key is read, and when a compaction brings the records and the value together. A program that adds to counters or to lists by the thousand no longer reads each one first.

    • mergeOperator, a new field of Config, gives a column family its operator, or the default one when it is in the Config of the database. UInt64Add is RocksDB’s own, for counters. MergeFunctions makes one of a name and the addresses of two C functions, a full merge and a partial merge, with the types FullMerge and PartialMerge. They are C functions because RocksDB calls them from its own threads; cbits/rocksdb_haskell.h says what they must do, and a third function of cbits/rocksdb_haskell.c makes the operator around them.
    • Merge and MergeCF are new constructors of BatchOp, written by write and packed by packOps as PackedMerge records. merge and mergeCF write one record. Code that matches on every BatchOp has two more cases.
    • get, multiGet, iterators and chunks return merged values, and need no change.
    • A column family with merge records must be given its operator by every program that opens the database. RocksDB 11.1 opens it without: it then drops the merge records that are still in the write-ahead log, with no error, and a read of a key with a value under merge records crashes the process. The README has the whole table.
  • flush writes the memtable of a column family to a table file and waits for it, and disableAutoCompactions, a new field of Config, stops RocksDB from compacting a column family in the background. They are for tests and benchmarks that need data in table files, a file for each flush, and for bulk loads that end with compactRange.

  • The Config record has new fields, so code that builds one without def has to set them; code that starts from def needs no change. This is why the version is 2.8.0 and not 2.7.2.

2.7.1

  • Documentation and tests only; the library behaves as in 2.7.0.
  • The documentation of prefixSameAsStart was wrong on one point and silent on another. It said that the option does nothing after iterFirst or iterLast: it does, the iterator stays within the prefix of the key that it finds there. And it did not say that every key, sought or stored, has to be at least as long as the prefixLength: RocksDB takes the prefix of a key without checking its length, and reads past the end of a shorter one. Both are now in the Haddock and in the README, and the first has tests.

2.7.0

  • Bounded iterators. withIterConfig creates an iterator that RocksDB itself stops, as an IterConfig says: at a lowerBound or an upperBound, or at the end of the prefix that it was sent to, with prefixSameAsStart. Until now a scan of a range could only stop by reading the first key past it, and when the keys past it had been deleted, RocksDB stepped over the marker of every one to find that key. A scan of an empty range followed by 20,000 deleted keys took 2,600 microseconds; with an upper bound it takes 2.5. prefixUpperBound gives the bound that ends a prefix, and createIteratorConfig manages such an iterator by hand.
  • totalOrderSeek, also in IterConfig, walks every key in order in a database that has a prefixLength, where a plain iterator only promises the keys of one prefix.
  • The DB record has a new field, snapshot, with the snapshot that its read options read from. Code that builds a DB by hand has to set it; code that gets one from withDB, withDBCF or withSnapshot needs no change. This is why the version is 2.7.0 and not 2.6.5.

2.6.4

  • Iterators read rows about three times faster. iterValid, iterKey and iterValue now reach RocksDB through unsafe foreign calls. The three only read what the iterator already holds, and reading one row with iterEntry and iterNext makes five of them; as safe calls each cost about 0.4 microseconds. Walking 700,000 rows took 2.8 microseconds a row before and 0.8 after. The calls that move an iterator, which can read from disk, are safe calls as before.

2.6.3

  • Documentation only; the library behaves as in 2.6.2.
  • References in the Haddock that named both a type and its constructor, such as DB and Config, now say that they mean the type, and two that named nothing in scope are fixed: the documentation builds without warnings about ambiguous or unresolved identifiers.
  • The documentation of withIter said to set a snapshot in the read options, which the API has no way to do. It now says how: create the iterator from the DB that withSnapshot provides.
  • Remove an unused import.
  • The test suite grows from 17 tests to 51. It now covers deletes, batches, withSnapshot, iteration backwards, empty keys and values, binary data, and the errors of opening a database, none of which had a test. Six of the tests are QuickCheck properties that compare the database with a Map after random writes. The suite also compiles without warnings.
  • A badge in the README shows the state of CI on master.

2.6.2

  • Build the Bloom filters that bloomFilter asks for. The bits per key were passed to rocksdb_filterpolicy_create_bloom_full as an integer, which is what RocksDB took until 6.23; since 6.24 the function takes a double. An integer and a double travel in different registers, so RocksDB read something else as the bits per key and built no filters: table files had filter block size=0, and a lookup of an absent key read a data block in every file that could hold it. Nothing failed, and no test noticed. In a measurement with 2,600,000 keys, such a lookup took about 5.5 microseconds without the filters and 1.5 with them.

    The function is now imported with the capi calling convention, through a stub compiled against rocksdb/c.h, so the C compiler passes the argument as the installed RocksDB declares it, before or after 6.24. The headers were already needed to install RocksDB for development; they are now needed to build this package too.

    Databases opened with bloomFilter start building filters when they are opened with this version: for the table files written from then on, at about 1.25 bytes per key. Files written before have none until compaction rewrites them.

  • Pass the error pointer that rocksdb_approximate_sizes takes since RocksDB 6.16. approximateSize called it with one argument too few, leaving RocksDB to read a pointer that was never passed, and to write an error message through it if the call failed. It now throws an IOError on failure.

  • Both were found by comparing every foreign import with the rocksdb/c.h of RocksDB 11.1: the other 70 match.

  • Tests that table files have filters when asked for and not otherwise, and of approximateSize, which had none.

2.6.1

  • Documentation, lots of it, with detailed explanations.

2.6.0

  • Add multiGet and multiGetCF, which read the values of many keys in one call to RocksDB’s MultiGet.
  • Add blockCacheSize to Config: the bytes of a least-recently-used block cache shared by the database and all its column families. Without it RocksDB gives each column family its own default cache, of 32 MiB in RocksDB 8.
  • Add cacheIndexAndFilterBlocks to Config, to keep the index and filter blocks of a column family in the block cache, at high priority, with those of level 0 pinned, instead of in memory outside the cache for every open file.
  • Stop leaking the block-based table options of every database and column family opened with a Bloom filter.

2.5.0

  • Add the EstimateTableReadersMem, CurSizeAllMemTables and BlockCacheUsage properties, and PropertyName for any other property by its name.
  • Add getPropertyCF to read a property of a column family.
  • Add maxTotalWalSize (max_total_wal_size) to Config, to bound the write-ahead log.
  • Add createMissingColumnFamilies (create_missing_column_families) to Config, to add column families to an existing database.
  • Destroy the handle of the default column family when withDBCF opens an existing database. It was leaked, and RocksDB built with assertions aborted when the database was closed.

2.4.0

  • Add writeBufferSize (write_buffer_size) and dbWriteBufferSize (db_write_buffer_size) to Config, to bound memtable memory.

2.3.1

  • Do not call next/prev on invalid iterators (fixes assertion failure).

2.3.0

  • Remove UnliftIO and remain in IO monad.

2.2.0

  • Move Iterator module to plain IO.
  • Properly manage read options in createSnapshot.
  • Add nix flake for development.
  • API allows manual database lifecycle management.

2.1.7

  • Simplify author name.

2.1.6

  • Refactor memory use further.
  • Improve tests.

2.1.5

  • Fix errors not releasing memory.

2.1.4

  • Add ResourceT support for snapshots and iterators.

2.1.3

  • Correct bug where database was ignoring prefix length config.

2.1.2

  • Correct bug when opening database with column families.

2.1.1

  • Expose ColumnFamily type.

2.1.0

  • Add support for column families.

2.0.0

  • Fork into a different project
  • Rewrite most of the code to make it faster and easier to use
  • Remove many features that I don’t need or want to maintain

1.0.1

  • Add support for UTF-8 characters in a database’s path

1.0.0

  • First version by Serokell

0.3x.0

  • ResourceT is no longer compulsory

0.2.0

  • requires LevelDB v1.7
  • support for filter policy (LevelDB v1.5), either custom or using the built-in bloom filter implementation
  • write batch values no longer require a memcpy to be early-finalizer-safe (introduced in 0.1.1)

0.1.0

  • memory (foreign pointers) is managed through ResourceT. Note that this requires to lift monadic actions inside the MonadResource monad, see the examples.
  • links against shared library (LevelDB v1.3 or higher)
  • LevelDB 1.3 API fully supported (including custom comparators, excluding custom environments)

0.0.x

  • experimental releases