I my previous post I introduced the Tea Key-Value Store. The store is designed for easy-to-use integration in existing processes without making any assumptions about the hosting process’ needs. In some cases, that is surely enough and gives developers just what they need to store semi-structured data with lookups through point queries.

However, one think I’ve wanted to implement for a while too was a write-ahead log (WAL) that can be use for recovery scenarios. The idea for a WAL is conceivably simple: before data is actually committed to persistent storage, a record describing that data is written to the WAL. The WAL itself is typically a pre-allocated file of some size that only ever gets appended to or deleted, but never overwritten. How much space needs to be pre-allocated depends on the usage as well as other factors like frequency of flushing data to disk or the size of typical records themselves. With the Tea Key-Value Store of course you can configure the size of the pre-allocated WAL should you select to use the WAL in the first place.

Anyway, since records are written to the WAL first, before writing them to the in-memory Key-Value Store, they can be recovered after a crash. It is important to remember that crashes really only ever affect the data written to the in-memory store: data already persisted to segments couldn’t be corrupted by a crash of the process, because segments are never overwritten. Really the only three situations where a segment could ever be corrupted are:

  1. The process crashes during the writing of the segment.
  2. The segment is overwritten by another process.
  3. The disk is corrupted, e.g. because of a hardware failure.

#1 above is also helped through recovery with the WAL, because the WAL is only cleaned up / rotated after a segment has successfully been committed to disk. #2 is something that the Tea Key-Value Store can only guard against by locking the files while the host process is running. If the host process isn’t running, other mechanisms from the file system may help, but the best guard is to just steer other processes away from the segment files in the first place. #3 isn’t something you can generally guard against only with software, but depending on your needs you’ll use a subset of combination of RAID and backups.

What this means though is that by configuring the Tea Key-Value Store to use a WAL, you can take advantage of automatic recovery after the process that uses the embedded store has crashed.

Configuring the WAL is as simple as this:

services
    .AddKeyValueStore<ulong, string>()
    .AddWriteAheadLog((settings) =>
    {
        // Which directory to put the write-ahead log files in.
        settings.LogDirectoryPath = ".wal";
        // The size (in bytes) to reserve upfront for write-ahead logs.
        settings.ReservedSize = 128 * 1024 * 1024; // 128 MiB
    });

Note that if the WAL needs to record data beyond the pre-allocated size, it currently does so in an inefficient way, by allocating block after block. It is therefore usually better to allocate more for the WAL than you think you’ll ever need.

If the WAL is configured for a key-value Store when it is created, it will automatically look for files for recovery and execute that recovery by replaying the records from the WAL to the in-memory store. From here, the process continues to work as always and depending on your policies, the in-memory key-value pairs will be persisted to segments.

Finally, as a reminder, you can look at the Tea Key-Value Store code on GitHub.