One problem you’ll sometimes encounter when working with cloud services from AWS, Azure or Google cloud is that developing locally can be made more difficult when working with services that do not have a standardized interface with an implementation readily available for local installation. For instance, when working with a pub/sub system that is compatible with Kafka you can just install a minimal Kafka cluster locally and all is good. But what to do when the APIs offered by the service you need are not standardized? That’s when emulators come in. In the rest of this post I’m going to focus on Azure, since that’s what I’m working with most often.

I my previous post I introduced the Tea Key-Value Store. The store is designed for easy-to-use integration in existing processes without making any assumptions about the hosting process’ needs. In some cases, that is surely enough and gives developers just what they need to store semi-structured data with lookups through point queries.

However, one think I’ve wanted to implement for a while too was a write-ahead log (WAL) that can be use for recovery scenarios. The idea for a WAL is conceivably simple: before data is actually committed to persistent storage, a record describing that data is written to the WAL. The WAL itself is typically a pre-allocated file of some size that only ever gets appended to or deleted, but never overwritten. How much space needs to be pre-allocated depends on the usage as well as other factors like frequency of flushing data to disk or the size of typical records themselves. With the Tea Key-Value Store of course you can configure the size of the pre-allocated WAL should you select to use the WAL in the first place.

I keep a little black book with ideas for businesses or projects and sometimes also technology I want to learn more about. One of those things I wanted to learn more about was the inner workings of a key-value store. I wanted to know how to allow for virtually infinite growth of such a store without sacrificing read or write speeds or how to best organize the data on a disk.