One problem you’ll sometimes encounter when working with cloud services from AWS, Azure or Google cloud is that developing locally can be made more difficult when working with services that do not have a standardized interface with an implementation readily available for local installation. For instance, when working with a pub/sub system that is compatible with Kafka you can just install a minimal Kafka cluster locally and all is good. But what to do when the APIs offered by the service you need are not standardized? That’s when emulators come in. In the rest of this post I’m going to focus on Azure, since that’s what I’m working with most often.

Perhaps the most commonly known emulator for Azure services is Azurite which emulates the services offered by Azure Storage services. I’ve used it for years and I’m still mostly happy with this emulator. The problem I’ll discuss in this post is indeed very real – I’ve seen this a few times already.

The problem

For .Net, Microsoft offers the Data Protection APIs which provide ways to protect and unprotect data, including key management and rotation. When building a service that runs in Azure (but anywhere, really) you can use Azure blob storage for persisting the keys and Azure Key Vault to protect those keys. Azurite therefore works well as local emulator for permanent storage of the protected keys. And Azure Key Vault Emulator takes care of emulating a Key Vault for key protection.

After first looking into this I’ve found a few emulators for Azure Key Vault, but most of them were old/outdated or abandoned/deprecated. I ended up forking one that was apparently maintained until just recently, the above mentioned Azure Key Vault Emulator. I did however quickly realize that the original code of this emulator was missing an important API that Azure Key Vault offers and which is used for Data Protection in .Net: the unwrapKey API. This API is used to decrypt the master key which is used to encrypt the derived keys used for data protection and is therefore essential for using the emulator in local development. Luckily this API is easy to implement, and so my fork of the emulator now implements this as needed. Here’s how you can use it in your local development too.

All technical details discussed in this post can be looked up in the repo holding the example on GitHub.

Dependency injection for data protection

Let’s start with the configuration for depdenency injection. When protecting or unprotecting data in code, it all starts with an instance of IDataProtectionProvider from which protectors can be created with dedicated purposes. In short, using the same purpose strings in different instances of protectors means that they can decrypt the same ciphertext, using different purposes means they cannot.

Setting up dependency injection to register the IDataProtectionProvider service is done with calls as in this example:

builder.Services
    // Add the data protection provider service ...
    .AddDataProtection()
    // ... tell it to persist managed keys to blob storage ...
    .PersistKeysToAzureBlobStorage(DataProtection.KeysFactory)
    // ... and protect the managed keys with a key in Azure Key Vault.
    .ProtectKeysWithAzureKeyVault(
        builder.Configuration.GetValue<Uri>("DataProtection:KeyId"),
        new DefaultAzureCredential())
    ;

The extension methods used here, PersistKeysToAzureBlobStorage and ProtectKeysWithAzureKeyVault, can be found in the Nuget packages Azure.Extensions.AspNetCore.DataProtection.Blobs and Azure.Extensions.AspNetCore.DataProtection.Keys respectively.

That’s it. You’ll note a few missing details here though. For instance, what is that reference to DataProtection.KeysFactory all about? We’ll get to that shortly. But first, let’s just discuss the other important parts needed to configure for dependency injection.

Configuring Azure service clients

The Azure SKDs for .Net include some useful Nuget packages, including the package Microsoft.Extensions.Azure. Make sure you add that to your project in addition to the service clients needed. The example below shows how to set up clients for blob services plus using the DefaultAzureCredential class from the Azure.Identity Nuget package. That is really useful for situations like this where the credentials used to connect to Azure services (or emulators) are different depending on the environment. The DefaultAzureCredential class tries different ways to get credentials and access tokens, including but not limited to using managed identities, Visual Studio / Visual Studio Code extensions, or azcli when running locally.

builder.Services.AddAzureClients((options) =>
{
   options.UseCredential(new DefaultAzureCredential());
   options.AddBlobServiceClient(builder.Configuration.GetSection("BlobStorage"));
});

All we do here really is to instruct the Azure SDKs to use the best fitting credential provider and get the configuration for which blob service to use from the BlobStorage section in the configuration.

The data protection key factory

Above we’ve seen the reference to DataProtection.KeysFactory, so let’s look at that in more detail. It’s actually very straight forward: it uses the registered BlobServiceClient, curtesy of the call to AddBlobServiceClient above, to get a container client, which is used to get a blob client to work with the blob that should store the managed (and protected) keys.

internal static class DataProtection
{
    internal static BlobClient KeysFactory(IServiceProvider services)
    {
        var settings = services.GetRequiredService<IOptions<DataProtectionSettings>>().Value;
        var blobServiceClient = services.GetRequiredService<BlobServiceClient>();
        var containerClient = blobServiceClient.GetBlobContainerClient(settings.BlobContainer);
        return containerClient.GetBlobClient(settings.BlobName);
    }
}

internal sealed class DataProtectionSettings
{
    public string BlobContainer { get; set; } = "keys";
    public string BlobName { get; set; } = "keys.xml";
}

The service for IOptions<DataProtectionSettings> can be registered like this.

builder.Services.Configure<DataProtectionSettings>(
    builder.Configuration.GetSection("DataProtection"));

Now you can control which storage service, blob container and blob to use for data protection entirely through the configuration, thus allowing you to use the appsettings.<Environment>.json files for different environments and even setting these values through environment variables or on the command line. Beyond that, the key in Azure Key Vault to use for protecting the keys can be configured just as well. The code therefore never needs to change between different environments.

Configuration for local development

With that all said, what should you put in the appsettings.Development.json configuration file? It’s very simple:

{
  "DataProtection": {
    "BlobContainer": "localkeys",
    "BlobName": "local-keys.xml",
    "KeyId": "https://emulator.vault.azure.net:11001/keys/my-data-protection-key/e572d2447ff04d51a41ab933adba00fe"
  },
  "BlobStorage": {
    "ConnectionString": "UseDevelopmentStorage=true"
  }
}

The DataProtection:KeyId setting needs to point to a key’s specific version in the Azure Key Vault emulator. Please make sure you create that key first. You can use the Azure SDKs, the Swagger GUI on the emulator or any other tool that let’s you craft a correctly formed API request for that. The BlobContainer and BlobName settings in the same section could just as well be removed in favor of the default values, but they are shown here for completeness.

The BlobStorage section configures the connection details for the blob service client. For local development, we want to use the default connection string, so that’s all we configure here.

Configuration for production environment

What does that all look like in the appsettings.Production.json config file? Well, the only difference is in which Azure service endpoints you use. For example:

{
  "DataProtection": {
    "KeyId": "https://mykeyvault.vault.azure.net/keys/my-data-protection-key/2c61d9115cdc4e488607b2cb9ffc9326"
  },
  "BlobStorage": {
    "ServiceUri": "https://mystorageaccount.blob.core.windows.net/"
  }
}

Of course you’ll want to replace the above used mykeyvault and mystorageaccount service names with your own. And that is really all there is to this.

Differences in configuration sections

You may have noticed that the BlobStorage section for production does not configure a ConnectionString setting, but instead a setting called ServiceUri. This still works because the setup call to AddBlobServiceClient in dependency injection configuration looks for the best fitting constructor given the setting key/value pairs from the configuration section passed to it. Unfortunately, the public documentation of that method is very thin, but if you leave the section entirely empty, the exception you will get when the service is used will tell you which options you have. For the BlobServiceClient for instance, the exception message is like the following:

Unable to find matching constructor while trying to create an instance of BlobServiceClient.
Expected one of the follow sets of configuration parameters:
1. connectionString
2. serviceUri
3. serviceUri, credential:accountName, credential:accountKey
4. serviceUri, credential:signature
5. serviceUri

You’ll notice that option #1 is what we used for local development whereas option #2 is what we configured for production.

Summary

This post shows a few different things:

  • How to use the Azurite and Azure Key Vault Emulators for local development
  • How to use a single code-base with different configurations for environments

As a reminder, you can see all example code shown here on GitHub: rokeller/dotnet-data-protection.