A very important point you must understand when it comes to working with Azure Functions is that that product is tightly bound to an Azure Storage account. You see, Azure Functions is a compute offering, when you provision that you are getting some CPU, some memory and a host to run the code on. The place your code resides is dependent on the tier you run, but let’s keep it simple and target consumption. In the server-less context, your code cannot remain on a server, aka host because of a whole lot of reason which I won’t go into. Well, maybe I’ll go into one reason, which is security. You don’t want to store or run a bunch of unrelated code on a single server, so each time the compute power is no longer needed, the host is wiped clean and put back into the pool for other server-less needs. OK, and a second reason is because we couldn’t call this a server-less option if the code was persistent and remained on server, that would be like dedicated then server-less. Anyway, the application source code is stored in an Azure Storage Account. There are currently 4 different features that exist within an Azure Storage Account and they are Containers (aka Blob), Files, Tables and Queues, see Figure 1.
Figure 1, Azure Storage Account, container, files, tables and queues
**Read this article about Azure Function Keys named: THE CASE OF THE INCONSISTENT AZURE FUNCTIONS HOST KEY, written by Michael Collier. It contains some very helpful information about this topic.
Part of the provisioning process for an Azure Function is to select an Azure Storage account, your choice is stored in the AzureWebJobsStorage application setting. The value of that setting is the connection string for the Azure Storage account, it contains a Account Key (key #1). The Account Key you see in the Connection string is accessed via the Access keys blade for the Azure Storage account. Those Access keys allow access to all of the 4 features shown in Figure 1 to whatever client that has it. For consumption mode your application is stored in File shares. If you access the File shares share via the portal it will look like the D:\home directory if you were to visit it in the KUDU/SCM portal. See Figure 2.
Figure 2, where is my Azure Function source code stored
Beware though, if you enable WEBSITE_RUN_FROM_PACKAGE then things get a bit more complicated. I won’t go into that because then I might loose my train of thought and go off into some other world and loose focus and not cover what this article is supposed to be about and that’s keys, right? I think that’s right, let me look at the title again, yep, that’s right. Great, now I lost my thought, I’ll be back later…
OK, I’m back.
There also exists the following additional keys:
- Host Keys (master and default) (#2 and #3)
- System Keys (#4)
- Function Key (default) (#5)
These keys are stored in the same Azure Storage account as the one configured into the AzureWebJobsStorage application setting, but this time you need to look into the azure-webjobs-secrets Container. Inside that Container, you will find a folder with the name of your Function App. Within that folder you will find a host.json and a json file for each of the Functions within the Function App. I have creatively named my Functions http and blob, therefore you see an http.json and a blob.json file. Figure 3.
Figure 3, Azure Function secrets and key and stuff
If you view the contents of those json files, you will see an encrypted version of the actual keys you would find in the App keys blade for the Function App. Figure 4 shows key #2, #3 and #4. If you click on the “Hidden value. Click to show value”, then it will do just that.
Figure 4, more about Azure Function keys, host, master, default and System keys
To have a look at the Function key #5, you can navigate to the Function and click on the Function Keys blade. Figure 5.
Figure 5 more about default Azure Function keys
So far, you should have learned about the tightly coupled dependencies between an Azure Function App and Azure Storage accounts. That’s clear right?
Here are a few experiences I have had.
Hosting all Function Apps on the same Storage Account
Firstly, I create a new Azure Function App almost every day, although I rarely delete them, I was thinking that if I did, that I would also need to delete the Storage Account which I created as well. Because of that, I have gotten into the habit of choosing to store all of my Function code and configurations into the same account. That means there are a lot of Function Apps with the same value for the AzureWebJobsStorage application setting. If I were to go into the Access keys blade for the Storage Account and regenerate the Access key, Figure 6, that would cause all the Function Apps to stop working. They would no longer have the correct Access key. That’s why there are two, so if you need to regenerate one, you can slowly update the clients with key2, once complete you can regenerate key1.
Figure 6, Azure Storage key regeneration
Don’t do that unless you are sure you know the impact, I am pretty sure it is not recommended to use the same Storage Account for all your Function Apps, there are some usage threshold limits in storage which you might hit too if you have a active Functions. So maybe don’t do that, but at least you know what can happen if you are already in that boat.
Deleting the json file from the container
If you look back at Figure 3 you will see the host.json file in the azure-webjobs-secrets Container. There is no referential integrity constraint so you can delete the JSON files even though the Function App and Functions need them still. I did do that and I saw no impact in my Functions. After some time, they appeared back by themselves, but I experience no downtime.
az resource invoke-action –resource-group FUNCTIONS –action syncfunctiontriggers –name secrets-host-keys –resource-type Microsoft.Web/sites
Figure 7, how to resync an Azure Function using Azure CLI
They also came back when I did a deployment. Probably because the deployment called SyncTriggers. However, when I manually ran it using Azure CLI, it only patricianly generated the keys, by that I mean only 1 of the JSON files showed up in the azure-webjobs-secrets Container. But, 100%, when I accessed the Function App App Keys blade, they (#2 and #3) got regenerated, also when I accessed the Function Key (#5) blade that one was regenerated as well. In all cases, it was the same key, so no clients needed to be updated after deleting the JSON files manually.
Deleting and recreating the Azure Function App
Like I mentioned earlier there is no link between the Azure Function and the Azure Storage, so if you delete the Function App the Storage Account remains. If you do this key #2, #3 and #5 will be regenerated, that is kind of expected. It might not be so expected that it uses the same Container folder, likely because it uses the name of the Function App, oh, when you recreate the the Function App, it has to be the same name in order to reproduce this. You will see the previous versions of the JSON files though, Figure 8.
Figure 8, Azure Function snapshot secrets and keys
In worst case, you can use the content of the previous file and replace the current one, or delete and rename the files to match the requirements. Like renaming http.snapshot.***.json to http.json after deleting the existing http.json file which contains the new encrypted #5 key. I didn’t have any System Keys to test, aka #4, so I have nothing to report on that front other than I would expect them to be regenerated too. I did notice that the key for my HTTP trigger shows up automatically, same for the HOST. But the BLOB one is only generated after I access the Function Keys blade for the BLOB Function, again, this likely triggers the SyncTrigger API.
Maliciously scrambling the host.json and http.json file
Just for fun I manually, partially destroyed the host.json file stored in the azure-webjobs-secrets Container. Inside those files there is a property named decryptionKeyId which I suppose is used to decrypt the encrypted key values. I picked that one the hack first. The host keys are missing, Figure 9, ah oh…
Figure 9, Azure Function Host keys are missing
I clicked the refresh button and they were regenerated. They also regenerated again by themselves after a few seconds. They were the same key value as well. Then I fat fingered the masterKey and functionKeys value, just for fun. Even though, I totally hacked up the files I was still able to invoke the Function. It means that the key must be cached someplace. I will shutdown the Function App for a while and see what happens when I am sure the worker has been released back into the consumption pool, in hopes that I will get another fresh one. I kind of feel guilty hoping it breaks…I’ll wait about 20 minutes, give me a chance to catch up on my email. Before I go, on another note, I saw this popup stating that you cannot view Functions or app keys while the Function App is stopped. Good to know, because I have seen a few cases like that before.
I waited 30 minutes and read about 60 emails.
I guess I should be happy it worked still. There must be a piece that I am missing but that would be an implementation detail and probably wouldn’t share that anyway. Either way, it looks like you are safe if the files are deleted, corrupted or you deploy a new Function app with the same name. In all cases, the files and keys are either regenerated or just the keys. That’s good news and something which helps administrators sleep at night.
The primary objectives of this post were to expose the link between the Function App and Storage. The second was to test a bit and see what happens when the files which apparently contain the key information gets removed or deleted. I think both have been fulfilled. If you ever get into a scenario where the keys are not working as expected, you now have a few more places to look before reaching out for some help.
Lastly, I did read something about the number 10. If you have 11 or more snapshot files, you might be hitting a limit. Consider removing some of them if you see that many. You can download them first, JIC you need them one day.