I am finding a particular liking to Python. I seem to be coding with it more and more. Especially when it comes to my Raspberry Pi cluster, the work on my 6502 project and my hacking labs. I thought that I should write a post about some details around coding an Event Hub triggered Azure Function with Python. I have chosen to do so using Visual Studio Code, with some help from the Azure Function Core Tools. I used Visual Studio Code to create the Project, the Azure Function and to code. I used Azure Function Core Tools to run and deploy the Azure Function. I’ll start off with the experience which took me the most time to get passed, it came down to cardinality.

You set the cardinality value in the function.json file, and when it is ‘many’ it means that event will be retrieved from the Event Hub in batches. When cardinality is set to ‘one’, you might can guess that it means an event is retrieved from the Event Hub one at a time. Note: this applies to non-C# languages. Down towards the bottom of this post you will see some of the exception I got and what I did to resolve them, in the case of cardinality and others. But here is the deal. When I created my Event Hub triggered Azure Function cardinality was set to many, and the template method was shown as here.

def main(event: func.EventHubEvent): logging.info('Python EventHub trigger processed an event: %s', event.get_body().decode('utf-8'))

Just a quick note that event, as you see in the code snippet is also set in the function.json file and identifies the object into which the retrieved event will placed into for reference. It took me some time to figure out, but that pattern is for a cardinality of one. Take a look at the code snippet which works with a cardinality of many.

def main(events: List[func.EventHubEvent]): for event in events: logging.info('Python EventHub trigger processed an event: %s', event.get_body().decode('utf-8'))

But wait, it got a bit more tricky because event is now events, which makes sense because when cardinality is many, at batch of events are retrieved instead of only one. What needs to happen in this case is to change the name attribute in the function.json to events. Or, if you want to run your function with a cardinality of one, then you need to change the cardinality from many to one.

So there is a mismatch in the default function.json when it comes to cardinality and name, as well, with the default main() method template. Simply, if you want to run cardinality of one, you need ‘name’ to be event, ‘cardinality ‘to be one and use the first code snippet which is expecting only a single event. If you want to retrieve a batch of events, you use the second code snippet then set the ‘name’ to events and ‘cardinality’ to many. For your information, you set the size of the batches using the maxBatchSize via the host.json. Other than that, it went pretty smooth. There were a few more things to look out for, which took me some moments, but they were mostly figured out through small, subtle references found scattered through the internet. Now let’s look a bit more about what I did and how I got this to work, starting with some useful links.

Useful Links

The first thing I did was get Python installed on my machine. Nothing special here, I had numerous versions already installed, but I wanted to make sure the one which showed up in Figure 1, was what I wanted and that it matched the version I was targeting my Azure Function to, see Figure 2.

Figure 1, Azure Event Hub triggered Azure Function Python

Figure 2, Azure Event Hub triggered Azure Function Python

There are some links on how to create an Azure Function using Python which I linked to above, so I won’t got step by step. Instead, I show Figure 3 which shows the Visual Studio Code IDE and in the Functions section I clicked on the Folder to create the Project and the lightening bolt to create the function.

Figure 3, Azure Event Hub triggered Azure Function Python, Visual Studio Code

I created a Project into folder PythonFunctions and the function was named EventHubPy which resulted in the following path shown in Figure 4. Have a look at the default folder structure described here.

Figure 4, Azure Event Hub triggered Azure Function Python, Visual Studio Code

Here is where some fun began. When I tried to run the function locally, using the command in Figure 5, I got some problems which I needed to work through.

Figure 5, Azure Event Hub triggered Azure Function Python

I did get the following issue first, but it went away after I had realized I needed to get my versions of Python I had on my workstation cleared up and my environment configured correctly.

Deploying “.” instead of selected folder , use azurefunctions.directorySubpath

I fixed the one you see in Figure 5, which looked like this, by changing the directory from which I was running the func start –python command. I moved from the function directory to the Project directory, see Figure 6.

Loading functions metadata Reading functions metadata 0 functions found 0 functions loaded Generating 0 job function(s)

Figure 6, Azure Event Hub triggered Azure Function Python

I also received the following output when I attempted to run the function.

Error parsing port. Local port to listen on. Default: 7071
Error parsing timeout. Timeout for on the functions host to start in seconds. Default: 20 seconds.

I restarted Visual Studio Code and I rebooted but that didn’t resolve it. What I did, but cannot be sure what happened, is that I ran these commands.

func host start –verbose CTRL+C func start –python

and it all started working again.
To send messages to the Event Hub I used a program I wrote called the Azure Function Consumer, look it over and download it from GitHub here. That program is written in C#, that means you can send messages to the Event Hub using any supported language and the function can be written in another or any supported language.

Figure 7, Azure Event Hub triggered Azure Function Python, Azure Function Consumer

Once I got everything working I deployed the Azure Function to Azure using func azure functionapp publish <APP-NAME>

The deployment was successful but I saw this error:

Error calling sync triggers (BadRequest). Request ID = ”.

This is discussed here. It is also triggered when you access the Function via the Azure portal which I did and all was well.

One point, a very important point, is the ‘connection’ attribute in the function.json file. The value for that attribute is the endpoint of your Event Hub, you added this to the local.settings.json file when you worked with the function locally. This value isn’t deployed for you, you need to add the value you set for ‘connection’ was a value of the endpoint as its value. The endpoint value is the same as you entered in the Azure Function Consumer which is referred to as the connection string. You can get this from the Azure Event Hub blade via the Shared Access Policies menu.

Then I added some more messages using the Azure Functions Consumer and it worked just fine. Since the Event Hub was the same, the details I used for the Azure Functions Consumer were the same too. The difference is that now my function is processing the events using compute power on Azure, in the Cloud. That’s cool!

Another important point

After you have deployed the function to Azure, make sure you so not run the local instance of the program because it too will be triggered / invoked when messages are added to the Event Hub. Any host which is configured with the access key to monitor that hub will compete with the other hosts which have the same configuration. You might notice that some messages are not getting processed. Make sure you protect that connection string and also keep your production and testing environment clearly separated.

General Experiences, exceptions and solutions

By default the the AzureWebJobsStorage was empty, when you create an Azure Function you need to provide a Storage Account endpoint. This is the value which is required to be the value of that attribute. There will be a Configuration aka. Application Setting named AzureWebJobsStorage with that value as well.

Missing value for AzureWebJobsStorage in local.settings.json. This is required for all triggers other than httptrigger, kafkatrigger. You can run func azure functionapp fetch-app-settings <functionAppName> or specify a connection string in local.settings.json.

This is an example of the verbose output of the func start –-python –-verbose command

Reading functions metadata
1 functions found
1 functions loaded

…

[2021-04-09T08:56:15.675Z] Loading functions metadata
[2021-04-09T08:56:15.679Z] FUNCTIONS_WORKER_RUNTIME set to python. Skipping WorkerConfig for language:java
[2021-04-09T08:56:15.681Z] FUNCTIONS_WORKER_RUNTIME set to python. Skipping WorkerConfig for language:node
[2021-04-09T08:56:15.684Z] FUNCTIONS_WORKER_RUNTIME set to python. Skipping WorkerConfig for language:powershell
[2021-04-09T08:56:15.704Z] Loading proxies metadata
[2021-04-09T08:56:15.789Z] Initializing Azure Function proxies
[2021-04-09T08:56:16.055Z] 0 proxies loaded
[2021-04-09T08:56:16.071Z] 1 functions loaded
[2021-04-09T08:56:16.106Z] Generating 1 job function(s)
[2021-04-09T08:56:16.178Z] Found the following functions:
[2021-04-09T08:56:16.181Z] Host.Functions.EventHubPy
[2021-04-09T08:56:16.183Z]
[2021-04-09T08:56:16.188Z] Initializing function HTTP routes
[2021-04-09T08:56:16.190Z] No HTTP routes mapped
[2021-04-09T08:56:16.192Z]
[2021-04-09T08:56:16.210Z] Host initialized (537ms)

…

[2021-04-09T08:56:25.252Z] Host lock lease acquired by instance ID ‘0000000000000000000000006F74C4E5’.

…

[2021-04-09T08:57:23.021Z] Stopping host…
[2021-04-09T08:57:23.050Z] Stopping JobHost
[2021-04-09T08:57:23.055Z] Stopping the listener ‘Microsoft.Azure.WebJobs.EventHubs.EventHubListener’ for function ‘EventHubPy’
[2021-04-09T08:57:23.376Z] Stopped the listener ‘Microsoft.Azure.WebJobs.EventHubs.EventHubListener’ for function ‘EventHubPy’
[2021-04-09T08:57:23.396Z] Job host stopped
[2021-04-09T08:57:23.466Z] Host shutdown completed.

When I ran func start -–python ––verbose I saw this exception. As the message states, it is noise and can be ignored.

An Event Hub exception of type ‘ReceiverDisconnectedException’ was thrown from Partition Id: ‘9’, Owner: ‘9185a7e6-8008-4588-afc0-e5e9bf138c23’, EventHubPath: ‘csharpguitar’. This exception type is typically a result of Event Hub processor rebalancing and can be safely ignored.

Cardinality

When the function was invoked after adding a message to the Event Hub, I received this exception.

System.Private.CoreLib: Exception while executing function: Functions.EventHubPy. System.Private.CoreLib: Result: Failure
Exception: AttributeError: ‘list’ object has no attribute ‘get_body’

I resolved this by making these changes to the function.json file:

Changing “cardinality”: “many” to “cardinality”: “one”
Changing “name”: “events” to “name”: “event”

I also received this exception when I was consuming the Event Hub, I received this exception.

System.Private.CoreLib: Exception while executing function: Functions.EventHubPy. System.Private.CoreLib: Result: Failure
Exception: FunctionLoadError: cannot load the EventHubPy function: the following parameters are declared in Python but not in function.json: {‘events’}
Stack:   File “C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.8\WINDOWS\X64\azure_functions_worker\dispatcher.py”, line 272, in _handle__function_load_request
     self._functions.add_function(
   File “C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.8\WINDOWS\X64\azure_functions_worker\functions.py”, line 112, in add_function
     raise FunctionLoadError(