Send Custom Events to Event Hubs using Azure Data Factory

Overview

• Overview
• Post Custom Event to Event Hubs via REST API
• Post Custom Event to Event Hubs via ADF
• Wrapping Up

Azure Data Factory is a fully managed, serverless data integration service. This service can visually integrate data sources with more than 90 built-in, maintenance-free connectors at no added cost. It can also easily construct ETL and ELT processes code-free in an intuitive environment or write your own code.

Data Factory connects to different Azure Services using these built-in connectors. Unfortunately, there is no built-in connector for integrating ADF with Event Hubs. This means that ADF can’t post events to Event Hub using its supported list of activities.

Workaround? I will use Event Hubs REST APIs combined with Web Activity of ADF to send our custom events.

Let’s first understand how these REST APIs for Event Hubs work.

Post Custom Event to Event Hubs via REST API

You can easily post an event to a specific event hub using the POST API provided by the Event Hubs REST service. You can refer the official documentation for more details.

The below shows the POST request with the corresponding headers.

POST https://<Event Hubs Namespace>.servicebus.windows.net/<Event Hub Name>/messages

HEADERS:
Authorization: < SAS TOKEN >

BODY:
{ 
    "Id": 1,
    "Name": "test-name-1", 
    "Value": "test-value-1" 
}

You can refer the following link to get the SAS token: Generate SAS token using PowerShell

If your POST request is successful with 201 Created then you should be able to query the events in your event hub and see your posted custom event as shown below.

Now, let’s try to post the same custom event via Azure Data Factory.

 

Post Custom Event to Event Hubs via ADF

In order to post the same custom event via ADF, I will leverage the usage of authenticating via Managed Identity (MSI based authentication).

Firstly, I will assign Azure Event Hubs Data Sender role to the managed identity of the Data Factory. This will authorise Data Factory to send an event to the Event Hubs as shown below.

Once that is done, then you can use Web Activity of the ADF to post the same custom event as shown below.

The settings are as follows:

• URL: https://<Event Hubs Namespace>.servicebus.windows.net/<Event Hub Name>/messages
• Method: POST

• Body: your custom event body. In this example, I am posting the below object.

    { 
        "Id": 2,
        "Name": "test-name-2", 
        "Value": "test-value-2" 
    }
  

• Authentication: System Assigned Managed Identity
• Resource: https://<Event Hubs Namespace>.servicebus.windows.net

With this configuration, your pipeline should ideally succeed and you should be able to query your custom event in your provided Event Hub as shown below.

Wrapping Up

In this post, I showcased how to post a custom event to Event Hubs via Azure Data Factory. Since, there is no in-built connector yet for Event Hubs in ADF, I showed an approach to do the same using REST APIs and MSI based authentication via Web Activity of ADF.

Hopefully, there is an active development to provide an in-built connector which can potentially avoid this workaround.