Part 3: Using Azure Event Hub for Stream Processing

In the previous part, we saw how to use Strimzi to create a Kafka connect cluster with the Debezium SQL Server plugin included. We then enabled CDC on SQL Server and got the snapshot and change tracking to work with the Kafka cluster.

In this post, we will see how to use Azure Event Hub instead of the Kafka cluster that is running inside the Kubernetes cluster. Azure Event Hub is almost fully compatible with Kafka.

Please pull down the github repo here to get access to the files used in this blog post.

Create the Event Hub instance on Azure

To create the event hub instance, type in “event hub” in the search bar and select Event Hubs from the search results. Created a Event Hub namespace with the standard pricing tier, called change-tracker.

Navigate to Settings → Shared Access Policies in the event hub namespace and click on the RootManageSharedAccessKey to access the connection string.

Copy out the primary connection string and create the secret in the Kafka namespace for use with connect,

Connection string will look like this,

Endpoint=sb://change-tracker.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<Your-Shared-Access-Key>

kubectl create secret generic…