Connecting to Confluent Kafka from Mulesoft 4

Recently, I was working on a project connecting Mulesoft to a Confluent Kafka cluster while migrating from Heroku Kafka Connect.

Heroku Kafka Connect in a Private Space requires mTLS and IP allowlisting to connect to the producer. This method utilizes Consumer Plaintext within the Kafka connector in Mulesoft.

Contrast this to Confluent, which connects through a keyed connection over public HTTPS. Since I don’t particularly appreciate recreating the wheel, I looked for configuration settings in the usual places. I found a helpful article on connecting Confluent with Mulesoft, but it utilizes a deprecated Kafka connector (3.5).

Since the above article does a great job with the description except for the connector, this article will focus on the connector only.

We’ll utilize the Apache Kafka Connector to set up our configuration files. In the screenshots below, I’m editing inline for ease of display. However, in your environment, you should leverage environment variables.

Connection

Digging through the Confluent documentation, we notice that connections are over SASL_SSL. If we look in Mulesoft, there’s no option for this particular setting.

According to Mulesoft Documentation, SASL_SSL is the protocol used when at least a Keystore is set for a TLS context.

This leads us to select Consumer SASL/Plain Connection for our Configuration and ensure we have a valid TLS Context.

General

1. Set the Connection to Consumer SASL/PLAIN Connection as described above
2. This is the server set in your environment in Confluent. It is also the server provided in the TXT file provided when you set your API Key.
3. Set the topic names that you are looking to subscribe to. These topic names can be found in your Confluent producer.
4. The username and password fields are for the API Key and API Secret set within the Confluent environment

Security

As described above, Mulesoft will utilize a SASL_SSL connection when at least a keystore setting is set. Creating a keystore certificate is outside the scope of this article.

Important to note that in a production environment, you should probably set a global TLS Context for easier maintenance.

And that’s it, setting Mulesoft to connect to Confluent is rather easy once you determine what is working behind the scenes.

The new connectors are easier to work with, although I wish the documentation would be clearer. Since Confluent has a few connectors with Mulesoft for schema registry, I was shocked that there wasn’t much clear documentation on how to use Confluent as a producer.