11.1 C
New York
Monday, March 4, 2024

Externalize Amazon MSK Join configurations with Terraform

Managing configurations for Amazon MSK Join, a characteristic of Amazon Managed Streaming for Apache Kafka (Amazon MSK), can change into difficult, particularly because the variety of subjects and configurations grows. On this publish, we handle this complexity by utilizing Terraform to optimize the configuration of the Kafka matter to Amazon S3 Sink connector. By adopting this strategic strategy, you’ll be able to set up a strong and automatic mechanism for dealing with MSK Join configurations, eliminating the necessity for handbook intervention or connector restarts. This environment friendly answer will save time, cut back errors, and supply higher management over your Kafka information streaming processes. Let’s discover how Terraform can simplify and improve the administration of MSK Join configurations for seamless integration together with your infrastructure.

Answer overview

At a well known AWS buyer, the administration of their continually rising MSK Join S3 Sink connector subjects has change into a major problem. The challenges lie within the overhead of managing configurations, in addition to coping with patching and upgrades. Manually dealing with Kubernetes (K8s) configs and restarting connectors could be cumbersome and error-prone, making it troublesome to maintain observe of modifications and updates. On the time of scripting this publish, MSK Join doesn’t supply native mechanisms to simply externalize the Kafka matter to S3 Sink configuration.

To deal with these challenges, we introduce Terraform, an infrastructure as code (IaC) device. Terraform’s declarative strategy and in depth ecosystem make it a great alternative for managing MSK Join configurations.

By externalizing Kafka matter to S3 configurations, organizations can obtain the next:

  • Scalability – Effortlessly handle a rising variety of subjects, guaranteeing the system can deal with rising information volumes with out problem
  • Flexibility – Seamlessly combine MSK Join configurations with different infrastructure elements and companies, enabling adaptability to altering enterprise wants
  • Automation – Automate the deployment and administration of MSK Join configurations, decreasing handbook intervention and streamlining operational duties
  • Centralized administration – Obtain improved governance with centralized administration, model management, auditing, and alter monitoring, guaranteeing higher management and visibility over the configurations

Within the following sections, we offer an in depth information on establishing Terraform for MSK Join configuration administration, defining and decentralizing Matter configurations, and deploying and updating configurations utilizing Terraform.


Earlier than continuing with the answer, guarantee you’ve gotten the next sources and entry:

  • You want entry to an AWS account with enough permissions to create and handle sources, together with AWS Id and Entry Administration (IAM) roles and MSK clusters.
  • To simplify the setup, use the supplied AWS CloudFormation template. This template will create the mandatory MSK cluster and required sources for this publish.
  • For this publish, we’re utilizing the newest Terraform model (1.5.6).

By guaranteeing you’ve gotten these conditions in place, you may be able to comply with the directions and streamline your MSK Join configurations with Terraform. Let’s get began!


Establishing Terraform for MSK Join configuration administration consists of the next:

  • Set up of Terraform and establishing the setting
  • Establishing the mandatory authentication and permissions

Defining and decentralizing matter configurations utilizing Terraform consists of the next:

  • Understanding the construction of Terraform configuration recordsdata
  • Figuring out the required variables and sources
  • Using Terraform’s modules and interpolation for flexibility

The choice to externalize the configuration was primarily pushed by the client’s enterprise requirement. They anticipated the necessity to add subjects periodically and needed to keep away from the necessity to convey down and write particular code every time. Given the constraints of MSK Join (as of this writing), it’s vital to notice that MSK Join can deal with as much as 300 employees. For this proof of idea (POC), we opted for a configuration with 100 subjects directed to a single Amazon Easy Storage Service (Amazon S3) bucket. To make sure compatibility inside the 300-worker restrict, we set the MCU depend to 1 and configured auto scaling with a most of two employees. This ensures that the configuration stays inside the bounds of the 300-worker most.

To make the configuration extra versatile, we specify the variables that may be utilized within the code.(variables.tf):

variable "aws_region" {
description = "The AWS area to deploy sources in."
kind = string

variable "s3_bucket_name" {
description = "s3_bucket_name."
kind = string

variable "subjects" {
description = "subjects"
kind = string

variable "msk_connect_name" {
description = "Identify of the MSK Join occasion."
kind = string

variable "msk_connect_description" {
description = "Description of the MSK Join occasion."
kind = string

# Remainder of the variables...

To arrange the AWS MSK Connector for the S3 Sink, we have to present numerous configurations. Let’s look at the connector_configuration block within the code snippet supplied within the primary.tf file in additional element:

connector_configuration = {
"connector.class" = "io.confluent.join.s3.S3SinkConnector"
"s3.area" = "us-east-1"
"flush.measurement" = "5"
"schema.compatibility" = "NONE"
"duties.max" = "1"
"subjects" = var.subjects
"format.class" = "io.confluent.join.s3.format.json.JsonFormat"
"partitioner.class" = "io.confluent.join.storage.partitioner.DefaultPartitioner"
"worth.converter.schemas.allow" = "false"
"worth.converter" = "org.apache.kafka.join.json.JsonConverter"
"storage.class" = "io.confluent.join.s3.storage.S3Storage"
"key.converter" = "org.apache.kafka.join.storage.StringConverter"
"s3.bucket.title" = var.s3_bucket_name
"subjects.dir" = "cxdl-data/KairosTelemetry"

The kafka_cluster block within the code snippet defines the Kafka cluster particulars, together with the bootstrap servers and VPC settings. You possibly can reference the variables to specify the suitable values:

kafka_cluster {
apache_kafka_cluster {
bootstrap_servers = var.bootstrap_servers

vpc {
security_groups = [var.security_groups]
subnets = [var.aws_subnet_example1_id, var.aws_subnet_example2_id, var.aws_subnet_example3_id]

To safe the connection between Kafka and the connector, the code snippet consists of configurations for authentication and encryption:

  • The kafka_cluster_client_authentication block units the authentication kind to IAM, enabling using IAM for authentication
  • The kafka_cluster_encryption_in_transit block allows TLS encryption for information switch between Kafka and the connector
  kafka_cluster_client_authentication {
    authentication_type = "IAM"

  kafka_cluster_encryption_in_transit {
    encryption_type = "TLS"

You possibly can externalize the variables and supply dynamic values utilizing a var.tfvars file. Let’s assume the content material of the var.tfvars file is as follows:

aws_region = "us-east-1"
msk_connect_name = "confluentinc-MSK-connect-s3-2"
msk_connect_description = "My MSK Join occasion"
s3_bucket_name = "msk-lab-xxxxxxxxxxxx-target-bucket"
subjects = "salesdb.salesdb.CUSTOMER,salesdb.salesdb.CUSTOMER_SITE,salesdb.salesdb.PRODUCT,salesdb.salesdb.PRODUCT_CATEGORY,salesdb.salesdb.SALES_ORDER,salesdb.salesdb.SALES_ORDER_ALL,salesdb.salesdb.SALES_ORDER_DETAIL,salesdb.salesdb.SALES_ORDER_DETAIL_DS,salesdb.salesdb.SUPPLIER"
bootstrap_servers = "b-2.mskclustermskconnectl.4xwlfx.c11.kafka.us-east-1.amazonaws.com:9098,b-3.mskclustermskconnectl.4xwlfx.c11.kafka.us-east-1.amazonaws.com:9098,b-1.mskclustermskconnectl.4xwlfx.c11.kafka.us-east-1.amazonaws.com:9098“
aws_subnet_example1_id = "subnet-016ef7bb5f5db5759"
aws_subnet_example2_id = "subnet-0114c390d379134fa"
aws_subnet_example3_id = "subnet-0f6352ad89a1454f2"
security_groups = "sg-07eb8f8e4559334e7"
aws_mskconnect_custom_plugin_example_arn = "arn:aws:kafkaconnect:us-east-1:xxxxxxxxxxxx:custom-plugin/confluentinc-kafka-connect-s3-10-0-3/e9aeb52e-d172-4dba-9de5-f5cf73f1cb9e-2"
aws_mskconnect_custom_plugin_example_latest_revision = "1"
aws_iam_role_example_arn = "arn:aws:iam::xxxxxxxxxxxx:function/msk-connect-lab-S3ConnectorIAMRole-3LBTU7YAV9CM"

Deploy and replace configurations utilizing Terraform

When you’ve outlined your MSK Join infrastructure utilizing Terraform, making use of these configurations is an easy course of for creating or updating your infrastructure. This turns into notably handy when a brand new matter must be added. Because of the externalized configuration, incorporating this modification is now a seamless process. The steps are as follows:

  1. Obtain and set up Terraform from the official web site (https://www.terraform.io/downloads.html) on your working system.
  2. Verify the set up by working the terraform model command in your command line interface.
  3. Guarantee that you’ve configured your AWS credentials utilizing the AWS Command Line Interface (AWS CLI) or by setting setting variables. You should utilize the aws configure command to configure your credentials if you happen to’re utilizing the AWS CLI.
  4. Place the primary.tf, variables.tf, and var.tfvars recordsdata in the identical Terraform listing.
  5. Open a command line interface, navigate to the listing containing the Terraform recordsdata, and run the command terraform init to initialize Terraform and obtain the required suppliers.
  6. Run the command terraform plan -var-file="var.tfvars" to assessment the run plan.

This command reveals the modifications that Terraform will make to the infrastructure based mostly on the supplied variables. This step is elective however is usually used as a preview of the modifications Terraform will make.

  1. If the plan appears appropriate, run the command terraform apply -var-file="var.tfvars" to use the configuration.

Terraform will create the MSK_Connect in your AWS account. This may immediate you for affirmation earlier than continuing.

  1. After the terraform apply command is full, confirm the infrastructure has been created or up to date on the console.
  2. For any modifications or updates, modify your Terraform recordsdata (primary.tf, variables.tf, var.tfvars) as wanted, after which rerun the terraform plan and terraform apply instructions.
  3. Whenever you not want the infrastructure, you should utilize terraform destroy -var-file="var.tfvars" to take away all sources created by your Terraform recordsdata.

Watch out with this command as a result of it can delete all of the sources outlined in your Terraform recordsdata.


On this publish, we addressed the challenges confronted by a buyer in managing MSK Join configurations and described a Terraform-based answer. By externalizing Kafka matter to Amazon S3 configurations, you’ll be able to streamline your configuration administration processes, obtain scalability, improve flexibility, automate deployments, and centralize administration. We encourage you to make use of Terraform to optimize your MSK Join configurations and discover additional potentialities in managing your streaming information pipelines effectively.

To get began with externalizing MSK Join configurations utilizing Terraform, discuss with the supplied implementation steps and the Getting Began with Terraform information, MSK Join documentation, Terraform documentation, and instance GitHub repository.

Utilizing Terraform to externalize the Kafka matter to Amazon S3 Sink configuration in MSK Join gives a strong answer for managing and scaling your streaming information pipelines. By automating the deployment, updating, and central administration of configurations, you’ll be able to guarantee effectivity, flexibility, and scalability in your information processing workflows.

Concerning the Creator

RamC Venkatasamy is a Options Architect based mostly in Bloomington, Illinois. He helps AWS Strategic prospects rework their companies within the cloud. With a fervent enthusiasm for Serverless, Occasion-Pushed Structure and GenAI.

Related Articles


Please enter your comment!
Please enter your name here

Latest Articles