Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

Démarrer avec Apache Kafka and Kubernetes

Enabling everyone to run Apache Kafka® on Kubernetes is an important part of our mission to put a streaming platform at the heart of every company. This is why we look forward to releasing an implementation of the Kubernetes Operator API for automated provisioning, management, and operations of Kafka on Kubernetes. (Read about that here and let us know if you’d like to take it for a spin when it’s available!)

Today, we are taking a step in this direction by sharing two resources to help you run Confluent Platform on Kubernetes:

  1. Helm Charts that provide a starting point for deployment of Confluent Platform components
  2. A white paper that shares the best practices we’ve learned by running Confluent Platform on Kubernetes

Getting started with deployment templates

Now available on GitHub in developer preview are open source Helm Chart deployment templates for Confluent Platform components. These templates enable developers to quickly provision Apache Kafka, Apache ZooKeeper, Confluent Schema Registry, Confluent REST Proxy, and Kafka Connect on Kubernetes, using official Confluent Platform Images Docker.

Helm is an open source packaging tool that helps you install applications and services on Kubernetes. Helm uses a packaging format called charts. A chart is a collection of YAML templates that describe a related set of Kubernetes resources.

For stateful components like Kafka and ZooKeeper, the Helm Charts use both StatefulSets to provide an identity to each pod in the form of an ordinal index, and Persistent Volumes that are always mounted for the pod. For stateless components, like REST Proxy, the Helm Charts utilize Deployments instead to provide an identity to each pod. Each component’s charts utilize Services to provide access to each pod.

It’s really easy to get started. Just install Helm on your local or deployed Kubernetes cluster, and use the parent chart to configure and deploy Confluent Platform components:

helm install cp-helm-charts

helm.install.6

Helm allows you to customize configurations using YAML templates. For example, to deploy five Kafka brokers and set a custom `min.insync.replicas`, create a copy of the default YAML template, update the values, and use it to install the configuration:

helm install -f custom-values.yaml cp-helm-charts

custom-values.yaml



## Number of Kafka brokers 

brokers: 5



## Kafka Server properties

configurationOverrides:

  "log.dirs": /opt/kafka/data/logs

  "offsets.topic.replication.factor": 3

  "min.insync.replicas": 3

Special thanks to Qi Shao, Nikhil Chandrappa, Li Wang, Amey Banarse and Prasad Radhakrishnan from Pivotal’s (NYSE: PVTL) Platform Architecture team for their contributions in developing and testing these Helm Charts.

Sharing best practices

What’s both exciting and challenging about working with the vibrant Kubernetes ecosystem is that everything progresses so fast. To help you understand and plan for running Confluent Platform on Kubernetes, we’ve provided a white paper that shares the best practices we’ve learned over time.

The Recommendations for Deploying Apache Kafka on Kubernetes white paper helps you see how Confluent Platform components fit into the Kubernetes ecosystem. It also covers our approach to networking, storage, traffic, log aggregation, metrics and more.

We will update this white paper as the Kubernetes ecosystem continues to evolve.

Conclusion

Our end goal for this is to make streaming data ubiquitous. Kubernetes lets you run your applications and services anywhere. Kafka enables you to make your data accessible instantaneously, anywhere.

Get started with Kafka on Kubernetes today by checking out the white paper and Helm Charts on our website.

With Confluent Operator, we are productizing years of Kafka experience with Kubernetes expertise to offer you the best way of using Apache Kafka on Kubernetes. Let us know if you are interested and we’ll notify you when it’s available!

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Measuring TCO: Apache Kafka vs. Confluent Cloud’s Managed Service

If your company has launched a digital transformation project or is getting ready to start one, you may be interested to know that recent studies conducted by McKinsey, Everest Group, […]

Elastically Scaling Confluent Platform on Kubernetes

This month, we kicked off Project Metamorphosis by introducing several Confluent features that make Apache Kafka® clusters more elastic—the first of eight foundational traits characterizing cloud-native data systems that map […]

Building Confluent Cloud – Here’s What We’ve Learned

In July 2017, Confluent launched a private preview of what would later be known as Confluent Cloud. This platform as a service product has grown rapidly; less than three years […]

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

Nouvelles inscriptions uniquement.

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Get Confluent Cloud

Get up to $200 off on each of your first 3 Confluent Cloud monthly bills


Choose one sign-up option below

Marketplaces

  • AWS
  • Azure
  • Google Cloud

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
*Billing admin role needed

Marketplaces

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
  • Billing admin role needed

*Billing admin role needed

Confluent


  • Pay with a credit card
  • Stream across multiple clouds

Confluent

  • Pay with a credit card
  • Stream across multiple clouds

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Gratuit à vie sur un seul broker Kafka
i

Le logiciel permettra une utilisation illimitée dans le temps de fonctionnalités commerciales sur un seul broker Kafka. Après l'ajout d'un second broker, un compteur de 30 jours démarrera automatiquement sur les fonctionnalités commerciales. Celui-ci ne pourra pas être réinitialisé en revenant à un seul broker.

Sélectionnez un type de déploiement
Déploiement manuel
  • tar
  • zip
  • deb
  • rpm
  • docker
ou
Déploiement automatique
  • kubernetes
  • ansible

En cliquant sur le bouton « télécharger gratuitement » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Téléchargement gratuit » ci-dessus, vous acceptez la Contrat de licence Confluent et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous acceptez également que vos renseignements personnels soient traitées conformément à notre Politique de confidentialité.

Ce site Web utilise des cookies afin d'améliorer l'expérience utilisateur et analyser les performances et le trafic sur notre site Web. Nous partageons également des informations concernant votre utilisation de notre site avec nos partenaires publicitaires, analytiques et de réseaux sociaux.