Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

How the New Confluent Hub Makes Finding Connectors Easier Than Ever

Confluent Hub launched in 2018 as a place to discover and share Apache Kafka® and Confluent Platform plugins. Users have found the site a much better place to discover useful code than seeking out individual git repos. These plugins include connectors, Single Message Transforms (SMTs), and converters for Kafka Connect. At launch, we had a humble inventory of about a dozen plugins. Now, there are over 140 available, including 100+ that are fully supported by Confluent and our partners!

Since its launch, we’ve looked for ways to improve Confluent Hub, and today we’re happy to introduce a complete frontend redesign, based on user feedback. In particular, we’ve tried to make it much easier to parse plugins across dimensions that matter most to developers.

Finding what you need

Confluent Hub

The most notable update to Confluent Hub is the addition of a detailed plugin taxonomy that lets you filter options across a number of key plugin dimensions.

Type de plugin

Determining the plugin type you need is likely your first concern as a developer. The plugin types on Confluent Hub are as follows:

  • Sink connector: sink connectors consume messages from Kafka topics and export them to external systems (e.g., Amazon S3 or MySQL).
  • Source connector: source connectors consume data from external systems (e.g., Amazon S3 or MySQL) and write them to Kafka topics.
  • Transform: SMTs are applied to messages as they flow through Connect. SMTs transform inbound messages after a source connector has produced them but before they are written to Kafka. Alternatively, SMTs transform outbound messages before they are sent to a sink connector.
  • Converter: converters change the format of data from bytes to a Connect internal data format, and vice versa. They are necessary in order for a Kafka Connect deployment to support a particular data format when writing to or reading from Kafka.

Assistance aux entreprises

Depending on where you’re doing your development, you may or may not need access to enterprise-level support. You might even be quite adventurous with your own projects. Lots of companies we talk to, however, aren’t comfortable putting something into production without the ability to get expert support on the phone. To help both types of users navigate the available plugins, Confluent Hub lets you filter by enterprise support type:

  • Confluent supported: Confluent enterprise support for these plugins is included as part of a Confluent Platform subscription. (You can contact us to learn more.)
  • Partner supported: enterprise support is available from the partner who maintains the plugin.
  • None: Many open source plugins fall in this category. No enterprise support is available, although varying levels of community support may be available. An open source Git repo and the Confluent Community Slack are often good places to look.

Vérification

Similar to enterprise support, you may or may not be developing in an environment that requires everything to be ready for “enterprise scale” from day zero. It’s impossible to point to a single benchmark that quantifies this, but these verification levels are our attempt to identify at least those plugins that are known to be built for high performance and stability. These categories include:

  • Confluent Built: built, tested, and supported by Confluent.
  • Confluent Tested: community plugins, which are tested and supported by Confluent.
  • Verified Gold: partner built and supported integrations that meet the highest level of functional compatibility with the Confluent Platform ecosystem. (See the Confluent Verified Integration FAQs for further details.)
  • Verified Standard: partner built and supported integrations that meet a standard level of functional compatibility with the Confluent Platform ecosystem. (See the Confluent Verified Integration FAQs for further details.)
  • None: community contributions that have not been verified or tested by Confluent. This doesn’t mean they’re not good—we just haven’t had a chance to rigorously test them.

Licence

This one is pretty straightforward:

  • Commercial: requires a Confluent Platform or partner subscription. Confluent supported plugins can be used for free on a single broker cluster (Developer License) or free for 30 days on an unlimited number of brokers (Evaluation License).
  • Free: available for free through an open source or community license.

Disponible dans Confluent Cloud

Our fully managed Apache Kafka as a service product, Confluent Cloud, offers some of these connectors, fully managed, on leading public clouds like Google Cloud Platform (GCP), Microsoft Azure, and Amazon Web Services (AWS).

Let us know what you think or submit a plugin

That’s a quick overview of what’s changed in the Confluent Hub frontend. We’re always looking for ways to improve the product and serve the community better, and one of the key ways we do this is by adding new open source and partner connectors to Confluent Hub. If you would like to get your code in front of more developers, or if you’d just like to give some feedback, send us an email at confluent-hub@confluent.io. We look forward to hearing from you!

Ethan Ruhe is a product manager at Confluent, where he leads the development of the Confluent CLI and formerly the Confluent Hub.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Apache Kafka DevOps with Kubernetes and GitOps

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring

Using the Fully Managed MongoDB Atlas Connector in a Secure Environment

Since the MongoDB Atlas source and sink became available in Confluent Cloud, we’ve received many questions around how to set up these connectors in a secure environment. By default, MongoDB

Streaming Data from Apache Kafka into Azure Data Explorer with Kafka Connect

Near-real-time insights have become a de facto requirement for Azure use cases involving scalable log analytics, time series analytics, and IoT/telemetry analytics. Azure Data Explorer (also called Kusto) is the