Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

Log Compaction | Highlights in the Kafka and Stream Processing Community | January 2016

Happy 2016! Wishing you a wonderful, highly scalable, and very reliable year. Log Compaction is a monthly digest of highlights in the Apache Kafka and stream processing community. Got a newsworthy item? Let us know.

Many things have happened since we last shared the state of Apache Kafka and the streams ecosystem. Lets take a look!

  • Kafka 0.9 was released and so was Plateforme Confluent 2.0
  • The call for proposals for Kafka Summit is closing soon – submit your abstract by Monday, January 11! Make it your new year’s resolution to participate more in the Kafka community and you can start by registering for the conference. Catch the Early Bird price (save $100) before it expires on January 15. 
  • Congratulations to Ewen Cheslack-Postava who joined Apache Kafka as a committer! We wish him much success and many patch reviews.
  • Kafka 0.9 added protocol support for managing groups so that clients no longer need to interact with ZooKeeper directly. KIP-40 gives a design for adding protocol support for clients to list available groups and to show the group members and their lag. The protocol is implemented in the new ConsumerGroupCommand (kafka-consumer-groups.sh), and is also available for use by 3rd party clients.
  • Kafka currently only supports a “logical” notion of time – the message offset, which indicates a relative order of messages. Many users want to be able to know the physical time that a message was produced. KIP-32, which is in active discussions and voting, will add a timestamp field to each Kafka message, indicating the time the client created the message. This would eventually allow adding special indexes and also support consuming messages based on their timestamp.
  • Kafka Connect is a new feature introduced in 0.9 that makes it really easy to directly integrate Kafka with external data systems like RDBMS’s or Hadoop. This tutorial shows how to use Kafka Connect to get events from a relational database to Kafka, and from there to HDFS and Hive, including automated partitioning of the data and updates to the Hive schema.
  • Looking to use Kafka with Spring for stream processing? There’s a 5-part blog post on how to do just that.
  • Yahoo published a benchmark comparing popular stream processing frameworks – Storm, SparkStreaming, and Flink.
  • At the first Seattle Kafka Meetup this past November, Microsoft shared how they use Kafka in Bing. One trillion messages per day!
  • Kafka, The Definitive Guide is now available for pre-order from O’Reilly.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Analysing Changes with Debezium and Kafka Streams

Change Data Capture (CDC) is an excellent way to introduce streaming analytics into your existing database, and using Debezium enables you to send your change data through Apache Kafka®. Although […]

Improved Robustness and Usability of Exactly-Once Semantics in Apache Kafka

This blog post talks about the recent improvements on exactly-once semantics (EOS) to make it simpler to use and more resilient. EOS was first released in Apache Kafka® 0.11 and […]

Data Privacy, Security, and Compliance for Apache Kafka

Why data privacy for Apache Kafka®? As companies seek to leverage all forms of data for competitive advantage, there is a growing regulatory and reputational risk that calls for the […]

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

Nouvelles inscriptions uniquement.

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Get Confluent Cloud

Get up to $200 off on each of your first 3 Confluent Cloud monthly bills


Choose one sign-up option below

Marketplaces

  • AWS
  • Azure
  • Google Cloud

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
*Billing admin role needed

Marketplaces

  • Billed through your Cloud provider*
  • Stream only on 1 cloud
  • Billing admin role needed

*Billing admin role needed

Confluent


  • Pay with a credit card
  • Stream across multiple clouds

Confluent

  • Pay with a credit card
  • Stream across multiple clouds

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Gratuit à vie sur un seul broker Kafka
i

Le logiciel permettra une utilisation illimitée dans le temps de fonctionnalités commerciales sur un seul broker Kafka. Après l'ajout d'un second broker, un compteur de 30 jours démarrera automatiquement sur les fonctionnalités commerciales. Celui-ci ne pourra pas être réinitialisé en revenant à un seul broker.

Sélectionnez un type de déploiement
Déploiement manuel
  • tar
  • zip
  • deb
  • rpm
  • docker
ou
Déploiement automatique
  • kubernetes
  • ansible

En cliquant sur le bouton « télécharger gratuitement » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Téléchargement gratuit » ci-dessus, vous acceptez la Contrat de licence Confluent et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous acceptez également que vos renseignements personnels soient traitées conformément à notre Politique de confidentialité.

Ce site Web utilise des cookies afin d'améliorer l'expérience utilisateur et analyser les performances et le trafic sur notre site Web. Nous partageons également des informations concernant votre utilisation de notre site avec nos partenaires publicitaires, analytiques et de réseaux sociaux.