Partitioning

Topic Replica and Partition Count Manipulation

KSQL provides a powerful way for you to change the properties of Kafka topics by defining new streams with the desired properties of the new topic, populated by the streaming events of the original topic.

Because these are streaming transformations, the existing Kafka topic gets streamed to the new topic with the desired properties, and the target topic is kept in sync with every single message that subsequently arrives on the source topic.

Directions

Follow these steps to change a topic’s replication factor and number of partitions.

1. Inspect the source topic’s properties:

[source,bash]
----
$ kafka-topics --zookeeper localhost:2181 --describe --topic mysql_users_avro
Topic:mysql_users_avro  PartitionCount:4        ReplicationFactor:1     Configs:
        Topic: mysql_users_avro Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 3    Leader: 0       Replicas: 0     Isr: 0
----

2. Define the source topic stream:

[source,sql]
----
CREATE STREAM SOURCE_TOPIC WITH (KAFKA_TOPIC='mysql_users_avro', VALUE_FORMAT='AVRO');
----

3. Create a derived topic with new properties:

[source,sql]
----
CREATE STREAM TARGET_TOPIC WITH (KAFKA_TOPIC='mysql_users_avro_v2', 
                                 PARTITIONS=1,                      
                                 REPLICAS=3) AS                     
SELECT * FROM SOURCE_TOPIC;
----

4. Inspect the resulting topic:

[source,bash]
----
$ kafka-topics --zookeeper localhost:2181 --describe --topic mysql_users_avro_v2
Topic:mysql_users_avro_v2       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: mysql_users_avro_v2      Partition: 0    Leader: 1       Replicas: 1,2,0     Isr: 1,2,0
----
< Back to the Stream Processing Cookbook

Nous utilisons des cookies afin de comprendre comment vous utilisez notre site et améliorer votre expérience. Cliquez ici pour en apprendre davantage ou pour modifier vos paramètres de cookies. En poursuivant la navigation, vous consentez à ce que nous utilisions des cookies.