I have used the kafka-topics.sh script to create a topic with a number of
partitions.
bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
--partitions 1024 --replication-factor 3
I was expecting this topic to be created with 1024 total partitions across
3 of my kafka brokers. Instead it seems like it ended up creating 1024 * 3
= 3072 partitions. Is this a bug?
I got the number of partitions running the code from the SimpleConsumer
example at
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
specifically the findLeader(...) function. I just count the number of
partitionsMetadata objects received. That seems to give me the number
3072. The relevant snippet looks like:
List<TopicMetadata> metaData = resp.topicsMetadata();
for (TopicMetadata item : metaData) {
for (PartitionMetadata part : item.partitionsMetadata()) {
numPartitions++;
Given I am running the above snippet for every broker, should I not expect
to see the numPartitions as 1024? I am assuming that
item.partitionsMetadata() only returns PartitionMetadata for the partitions
this broker is responsible for? If not then how do I exactly figure out
what partitions this broker is the leader for?
I am using the SimpleConsumer and my logic is getting completely muddled
because of this problem.
Also is there a way through the kafka console scripts to find out the
number of partitions for a given topic?
Thanks,
Rajiv
partitions.
bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
--partitions 1024 --replication-factor 3
I was expecting this topic to be created with 1024 total partitions across
3 of my kafka brokers. Instead it seems like it ended up creating 1024 * 3
= 3072 partitions. Is this a bug?
I got the number of partitions running the code from the SimpleConsumer
example at
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
specifically the findLeader(...) function. I just count the number of
partitionsMetadata objects received. That seems to give me the number
3072. The relevant snippet looks like:
List<TopicMetadata> metaData = resp.topicsMetadata();
for (TopicMetadata item : metaData) {
for (PartitionMetadata part : item.partitionsMetadata()) {
numPartitions++;
Given I am running the above snippet for every broker, should I not expect
to see the numPartitions as 1024? I am assuming that
item.partitionsMetadata() only returns PartitionMetadata for the partitions
this broker is responsible for? If not then how do I exactly figure out
what partitions this broker is the leader for?
I am using the SimpleConsumer and my logic is getting completely muddled
because of this problem.
Also is there a way through the kafka console scripts to find out the
number of partitions for a given topic?
Thanks,
Rajiv