Hi,
I am trying to do capacity sizing estimate for our kafka cluster. I started with 5 broker cluster and 3 node zk. Used a simple java based producer to send messages to 5 topics that are created in the cluster. I used 2 client machines with 100 worker threads each sending messages continuously. I didn't see any exceptions or issues when all 5 brokers are up. When I take down 3 brokers out of 5, and 4 topics out of 5. I am seeing below in broker logs.
[2014-08-08 16:47:34,977] DEBUG Closing connection from /10.254.243.142:33944 (kafka.network.Processor)
[2014-08-08 16:47:35,179] DEBUG Accepted connection from /10.254.243.142 on /10.66.107.231:9092. sendBufferSize [actual|requested]: [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] (kafka.network.Acceptor)
[2014-08-08 16:47:35,179] DEBUG Processor 860 listening to new connection from /10.254.243.142:33947 (kafka.network.Processor)
[2014-08-08 16:47:35,179] DEBUG [KafkaApi-1] Error while fetching metadata for [item_topic_0,0]. Possible cause: null (kafka.server.KafkaApis)
[2014-08-08 16:47:35,207] INFO Closing socket connection to /10.254.243.142. (kafka.network.Processor)
[2014-08-08 16:47:35,207] DEBUG Closing connection from /10.254.243.142:33947 (kafka.network.Processor)
[2014-08-08 16:47:35,294] DEBUG Accepted connection from /10.254.243.142 on /10.66.107.231:9092. sendBufferSize [actual|requested]: [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] (kafka.network.Acceptor)
On the client machine(producer) I am seeing below error. Again only one topic and 2 broker nodes are running.
Exception in thread "pool-1-thread-107" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
Sending message 181171284512 for topic item_topic_0
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "pool-1-thread-108" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
Sending message 221252534837 for topic item_topic_0
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Not all messages are erroring out. Only a few messages are failing. Any idea what could be going on?
Thanks,
Raj Tanneru
I am trying to do capacity sizing estimate for our kafka cluster. I started with 5 broker cluster and 3 node zk. Used a simple java based producer to send messages to 5 topics that are created in the cluster. I used 2 client machines with 100 worker threads each sending messages continuously. I didn't see any exceptions or issues when all 5 brokers are up. When I take down 3 brokers out of 5, and 4 topics out of 5. I am seeing below in broker logs.
[2014-08-08 16:47:34,977] DEBUG Closing connection from /10.254.243.142:33944 (kafka.network.Processor)
[2014-08-08 16:47:35,179] DEBUG Accepted connection from /10.254.243.142 on /10.66.107.231:9092. sendBufferSize [actual|requested]: [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] (kafka.network.Acceptor)
[2014-08-08 16:47:35,179] DEBUG Processor 860 listening to new connection from /10.254.243.142:33947 (kafka.network.Processor)
[2014-08-08 16:47:35,179] DEBUG [KafkaApi-1] Error while fetching metadata for [item_topic_0,0]. Possible cause: null (kafka.server.KafkaApis)
[2014-08-08 16:47:35,207] INFO Closing socket connection to /10.254.243.142. (kafka.network.Processor)
[2014-08-08 16:47:35,207] DEBUG Closing connection from /10.254.243.142:33947 (kafka.network.Processor)
[2014-08-08 16:47:35,294] DEBUG Accepted connection from /10.254.243.142 on /10.66.107.231:9092. sendBufferSize [actual|requested]: [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] (kafka.network.Acceptor)
On the client machine(producer) I am seeing below error. Again only one topic and 2 broker nodes are running.
Exception in thread "pool-1-thread-107" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
Sending message 181171284512 for topic item_topic_0
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "pool-1-thread-108" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
Sending message 221252534837 for topic item_topic_0
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33)
at com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Not all messages are erroring out. Only a few messages are failing. Any idea what could be going on?
Thanks,
Raj Tanneru
