kafka availability

Hi,

I perform availability tests on next kafka setup:
- 3 nodes(2f+1) for zookeeper(zookeeper1, zookeeper2, zookeeper3)
- 2 nodes(f+1) for kafka brokers(id:0,host:kafka.broker1,port:9092;
id:1,host:kafka.broker2,port:9092)
I use java producer and console consumer.

Test stepts:
1. create topic test.app4

bin/kafka-topics.sh --zookeeper zookeeper1 --create --replication-factor 2
--partitions 1 --topic test.app4

bin/kafka-topics.sh --zookeeper zookeeper1 --topic test.app4 --describe
Topic:test.app4 PartitionCount:1 ReplicationFactor:2 Configs:
Topic: test.app4 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1

2. shutdown kafka.broker2(the leader for topic), kafka.broker1 is alive
--> consumer fails in infinite loop:

WARN Fetching topic metadata with correlation id 13 for topics
[Set(test.app4)] from broker [id:0,host:kafka.broker1,port:9092] failed
(kafka.client.ClientUtils$)
java.net.ConnectException: Connection refused
...
WARN
[console-consumer-95116_ad-laptop-1396433768258-bb700f4c-leader-finder-thread],
Failed to find leader for Set([test.app4,0])
(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics
[Set(test.app4)] from broker
[ArrayBuffer(id:0,host:kafka.broker1,port:9092)] failed
...
Caused by: java.net.ConnectException: Connection refused
...
ERROR Producer connection to kafka.broker1:9092 unsuccessful
(kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
...

bin/kafka-topics.sh --zookeeper zookeeper1 --topic test.app4
--describeTopic:test.app4 PartitionCount:1 ReplicationFactor:2
Configs:
Topic: test.app4 Partition: 0 Leader: 0 Replicas: 1,0 Isr: 0

The java producer error:
INFO [kafka.producer.async.DefaultEventHandler] Back off for 100 ms before
retrying send. Remaining retries = 2
INFO [kafka.client.ClientUtils$] Fetching metadata from broker
id:0,host:kafka.broker1,port:9092 with correlation id 236 for 1 topic(s)
Set(test.app4)
ERROR [kafka.producer.SyncProducer] Producer connection to
kafka.broker1:9092 unsuccessful
java.net.ConnectException: Connection refused
...
WARN [kafka.client.ClientUtils$] Fetching topic metadata with correlation
id 236 for topics [Set(test.app4)] from broker
[id:0,host:kafka.broker1,port:9092] failed
java.net.ConnectException: Connection refused
...
INFO [kafka.client.ClientUtils$] Fetching metadata from broker
id:1,host:kafka.broker2,port:9092 with correlation id 236 for 1 topic(s)
Set(test.app4)
ERROR [kafka.producer.SyncProducer] Producer connection to
kafka.broker2:9092 unsuccessful
java.net.ConnectException: Connection refused
...
WARN [kafka.client.ClientUtils$] Fetching topic metadata with correlation
id 236 for topics [Set(test.app4)] from broker
[id:1,host:kafka.broker2,port:9092] failed
java.net.ConnectException: Connection refused
...
ERROR [kafka.utils.Utils$] fetching topic metadata for topics
[Set(test.app4)] from broker
[ArrayBuffer(id:0,host:kafka.broker1,port:9092,
id:1,host:kafka.broker2,port:9092)] failed
kafka.common.KafkaException: fetching topic metadata for topics
[Set(test.app4)] from broker
[ArrayBuffer(id:0,host:kafka.broker1,port:9092,
id:1,host:kafka.broker2,port:9092)] failed
...
Caused by: java.net.ConnectException: Connection refused
...
2014-04-02 13:31:12,419 DEBUG [kafka.producer.BrokerPartitionInfo] Getting
broker partition info for topic test.app4
2014-04-02 13:31:12,419 DEBUG [kafka.producer.BrokerPartitionInfo]
Partition [test.app4,0] has leader 1
2014-04-02 13:31:12,419 DEBUG [kafka.producer.async.DefaultEventHandler]
Broker partitions registered for topic: test.app4 are 0
2014-04-02 13:31:12,419 DEBUG [kafka.producer.async.DefaultEventHandler]
Sending 1 messages with no compression to [test.app4,0]
2014-04-02 13:31:12,420 DEBUG [kafka.producer.async.DefaultEventHandler]
Producer sending messages with correlation id 238 for topics [test.app4,0]
to broker 1 on kafka.broker2:9092
2014-04-02 13:31:12,423 ERROR [kafka.producer.SyncProducer] Producer
connection to kafka.broker2:9092 unsuccessful
java.net.ConnectException: Connection refused
...
WARN [kafka.producer.async.DefaultEventHandler] Failed to send producer
request with correlation id 238 to broker 1 with data for partitions
[test.app4,0]
java.net.ConnectException: Connection refused
...
INFO [kafka.producer.async.DefaultEventHandler] Back off for 100 ms before
retrying send. Remaining retries = 1

3. startup the kafka.broker2
--> it fails in infinite loop too:

INFO Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
WARN [ReplicaFetcherThread-0-0], Error in fetch Name: FetchRequest;
Version: 0; CorrelationId: 183; ClientId: ReplicaFetcherThread-0-0;
ReplicaId: 1; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:
[test.app4,0] -> PartitionFetchInfo(2,1048576)
(kafka.server.ReplicaFetcherThread)
java.net.ConnectException: Connection refused
...
[2014-04-02 12:56:33,816] INFO Reconnect due to socket error: null
(kafka.consumer.SimpleConsumer)

5. shutdown kafka.broker1
--> the initialization of kafka.broker2 is complete. System works fine.

6. startup kafka.broker1
--> System works fine

Please advise how to achieve high availability and what is wrong in this
case.

Regards,
Anatoly