I am writing a unit test to simulate a scenario where server changes IP,
which can happen in cloud env. During push, new instance can get a
different private IP.
As you can see from pasted code, I am running the test using zkclient.
ZooKeeper#updateServerList works well by forcing resolution to new IP
address that 2nd ZooKeeperServer object binds to. but the weird thing is
that after connection established, it seems that server closes connection
immediately. please see the log output in the end. I added some debug
System.out.
I am using zookeeper-3.5.0-rc1. Any help is appreciated!
Thanks,
Steven
###############################################
/**
* To test this unit testing, the followings should be done first
* ====== pre-test setup ======
* # bind 192.168.1.1 and 192.168.1.2 to lo0 interface as alias
* sudo ifconfig lo0 192.168.1.1 alias
* sudo ifconfig lo0 192.168.1.2 alias
* ====== manual test ======
* - add mapping to /etc/hosts
* 192.168.1.1 zkserver
* - run test. it should stuck in the line of second
client.createEphemeral(...)
* - change mapping in /etc/hosts
* 192.168.1.2 zkserver
* - test should proceed and finish
*/
@Ignore
public class TestServerIpChange {
@Test
public void test() throws InterruptedException, IOException {
System.setProperty("log4j.logger.org.apache.zookeeper", "INFO");
ZkServer _zkServer = TestUtil.startZkServer("Zk_SERVER_IP_Change",
"192.168.1.1", 2181);
ZkClient client = new ZkClient("zkserver:2181", 10000, 5000);
client.createEphemeral("/a");
for (int i = 0; i < 5; ++i) {
System.out.println("send and receive data: " + i);
client.readData("/a");
client.writeData("/a", Integer.toString(i));
Thread.sleep(1000);
System.out.println("sleeping..." + i);
_zkServer.shutdown();
_zkServer = TestUtil.startZkServer("Zk_SERVER_IP_Change",
"192.168.1.2", 2181);
// change /etc/hosts
client.createEphemeral("/a");
for (int i = 0; i < 5; ++i) {
System.out.println("send and receive data: " + i);
client.readData("/a");
client.writeData("/a", Integer.toString(i));
Thread.sleep(1000);
System.out.println("sleeping..." + i);
_zkServer.shutdown();
###########################################
14/08/22 21:17:46 INFO [main] (ZkServer.java:85) - Starting ZkServer on:
hostname = 192.168.1.1, port = 2181
14/08/22 21:17:46 INFO [main] (ZkServer.java:99) - Start single zookeeper
server...
14/08/22 21:17:46 INFO [main] (ZkServer.java:100) - data dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/data
14/08/22 21:17:46 INFO [main] (ZkServer.java:101) - data log dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/log
14/08/22 21:17:47 INFO [ZkClient-EventThread-17-zkserver:2181]
(ZkEventThread.java:64) - Starting ZkClient event thread.
connected to null
14/08/22 21:17:47 INFO [main-EventThread] (ZkClient.java:450) - zookeeper
state changed (SyncConnected)
send and receive data: 0
sleeping...0
send and receive data: 1
sleeping...1
send and receive data: 2
sleeping...2
send and receive data: 3
sleeping...3
send and receive data: 4
sleeping...4
14/08/22 21:17:52 INFO [main] (ZkServer.java:124) - Shutting down
ZkServer...
14/08/22 21:17:52 INFO [main] (ZkServer.java:143) - Shutting down
ZkServer...done
14/08/22 21:17:52 INFO [main] (ZkServer.java:85) - Starting ZkServer on:
hostname = 192.168.1.2, port = 2181
14/08/22 21:17:52 INFO [main] (ZkServer.java:99) - Start single zookeeper
server...
14/08/22 21:17:52 INFO [main] (ZkServer.java:100) - data dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/data
14/08/22 21:17:52 INFO [main] (ZkServer.java:101) - data log dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/log
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
14/08/22 21:17:52 INFO [main-EventThread] (ZkClient.java:450) - zookeeper
state changed (Disconnected)
...
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
connected to zkserver/192.168.1.2:2181
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
connected to zkserver/192.168.1.2:2181
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
which can happen in cloud env. During push, new instance can get a
different private IP.
As you can see from pasted code, I am running the test using zkclient.
ZooKeeper#updateServerList works well by forcing resolution to new IP
address that 2nd ZooKeeperServer object binds to. but the weird thing is
that after connection established, it seems that server closes connection
immediately. please see the log output in the end. I added some debug
System.out.
I am using zookeeper-3.5.0-rc1. Any help is appreciated!
Thanks,
Steven
###############################################
/**
* To test this unit testing, the followings should be done first
* ====== pre-test setup ======
* # bind 192.168.1.1 and 192.168.1.2 to lo0 interface as alias
* sudo ifconfig lo0 192.168.1.1 alias
* sudo ifconfig lo0 192.168.1.2 alias
* ====== manual test ======
* - add mapping to /etc/hosts
* 192.168.1.1 zkserver
* - run test. it should stuck in the line of second
client.createEphemeral(...)
* - change mapping in /etc/hosts
* 192.168.1.2 zkserver
* - test should proceed and finish
*/
@Ignore
public class TestServerIpChange {
@Test
public void test() throws InterruptedException, IOException {
System.setProperty("log4j.logger.org.apache.zookeeper", "INFO");
ZkServer _zkServer = TestUtil.startZkServer("Zk_SERVER_IP_Change",
"192.168.1.1", 2181);
ZkClient client = new ZkClient("zkserver:2181", 10000, 5000);
client.createEphemeral("/a");
for (int i = 0; i < 5; ++i) {
System.out.println("send and receive data: " + i);
client.readData("/a");
client.writeData("/a", Integer.toString(i));
Thread.sleep(1000);
System.out.println("sleeping..." + i);
_zkServer.shutdown();
_zkServer = TestUtil.startZkServer("Zk_SERVER_IP_Change",
"192.168.1.2", 2181);
// change /etc/hosts
client.createEphemeral("/a");
for (int i = 0; i < 5; ++i) {
System.out.println("send and receive data: " + i);
client.readData("/a");
client.writeData("/a", Integer.toString(i));
Thread.sleep(1000);
System.out.println("sleeping..." + i);
_zkServer.shutdown();
###########################################
14/08/22 21:17:46 INFO [main] (ZkServer.java:85) - Starting ZkServer on:
hostname = 192.168.1.1, port = 2181
14/08/22 21:17:46 INFO [main] (ZkServer.java:99) - Start single zookeeper
server...
14/08/22 21:17:46 INFO [main] (ZkServer.java:100) - data dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/data
14/08/22 21:17:46 INFO [main] (ZkServer.java:101) - data log dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/log
14/08/22 21:17:47 INFO [ZkClient-EventThread-17-zkserver:2181]
(ZkEventThread.java:64) - Starting ZkClient event thread.
connected to null
14/08/22 21:17:47 INFO [main-EventThread] (ZkClient.java:450) - zookeeper
state changed (SyncConnected)
send and receive data: 0
sleeping...0
send and receive data: 1
sleeping...1
send and receive data: 2
sleeping...2
send and receive data: 3
sleeping...3
send and receive data: 4
sleeping...4
14/08/22 21:17:52 INFO [main] (ZkServer.java:124) - Shutting down
ZkServer...
14/08/22 21:17:52 INFO [main] (ZkServer.java:143) - Shutting down
ZkServer...done
14/08/22 21:17:52 INFO [main] (ZkServer.java:85) - Starting ZkServer on:
hostname = 192.168.1.2, port = 2181
14/08/22 21:17:52 INFO [main] (ZkServer.java:99) - Start single zookeeper
server...
14/08/22 21:17:52 INFO [main] (ZkServer.java:100) - data dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/data
14/08/22 21:17:52 INFO [main] (ZkServer.java:101) - data log dir:
/Users/stevenwu/workspaces/github/stevenzwu/zkclient/./build/test/Zk_SERVER_IP_Change/log
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
14/08/22 21:17:52 INFO [main-EventThread] (ZkClient.java:450) - zookeeper
state changed (Disconnected)
...
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
connected to zkserver/192.168.1.2:2181
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)
...
connected to zkserver/192.168.1.2:2181
EndOfStreamException: Unable to read additional data from server sessionid
0x14801159e240000, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1190)