Quantcast
Viewing all articles
Browse latest Browse all 5648

Connection issues to Zookeeper caused shutting down

We've occasional connection issues from client (Storm Supervisor) and
zk running in EC2 VPC, here is the log:

Storm Supervisor:

2014-05-11 00:09:52 o.a.z.ClientCnxn [INFO] Unable to read additional
data from server sessionid 0x244ec3718bb0089, likely server has closed
socket, closing socket connection and attempting reconnect
2014-05-11 00:09:52 c.n.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2014-05-11 00:09:52 c.n.c.f.s.ConnectionStateManager [WARN] There are
no ConnectionStateListeners registered.
2014-05-11 00:09:52 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.

One of the zk (out of three):

2014-05-11 00:09:51,696 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory [ at ] 197] -
Accepted socket connection from /10.0.13.21:58611
2014-05-11 00:09:51,697 [myid:2] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer [ at ] 793] -
Connection request from old client /10.0.13.21:58611; will be dropped
if server is in r-o mode
2014-05-11 00:09:51,697 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer [ at ] 832] -
Client attempting to renew session 0x244ec3718bb008f at
/10.0.13.21:58611
2014-05-11 00:09:51,698 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer [ at ] 595] -
Established session 0x244ec3718bb008f with negotiated timeout 20000
for client /10.0.13.21:58611
2014-05-11 00:09:52,092 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader [ at ] 490] - Shutting down
2014-05-11 00:09:52,109 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader [ at ] 496] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Only 1 followers, need 1
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:496)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:471)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:753)
2014-05-11 00:09:52,121 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.61:39639 which had sessionid
0x344ec37d11b0071
2014-05-11 00:09:52,121 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.62:36706 which had sessionid
0x144ec37184100e0
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.63:49909 which had sessionid
0x144ec37184100f0
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.63:49240 which had sessionid
0x244ec3718bb0105
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.61:39636 which had sessionid
0x144ec37184100f4
2014-05-11 00:09:52,123 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.62:47862 which had sessionid
0x244ec3718bb0089
2014-05-11 00:09:52,123 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.62:36709 which had sessionid
0x344ec37d11b00e8
2014-05-11 00:09:52,124 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.61:39633 which had sessionid
0x144ec37184100f5
2014-05-11 00:09:52,124 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.62:36710 which had sessionid
0x144ec37184100e3
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.62:37290 which had sessionid
0x244ec3718bb00fc
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.61:39640 which had sessionid
0x344ec37d11b00f2
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.63:49910 which had sessionid
0x344ec37d11b00e0
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.63:49906 which had sessionid
0x344ec37d11b0064
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.61:39634 which had sessionid
0x344ec37d11b00e1
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.63:49908 which had sessionid
0x144ec37184100ec
2014-05-11 00:09:52,127 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn [ at ] 1001] - Closed
socket connection for client /10.0.13.21:58611 which had sessionid
0x244ec3718bb008f
2014-05-11 00:09:52,127 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer [ at ] 419] -
shutting down
2014-05-11 00:09:52,128 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:SessionTrackerImpl [ at ] 225] -
Shutting down
2014-05-11 00:09:52,128 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:PrepRequestProcessor [ at ] 743] -
Shutting down
2014-05-11 00:09:52,138 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ProposalRequestProcessor [ at ] 88]
- Shutting down
2014-05-11 00:09:52,138 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor [ at ] 181] -
Shutting down

Anyway I can troubleshoot the problem?

Thanks

Viewing all articles
Browse latest Browse all 5648

Trending Articles