I am using Kazoo client and see the following error:
Aug 9 07:07:20 a2s1 python[2085]: File "/usr/lib/python2.7/site-packages/kazoo/client.py", line 894, in set
Aug 9 07:07:20 a2s1 python[2085]: return self.set_async(path, data, version).get()
Aug 9 07:07:20 a2s1 python[2085]: File "/usr/lib64/python2.7/site-packages/gevent/event.py", line 223, in get
Aug 9 07:07:20 a2s1 python[2085]: raise self._exception
Aug 9 07:07:20 a2s1 python[2085]: OperationTimeoutException: operation timeout
zookeeper logs around the error time:
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
…
2013-08-09 07:07:19,827 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0001, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:19,828 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 1000] - Closed socket connection for client /127.0.0.1:54698 which had sessionid 0x1406356c3bc0001
2013-08-09 07:07:20,246 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0000, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:20,246 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 1000] - Closed socket connection for client /127.0.0.1:54579 which had sessionid 0x1406356c3bc0000
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,680 [myid:] - ERROR [SyncThread:0:NIOServerCnxn [ at ] 180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
[root [ at ] a2s1 zookeeper]# grep fsync /var/log/zookeeper/zookeeper.out
2013-08-09 07:06:52,280 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 1724ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:06:58,315 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2378ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:01,389 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 1113ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
Could the client (Kazoo) be timing out because of fsync delay? What parameter would control duration for OperationTimeoutException that I can perhaps increase to verify? There is only ZooKeeper client and the load isn't much - 1 read/sec and 2 writes/sec roughly. Zookeeper configuration is default. Kazoo client params are also default.
Thanks
Deepinder
Aug 9 07:07:20 a2s1 python[2085]: File "/usr/lib/python2.7/site-packages/kazoo/client.py", line 894, in set
Aug 9 07:07:20 a2s1 python[2085]: return self.set_async(path, data, version).get()
Aug 9 07:07:20 a2s1 python[2085]: File "/usr/lib64/python2.7/site-packages/gevent/event.py", line 223, in get
Aug 9 07:07:20 a2s1 python[2085]: raise self._exception
Aug 9 07:07:20 a2s1 python[2085]: OperationTimeoutException: operation timeout
zookeeper logs around the error time:
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
…
2013-08-09 07:07:19,827 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0001, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:19,828 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 1000] - Closed socket connection for client /127.0.0.1:54698 which had sessionid 0x1406356c3bc0001
2013-08-09 07:07:20,246 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1406356c3bc0000, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:20,246 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn [ at ] 1000] - Closed socket connection for client /127.0.0.1:54579 which had sessionid 0x1406356c3bc0000
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,680 [myid:] - ERROR [SyncThread:0:NIOServerCnxn [ at ] 180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
[root [ at ] a2s1 zookeeper]# grep fsync /var/log/zookeeper/zookeeper.out
2013-08-09 07:06:52,280 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 1724ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:06:58,315 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2378ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:01,389 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 1113ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog [ at ] 321] - fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
Could the client (Kazoo) be timing out because of fsync delay? What parameter would control duration for OperationTimeoutException that I can perhaps increase to verify? There is only ZooKeeper client and the load isn't much - 1 read/sec and 2 writes/sec roughly. Zookeeper configuration is default. Kazoo client params are also default.
Thanks
Deepinder