Hello,
I'm looking for a way to get access to ClientCnxnSocket.lastSend without trying to break ZooKeeperEncapsulation. Broadly, my goal is to use this in a Java process in order to increase confidence in transactions.
There are resource starved situations where the ClientCnxn.SendThread may not be scheduled for greater than negotiatedSessionTimeout. My understanding is this will lead to session loss because even though the connection might still be considered alive (the OS could be sending ACKs to packets ZK sends), the ZK server requires the client to be sending the packets.
So, assuming
...a JVM process was connected to a ZK ensemble
...the JVM process is performing transactions
...ZK is being used for distributed locking with coarse granularity
...a reliable low-latency network connection to a healthy low-latency ensemble
...a rare event causes the machine hosting the JVM to be resource starved
...none of the JVM threads are scheduled for a window twice the length of the negotiatedSessionTimeout
...during this window, the process has lost the coarse lock on the ensemble (it was an ephemeral node)
Then the ensemble should have agreed that the session is dead, correct? Even though the connection may be considered alive at a TCP/IP transport level. What is more, just coming out of the state where the threads are scheduled, there is a race condition between the ZK threads firing session death event and the transaction threads committing transactions. As I write this, I realize I'm not entirely sure what events ZK would send and in what order, it depending on what was done before the freeze and where it was frozen.
Back to the broad goal, I want to increase confidence in this situation that the process still owns the ZK lock without firing off network events before committing every transaction. Obviously, fine granular locks would solve this problem, but that comes with an unacceptable performance trade off.
Now, let's say I could do something like "long org.apache.zookeeper.ZooKeeper.getLastSent()". Well, I don't know if the ZK server actually received the packet, assuming it did receive the packet I don't know when it received the packet, and I don't know when the OS received the ack. However, it does assert that the SendThread was scheduled and able to call System.getNanos() in ClientCnxnSocket. This increasing the likelihood that the process was sending heartbeats. In addition to this, if I haven't received a push notification from the ZK event thread implying I've lost the lock, I have higher confidence that the session hasn't been lost and that I still have the coarse lock, which satisfies my broad goal somewhat better than the current state.
Any thoughts?
Austin
________________________________
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
I'm looking for a way to get access to ClientCnxnSocket.lastSend without trying to break ZooKeeperEncapsulation. Broadly, my goal is to use this in a Java process in order to increase confidence in transactions.
There are resource starved situations where the ClientCnxn.SendThread may not be scheduled for greater than negotiatedSessionTimeout. My understanding is this will lead to session loss because even though the connection might still be considered alive (the OS could be sending ACKs to packets ZK sends), the ZK server requires the client to be sending the packets.
So, assuming
...a JVM process was connected to a ZK ensemble
...the JVM process is performing transactions
...ZK is being used for distributed locking with coarse granularity
...a reliable low-latency network connection to a healthy low-latency ensemble
...a rare event causes the machine hosting the JVM to be resource starved
...none of the JVM threads are scheduled for a window twice the length of the negotiatedSessionTimeout
...during this window, the process has lost the coarse lock on the ensemble (it was an ephemeral node)
Then the ensemble should have agreed that the session is dead, correct? Even though the connection may be considered alive at a TCP/IP transport level. What is more, just coming out of the state where the threads are scheduled, there is a race condition between the ZK threads firing session death event and the transaction threads committing transactions. As I write this, I realize I'm not entirely sure what events ZK would send and in what order, it depending on what was done before the freeze and where it was frozen.
Back to the broad goal, I want to increase confidence in this situation that the process still owns the ZK lock without firing off network events before committing every transaction. Obviously, fine granular locks would solve this problem, but that comes with an unacceptable performance trade off.
Now, let's say I could do something like "long org.apache.zookeeper.ZooKeeper.getLastSent()". Well, I don't know if the ZK server actually received the packet, assuming it did receive the packet I don't know when it received the packet, and I don't know when the OS received the ack. However, it does assert that the SendThread was scheduled and able to call System.getNanos() in ClientCnxnSocket. This increasing the likelihood that the process was sending heartbeats. In addition to this, if I haven't received a push notification from the ZK event thread implying I've lost the lock, I have higher confidence that the session hasn't been lost and that I still have the coarse lock, which satisfies my broad goal somewhat better than the current state.
Any thoughts?
Austin
________________________________
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.