Problem:
We're experiencing a problem whereby openwire communication completely locks
up and the tcp send and receive buffers fill up preventing any further
communication between brokers, whilst the stomp client connectors continue
to function perfectly.
Below is an example of this happening between our hub broker (B) and another
broker (C)
$ netstat -a | grep :61616
tcp6 0 0 [::]:61616 [::]:* LISTEN
tcp6 0 0 B:34326 A:61616 ESTABLISHED
tcp6 116166 142456 B:39099 C:61616 ESTABLISHED
tcp6 115169 156860 B:39100 C:61616 ESTABLISHED
tcp6 0 0 B:34328 A:61616 ESTABLISHED
Topology:
I've reduced our mesh down to 3 nodes, A <-> B <-> C where the connectors
are dual openwire connections, using failover:(static:(nio://...)), one with
topics being pushed down it and the other with queues, both in duplex. All
networkConnectors are initiated from B
Usage summary:
We are primarily using the mesh for MCollective
(http://puppetlabs.com/mcollective) which only utilises the STOMP transport
connectors directly, servicing approximately 600 nodes on a roughly equal
distribution across the three brokers.
We also have a team using virtual topics primarily via C (that will
multiplex to queues in the presence of consumers on the relevant
destinations - usually 2 to any one virtual queue), but with a couple of
consumers on B. We are still trying to gauge their level of usage in some
meaningful metrics, but my understanding is that the volume of produced
messages is not high.
Configuration:
E.g. networkConnectors from B to A
<networkConnector
name="B-A-topics"
uri="static:(failover:(nio://A:61616))"
userName="amq"
password="secret"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="4"
dynamicOnly="true">
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="B-A-queues"
uri="static:(failover:(nio://A:61616))"
userName="amq"
password="secret"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="4"
dynamicOnly="true"
conduitSubscriptions="false">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
Other information:
Version 5.9.0
Ubuntu 10.04
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
Any help or advice is very much appreciated
We're experiencing a problem whereby openwire communication completely locks
up and the tcp send and receive buffers fill up preventing any further
communication between brokers, whilst the stomp client connectors continue
to function perfectly.
Below is an example of this happening between our hub broker (B) and another
broker (C)
$ netstat -a | grep :61616
tcp6 0 0 [::]:61616 [::]:* LISTEN
tcp6 0 0 B:34326 A:61616 ESTABLISHED
tcp6 116166 142456 B:39099 C:61616 ESTABLISHED
tcp6 115169 156860 B:39100 C:61616 ESTABLISHED
tcp6 0 0 B:34328 A:61616 ESTABLISHED
Topology:
I've reduced our mesh down to 3 nodes, A <-> B <-> C where the connectors
are dual openwire connections, using failover:(static:(nio://...)), one with
topics being pushed down it and the other with queues, both in duplex. All
networkConnectors are initiated from B
Usage summary:
We are primarily using the mesh for MCollective
(http://puppetlabs.com/mcollective) which only utilises the STOMP transport
connectors directly, servicing approximately 600 nodes on a roughly equal
distribution across the three brokers.
We also have a team using virtual topics primarily via C (that will
multiplex to queues in the presence of consumers on the relevant
destinations - usually 2 to any one virtual queue), but with a couple of
consumers on B. We are still trying to gauge their level of usage in some
meaningful metrics, but my understanding is that the volume of produced
messages is not high.
Configuration:
E.g. networkConnectors from B to A
<networkConnector
name="B-A-topics"
uri="static:(failover:(nio://A:61616))"
userName="amq"
password="secret"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="4"
dynamicOnly="true">
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="B-A-queues"
uri="static:(failover:(nio://A:61616))"
userName="amq"
password="secret"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="4"
dynamicOnly="true"
conduitSubscriptions="false">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
Other information:
Version 5.9.0
Ubuntu 10.04
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
Any help or advice is very much appreciated