We are running ActiveMQ 5.10.0 with a network of brokers. For one topic
called 'results' we have one subscriber running on a single broker 'X' and
many (9 in staging, 24 in production and growing) brokers (let's call them
sensors) connected in with full duplex connections, each broker with a
producer connected and sending messages to that topic. Consumers and
producers are all connected using stomp. brokers are connected using
openwire.
There are other queues and topics on the brokers, but let's focus on just
this one.
Until yesterday, we were using non-durable consumers, and non-persistent
messaging. Yesterday we enabled durable consumers by setting the client-id
on the CONNECT header, and setting activemq.subscriptionName on the
SUBSCRIBE header. At first things seem to work, but very quickly the brokers
on the sensors stopped forwarding messages and went into producer flow
control.
Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query
-QTopic=sensor-results" I should see one network connector section per
sensor broker. However once we enabled durable subscribers, only a few of
the sensor brokers would show up, even though they were actually connected
(tcp connection was active). No amount of restarts of anything fixed it. The
number of sensor brokers connected/subscribed was transient, between 2 and 8
would be connected, usually around 4.
Then it got worse. We rolled back our code change and went back to
non-durable subscribers, but things did not get better. We had
offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule set
to one minute, but the durable subscritions did not clear out. We eventually
shutdown the entire network of brokers, deleted the kahadb directory on
every broker, and restarted everything.
It appears to us that the subscription from single broker 'X' did not make
it to the sensor brokers, or only did sometimes, but then refused to clear
out. We don't know how to further debug this, and would appreciate any help
and suggestions.
thanks
mike
called 'results' we have one subscriber running on a single broker 'X' and
many (9 in staging, 24 in production and growing) brokers (let's call them
sensors) connected in with full duplex connections, each broker with a
producer connected and sending messages to that topic. Consumers and
producers are all connected using stomp. brokers are connected using
openwire.
There are other queues and topics on the brokers, but let's focus on just
this one.
Until yesterday, we were using non-durable consumers, and non-persistent
messaging. Yesterday we enabled durable consumers by setting the client-id
on the CONNECT header, and setting activemq.subscriptionName on the
SUBSCRIBE header. At first things seem to work, but very quickly the brokers
on the sensors stopped forwarding messages and went into producer flow
control.
Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query
-QTopic=sensor-results" I should see one network connector section per
sensor broker. However once we enabled durable subscribers, only a few of
the sensor brokers would show up, even though they were actually connected
(tcp connection was active). No amount of restarts of anything fixed it. The
number of sensor brokers connected/subscribed was transient, between 2 and 8
would be connected, usually around 4.
Then it got worse. We rolled back our code change and went back to
non-durable subscribers, but things did not get better. We had
offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule set
to one minute, but the durable subscritions did not clear out. We eventually
shutdown the entire network of brokers, deleted the kahadb directory on
every broker, and restarted everything.
It appears to us that the subscription from single broker 'X' did not make
it to the sensor brokers, or only did sometimes, but then refused to clear
out. We don't know how to further debug this, and would appreciate any help
and suggestions.
thanks
mike