Hi all,
I've got a testing activeMQ installation that I'm trying to test regarding
different limits on destinations and memory usage in general, with and without
flowControl enable. I'm running into some issues and I don't quite understand
how this works, so I would appreciate any tips or advise here.
My setup is activeMQ 5.6 in a testing environment. I'm using a couple of
python scripts that use stomp.py to send some messages and read them from a
testqueue. Depending on a couple of settings, the behaviour changes, and it
behaves differently.
I'm sendind 1 Mb messages to the queue as per:
data = "x" * 1024 * 1024
number = 1000
print "producing %s %s kb messages" % (number, len(data) / 1024)
for i in range(number):
print i
conn.send(data, destination='/queue/testqueue')
in python.
Now, I have two settings:
Scenario number 1
<policyEntries>
<policyEntry queue=">" memoryLimit="50mb" producerFlowControl="true">
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
<policyEntry topic=">" memoryLimit="50mb" producerFlowControl="true">
<!-- We want to store both persistent and non-persistent messages -->
<pendingSubscriberPolicy>
<fileCursor />
</pendingSubscriberPolicy>
<pendingDurableSubscriberPolicy>
<fileDurableSubscriberCursor/>
</pendingDurableSubscriberPolicy>
...
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="40 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="800 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
Where the memoryLimit of each of the queues is bigger than the limit for
memoryUsage. When I try to insert into the queue, it insert 32 messages (32Mb)
and it hangs in there. I see this log messages:
[2013-06-05 15:54:19,816] INFO ActiveMQ Transport: tcp:///10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage Manager Memory Limit (52428800) reached on queue://testqueue. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info
2013-06-05 15:54:19,816 [0.0.9.124:36431] INFO Queue - Usage Manager Memory Limit (52428800) reached on queue://testqueue. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info
[2013-06-05 15:54:20,818] INFO ActiveMQ Transport: tcp:///10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage(default:memory:queue://testqueue:memory) percentUsage=62%, usage=32537724, limit=52428800, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%, usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1) stopped to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
2013-06-05 15:54:20,818 [0.0.9.124:36431] INFO Queue - Usage(default:memory:queue://testqueue:memory) percentUsage=62%, usage=32537724, limit=52428800, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%, usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1) stopped to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
Which would indicate that flow control kicked in as per the first message,
that they reach a memory limit (50Mb) as per the per destination limit (second
message). It also says that the memory (40Mb) got used as a 100%.
Also, noted that it says: default:memory:queue.
Scenario number 2
Now, if I change the memoryUsage to be slightly bigger than the memoryLimit on
each queue, from 40 to 60Mb, as per:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="60 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="800 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
taking into account that the per-destination limits are just the same, 50Mb as
per the config above. When I run the same exact procedure, it inserts 801 1Mb
messages and it stops with this message:
[2013-06-05 16:05:01,972] INFO BrokerService.worker.1 org.apache.activemq.store.kahadb.plist.PListStore - PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
2013-06-05 16:05:01,972 [ervice.worker.1] INFO PListStore - PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
[2013-06-05 16:05:12,959] INFO ActiveMQ Transport: tcp:///10.0.9.124:36615 org.apache.activemq.broker.region.Queue - Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944, limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is Full (99% of 838860800). Stopping producer (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
2013-06-05 16:05:12,959 [0.0.9.124:36615] INFO Queue - Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944, limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is Full (99% of 838860800). Stopping producer (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
Where I see that it starts using the tempUsage space, and it pauses when it's
used the 800 Mb available, whining that the temp store is full.
Now, the type is default:temp:queue rather than memory
So,
Could anyone explain to me why this is? I've read quite a few blog posts and
entries on the mailing list and I'm still a bit confused about it, so, to have
more concrete questions,
1) Why this change on the memoryUsage limit affects how activeMQ deals with
way too many messages? More so, why on the second example, the 50 Mb limit
doesn't kick in?
2) I uderstood that activeMQ passes messages to the temp storage area when a
queue is full 70% (unless configured otherwise), but this is not happening
here. Why?
3) I've seen a case where each queue is configured with a 500Mb limit,
memoryUsage limit is 20Mb and tempUsage is 100Mb, activeMQ accepts 350 Mb
worth of messages (70% of 500 Mb) and then it writes it the temporary storage,
and then it hangs, complaining temp storage is 350% of the maximum value.
Then, there's no way to recover activeMQ unless with a restart.
Any ideas/suggestions/pointers would be appreciated, as I'm getting a bit
frustrated trying to understand this :-)
Apologies for the length of the email, hope is easily understandable.
Thanks.
Regards.
I've got a testing activeMQ installation that I'm trying to test regarding
different limits on destinations and memory usage in general, with and without
flowControl enable. I'm running into some issues and I don't quite understand
how this works, so I would appreciate any tips or advise here.
My setup is activeMQ 5.6 in a testing environment. I'm using a couple of
python scripts that use stomp.py to send some messages and read them from a
testqueue. Depending on a couple of settings, the behaviour changes, and it
behaves differently.
I'm sendind 1 Mb messages to the queue as per:
data = "x" * 1024 * 1024
number = 1000
print "producing %s %s kb messages" % (number, len(data) / 1024)
for i in range(number):
print i
conn.send(data, destination='/queue/testqueue')
in python.
Now, I have two settings:
Scenario number 1
<policyEntries>
<policyEntry queue=">" memoryLimit="50mb" producerFlowControl="true">
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
<policyEntry topic=">" memoryLimit="50mb" producerFlowControl="true">
<!-- We want to store both persistent and non-persistent messages -->
<pendingSubscriberPolicy>
<fileCursor />
</pendingSubscriberPolicy>
<pendingDurableSubscriberPolicy>
<fileDurableSubscriberCursor/>
</pendingDurableSubscriberPolicy>
...
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="40 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="800 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
Where the memoryLimit of each of the queues is bigger than the limit for
memoryUsage. When I try to insert into the queue, it insert 32 messages (32Mb)
and it hangs in there. I see this log messages:
[2013-06-05 15:54:19,816] INFO ActiveMQ Transport: tcp:///10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage Manager Memory Limit (52428800) reached on queue://testqueue. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info
2013-06-05 15:54:19,816 [0.0.9.124:36431] INFO Queue - Usage Manager Memory Limit (52428800) reached on queue://testqueue. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info
[2013-06-05 15:54:20,818] INFO ActiveMQ Transport: tcp:///10.0.9.124:36431 org.apache.activemq.broker.region.Queue - Usage(default:memory:queue://testqueue:memory) percentUsage=62%, usage=32537724, limit=52428800, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%, usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1) stopped to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
2013-06-05 15:54:20,818 [0.0.9.124:36431] INFO Queue - Usage(default:memory:queue://testqueue:memory) percentUsage=62%, usage=32537724, limit=52428800, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=101%, usage=42443902, limit=41943040, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:meme-49473-1370444050426-2:1:-1:1) stopped to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
Which would indicate that flow control kicked in as per the first message,
that they reach a memory limit (50Mb) as per the per destination limit (second
message). It also says that the memory (40Mb) got used as a 100%.
Also, noted that it says: default:memory:queue.
Scenario number 2
Now, if I change the memoryUsage to be slightly bigger than the memoryLimit on
each queue, from 40 to 60Mb, as per:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="60 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="800 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
taking into account that the per-destination limits are just the same, 50Mb as
per the config above. When I run the same exact procedure, it inserts 801 1Mb
messages and it stops with this message:
[2013-06-05 16:05:01,972] INFO BrokerService.worker.1 org.apache.activemq.store.kahadb.plist.PListStore - PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
2013-06-05 16:05:01,972 [ervice.worker.1] INFO PListStore - PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
[2013-06-05 16:05:12,959] INFO ActiveMQ Transport: tcp:///10.0.9.124:36615 org.apache.activemq.broker.region.Queue - Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944, limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is Full (99% of 838860800). Stopping producer (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
2013-06-05 16:05:12,959 [0.0.9.124:36615] INFO Queue - Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944, limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is Full (99% of 838860800). Stopping producer (ID:meme-55481-1370444692514-2:1:-1:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s)
Where I see that it starts using the tempUsage space, and it pauses when it's
used the 800 Mb available, whining that the temp store is full.
Now, the type is default:temp:queue rather than memory
So,
Could anyone explain to me why this is? I've read quite a few blog posts and
entries on the mailing list and I'm still a bit confused about it, so, to have
more concrete questions,
1) Why this change on the memoryUsage limit affects how activeMQ deals with
way too many messages? More so, why on the second example, the 50 Mb limit
doesn't kick in?
2) I uderstood that activeMQ passes messages to the temp storage area when a
queue is full 70% (unless configured otherwise), but this is not happening
here. Why?
3) I've seen a case where each queue is configured with a 500Mb limit,
memoryUsage limit is 20Mb and tempUsage is 100Mb, activeMQ accepts 350 Mb
worth of messages (70% of 500 Mb) and then it writes it the temporary storage,
and then it hangs, complaining temp storage is 350% of the maximum value.
Then, there's no way to recover activeMQ unless with a restart.
Any ideas/suggestions/pointers would be appreciated, as I'm getting a bit
frustrated trying to understand this :-)
Apologies for the length of the email, hope is easily understandable.
Thanks.
Regards.