Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

Rate limit producer to match consumer

$
0
0
We are trying to improve the responsiveness of some bulk message
processes such that a large batch does not flood a queue and prevent
subsequent smaller batches getting through in a timely fashion. For
example, a job to import millions of records from CSV may take an hour
but a smaller job to import just one thousand records should be able to
begin processing immediately in parallel instead of being sent to the
back of a very long queue.

This seems to me like the sort of thing Camel should be good at but I
have so far not been about to see how this could be achieved. The idea
we have in mind is to have a queue with limited size that will block
when it is full so that the rate of queuing from a large batch would be
limited to consumer capacity. Subsequent batches would have equal
opportunity to get the next message on to the queue.

At first I thought the Camel maxInflightExchanges property could be used
for this but I don't think it has this affect. Is there a way a Camel
route can inspect the size of the target queue to decide whether to
suspend or resume?

Perhaps a message broker can help solve this with either blocking queues
or virtual aggregate queues but I haven't found these in RabbitMQ or
ActiveMQ.

Does anyone have any advice on a way solve this problem with Camel or
otherwise?

- Nathan

Viewing all articles
Browse latest Browse all 5648

Trending Articles