Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

Large file processing using ExecutorService and IBM MQ

$
0
0
Hi,
I have a requirement where I need to read quite a lot of large files. For
each file, store each record by record (separated by new line delimiter)
into the database. Now I have used ExecutorService with a fixed thread pool
of 100, in Camel split component to split the file (say has 20,000 records)
and send 200 messages per thread to IBM MQ Websphere.

Now I created another processor which reads these messages from the IBM's
Message Queue. Here I have configured as
<bean id="websphere" class="org.apache.camel.component.jms.JmsComponent">
<property name="concurrentConsumers" value="200" />
<property name="maxConcurrentConsumers" value="200" />
<property name="taskExecutor" ref="taskExecutor" />

1) Does this mean there are 200 threads to consume 200 message?
2) Need some more information on ExecutorService and Concurrent Consumers.

Also, this configuration would invoke 200 parallel message consumption. But
how much ever i increase my thread size and concurrent consumers, my
processing speed is slower than a stand alone program which processes that
file spawning multiple threads.

Thanks!!

Viewing all articles
Browse latest Browse all 5648

Trending Articles