Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

Writing big files : stream or file ?

$
0
0
Hello.

We have to produce some rather big volumes of data and generate output
files, in several steps, some of these steps using splitting to process
inputs. I've been looking for two ways...

The first way is writing a file, setting an "append" mode, that could be a
nice solution to write batches of lines, but I suppose that the file if
closed and reopened for each exchange, and thus may be a quite bad solution
when writing millions of line one by one from a splitting loop.

The second way is to use a stream, using "stream:file", but I've not found
any way to control actually how the file is closed. FYI, we use Camel with
Service Mix with an emdedded 2.10, and the "closeOnDone" option is not
available. The "autoCloseCount" seemed to be the beginning of a solution,
but as the value can't be set dynamically, using a property for example,
this doesn't give a full control on the file closing, and outputs might stay
opened, and make fail the following tasks.

Any idea to write big files and control how to close them ?

Thanks in advance.

Regards.

Viewing all articles
Browse latest Browse all 5648

Trending Articles