Hi all:
Considering an environment with thousands of sources, which are the best
practices for managing the agent configuration (flume.conf)? Is it
recommended to create a multi-layer topology where each agent takes control
of a subset of sources?
In that case, a conf mgmg server (such as Puppet) would be responsible for
editing flume.conf with parameters 'agent.sources' from source1 to
source3000 (assuming we have 3000 sources machines).
Are my thoughts aligned with that scenarios of large scale data ingest?
Thanks a lot!
JuanFra
Considering an environment with thousands of sources, which are the best
practices for managing the agent configuration (flume.conf)? Is it
recommended to create a multi-layer topology where each agent takes control
of a subset of sources?
In that case, a conf mgmg server (such as Puppet) would be responsible for
editing flume.conf with parameters 'agent.sources' from source1 to
source3000 (assuming we have 3000 sources machines).
Are my thoughts aligned with that scenarios of large scale data ingest?
Thanks a lot!
JuanFra