flume hdfs sink notify / callback to add partition

Hi,

Is there a way to get the hdfs sink to signal that a file was just closed
and then use that signal to add a partition to hive if one does not exist
already.

Right now, what I do is:

- move files to s3
- run recover partitions <--- step takes forever.

But given that I have so much historical data, it's not feasible to run
recover partitions every single day since it takes forever.

I had much rather add an extra partition whenever I see a file in that
partition for the first time.

I looked around the code base and it seems the Flume-OG had something like
this but I don't see the capability in Flume-NG.

I can see a way to adding this by adding another Callback parameter to the
HdfsEventSink and create a customer wrapper around it.

Any other suggestions ?

Thanks,
Viral