Hi,
I'm a developer at the Apache Flink (incubating) project. I'm trying to
pass the version of a dependency using a property from the command line to
the binary build. But it seems that the version I've set is not respected
everywhere.
We have the following setup:
- flink-parent
- flink-core
- ... more modules ...
- flink-yarn
- flink-dist
The "flink-yarn" module (and others) have dependencies to Apache Hadoop. We
have a property (called hadoop.version) to have one place for setting the
Hadoop version. The default Hadoop version 2.2.0.
The "flink-dist" module is creating the binary release for Flink. It is
using the maven-assembly-plugin for that. All jars we depend on are placed
in the "lib" directory.
The "flink-dist" module has most of our modules (also the flink-yarn)
module as a dependency so that we can use the modules in the assembly.
What I'm trying to do is building Flink for a different Hadoop version, say
"2.2.0-cdh5.0.0-beta-2". Therefore, I'm passing this version as
"-Dhadoop.version=2.2.0-cdh5.0.0-beta-2".
However, some files from Hadoop "2.2.0" end up in the "lib" directory.
mvn dependency:tree reveals for "flink-dist":
[INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
however, for "flink-yarn" the versions are correct:
[INFO] +-
org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
Here is the full dependency:tree output:
https://gist.github.com/rmetzger/70c5f35d4cfd06a91169
How can I create builds for different Hadoop versions using Maven?
Regards,
Robert
I'm a developer at the Apache Flink (incubating) project. I'm trying to
pass the version of a dependency using a property from the command line to
the binary build. But it seems that the version I've set is not respected
everywhere.
We have the following setup:
- flink-parent
- flink-core
- ... more modules ...
- flink-yarn
- flink-dist
The "flink-yarn" module (and others) have dependencies to Apache Hadoop. We
have a property (called hadoop.version) to have one place for setting the
Hadoop version. The default Hadoop version 2.2.0.
The "flink-dist" module is creating the binary release for Flink. It is
using the maven-assembly-plugin for that. All jars we depend on are placed
in the "lib" directory.
The "flink-dist" module has most of our modules (also the flink-yarn)
module as a dependency so that we can use the modules in the assembly.
What I'm trying to do is building Flink for a different Hadoop version, say
"2.2.0-cdh5.0.0-beta-2". Therefore, I'm passing this version as
"-Dhadoop.version=2.2.0-cdh5.0.0-beta-2".
However, some files from Hadoop "2.2.0" end up in the "lib" directory.
mvn dependency:tree reveals for "flink-dist":
[INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
however, for "flink-yarn" the versions are correct:
[INFO] +-
org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
Here is the full dependency:tree output:
https://gist.github.com/rmetzger/70c5f35d4cfd06a91169
How can I create builds for different Hadoop versions using Maven?
Regards,
Robert