Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

pig not respecting parameters in mapred-site.xml

$
0
0
I am running pig on a custom hadoop implementation but it doesnt respect
params in mapred-site.xml.

Looking into the code, I find that the following 2 files are slightly
different from stock hadoop in that some patches are not present.

hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
and
src/mapred/org/apache/hadoop/mapred/JobConf.java

Given the constraint that I cannot modify these files, what change should I
make within pig to recognize mapred-site.xml parameters?

I pulled in PIG-3135 and PIG-3145 which make changes to
org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java

But the params in mapred-site.xml are still not getting recognized. Upon
remote eclipse debugging with breakpoints in the file above, this is what
I found -

HExecutionEngine.java - jc = new jobConf()
calls
1st call upon JobConf() constructor -
Configuration.get(String)
Configuration.getProps() --> if properties ==null, properties = new
Properties(); loadResources(properties, resources...);

JobConf static constructor -
Configuration.addDefaultResource(mapred-site.xml)

HExecutionEngine: 2nd call jc.addResource("mapred-site.xml") -
Configuration.get
val = getProps().getProperty(name)
if (val==null) val =
<custom_hadoop_impl_configuration_object>.getDefault(name);

HExecutionEngine: 3rd call recomputeProperties(jc, properties) -->
clearing properties which were added so it gets reloaded again.

What do I ned to do to make sure the getProps().getProperty call is not
null so that the mapred-site.xml values are not over-ridden by defaults in
custom_hadoop implementation ?

Thanks,
Suhas.

Viewing all articles
Browse latest Browse all 5648

Trending Articles