Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

Using Mahout 1.0-SNAPSHOT with yarn cluster

$
0
0
Hello,

i am working on a web application that should execute lda on a external
yarn cluster.

I am uploading all the relevant sequence files onto the yarn cluter.
This is how it try to remotely execute lda on the cluster.

try {
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
Configuration hdoopConf = new Configuration();
hdoopConf.set("fs.defaultFS",
"hdfs://xxx.xxx.xxx.xxx:9000/user/xx");
hdoopConf.set("yarn.resourcemanager.hostname",
"xxx.xxx.xxx.xxx");
hdoopConf.set("mapreduce.framework.name", "yarn");
hdoopConf.set("mapred.framework.name", "yarn");
hdoopConf.set("mapred.job.tracker", "xxx.xxx.xxx.xxx");
hdoopConf.set("dfs.permissions.enabled", "false");
hdoopConf.set("hadoop.job.ugi", "xx");
hdoopConf.set("mapreduce.jobhistory.address","xxx.xxx.xxx.xxx:10020" );
CVB0Driver driver = new CVB0Driver();
try {
driver.run(hdoopConf,
sparseVectorIn.suffix("/matrix"),
topicsOut, k, numTerms,
doc_topic_smoothening, term_topic_smoothening,
maxIter, iteration_block_size,
convergenceDelta,
sparseVectorIn.suffix("/dictionary.file-0"),
topicsOut.suffix("/DocumentTopics/"), sparseVectorIn,
seed, testFraction, numTrainThreads,
numUpdateThreads, maxItersPerDoc,
numReduceTasks, backfillPerplexity);
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();

return null;

});
} catch (InterruptedException e) {
e.printStackTrace();

I am getting the following error message:

java.lang.InterruptedException: Failed to complete iteration 1 stage 1
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.runIteration(CVB0Driver.java:502)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:319)
at
org.plista.kornakapi.core.training.LDATopicFactorizer$1.run(LDATopicFactorizer.java:227)
at
org.plista.kornakapi.core.training.LDATopicFactorizer$1.run(LDATopicFactorizer.java:208)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
...

Is my general setup correct or did i understood something wrong?

Best,

Max

Viewing all articles
Browse latest Browse all 5648

Trending Articles