Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

using KmeansDriver with HDFS

$
0
0
Hi all,
I'm using mahout 0.7

I'm trying to use KmeansDriver (org.apache.mahout.clustering.kmeans.KMeansDriver) with HDFS and I'm having some issues.
When I use it with my local file system everything seems to be working fine.
However, as soon as I change the Configuration object to use HDFS:
        Configuration conf = new Configuration();
        conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\core-site.xml"));
        conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\hdfs-site.xml"))

I run into problems

I was looking at the exception I get:

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.RangeCheck(ArrayList.java:547)
    at java.util.ArrayList.get(ArrayList.java:322)
   at org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)

I pulled that code (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and I think is trying to read a file from one of the paths I passed to the method but with a new instance of the configuration object (not the configuration object I passed to the method but one that doesn't have my HDFS configured)

 

205

  public void  [More ...] readFromSeqFiles(Configuration conf, Path path) throws IOException {

206

    Configuration config = new Configuration();

207

    List<Cluster> clusters = Lists.newArrayList();

208

    for (ClusterWritable cw : new SequenceFileDirValueIterable<ClusterWritable>(path, PathType.LIST,

209

        PathFilters.logsCRCFilter(), config)) {

210

      Cluster cluster = cw.getValue();

211

      cluster.configure(conf);

212

      clusters.add(cluster);

213

    }

214

    this.models = clusters;

215

    modelClass = models.get(0).getClass().getName();

216

    this.policy = readPolicy(path);

217

  }

any help would be really appreciated :)

Thanks!

Alan

Viewing all articles
Browse latest Browse all 5648

Trending Articles