java.io.FileNotFoundException using Distributed Cache with Eclipse plug-in

Sometime, it is very useful to distribute some file cross nodes for a task. A classical case is JOIN with a small size metadata file. The file can be local or HDFS. 

If you use this Java snippet: DistributedCache.addCacheFile(new URI(“/model/conf/txn_header”), conf); We assume the file is at HDFS, but actually it will look at local file system and generate java.io.FileNotFoundException if you use Eclipse plug-in (no problem with Hadoop command line). To solve  this, please add Hadoop configuration into job config:

conf.addResource(new Path(“/usr/local/hadoop/conf/core-site.xml”));
conf.addResource(new Path(“/usr/local/hadoop/conf/hdfs-site.xml”));

Reference: http://blog.rajeevsharma.in/2009/06/using-hdfs-in-java-0200.html

Advertisements

One thought on “java.io.FileNotFoundException using Distributed Cache with Eclipse plug-in

  1. Woah! I’m really enjoying the template/theme of this blog. It’s simple, yet effective.
    A lot of times it’s very hard to get that “perfect balance” between user friendliness and appearance. I must say you have done a superb job with this. Also, the blog loads super quick for me on Opera. Excellent Blog!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s