Sometime, it is very useful to distribute some file cross nodes for a task. A classical case is JOIN with a small size metadata file. The file can be local or HDFS.
If you use this Java snippet: DistributedCache.addCacheFile(new URI(“/model/conf/txn_header”), conf); We assume the file is at HDFS, but actually it will look at local file system and generate java.io.FileNotFoundException if you use Eclipse plug-in (no problem with Hadoop command line). To solve this, please add Hadoop configuration into job config:
The problem is I have exported all third party libraries into the lib directory in the jar file, but I still receive java.lang.ClassNotFoundException. As we know, all libraries need to be distributed to all task nodes. In general, there are several ways to put the jar libraries:
- package into jar file
- use “-libjars” command line option of the “hadoop jar “
- load libraries into nodes manually
The first method works well for manually installed Hadoop system, but failed at CDH4. My guess is the CDH has its own wrapper for Hadoop. So the simple solution is the second method: hadoop jar Model.jar hadoop.mr.ElementDriver -libjars lib/your_library.jar /model/input /model/output_element. Using “,” if you have more than one library.