The problem is I have exported all third party libraries into the lib directory in the jar file, but I still receive java.lang.ClassNotFoundException. As we know, all libraries need to be distributed to all task nodes. In general, there are several ways to put the jar libraries:
- package into jar file
- use “-libjars” command line option of the “hadoop jar “
- load libraries into nodes manually
The first method works well for manually installed Hadoop system, but failed at CDH4. My guess is the CDH has its own wrapper for Hadoop. So the simple solution is the second method: hadoop jar Model.jar hadoop.mr.ElementDriver -libjars lib/your_library.jar /model/input /model/output_element. Using “,” if you have more than one library.