[Revisit] Eclipse Hadoop plug-in under Ubuntu Linux

I had a post at: http://webpages.uncc.edu/~fxu/Programming/Eclipse%20hadoop%20plug-in.htm

Set up the Hadoop

First of all, you should set up the Hadoop properly especial the listening port. Check [5] for more details. My hadoop is running on single node model. So go to section “Pseudo-Distributed Operation”, and follow the steps. My hadoop is under “/usr/local/hadoop”.

change conf/core-site.xml:

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

change conf/hdfs-site.xml:

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

change conf/mapred-site.xml:

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

Install Java 1.6

Default Java for Ubuntu 8 is gij 1.5. You can install Java 1.6 from “Synaptic Package Management”. After that, you should modify command “java” to new version.

hadoop@ubuntu:~$ java -version
 java version "1.5.0"
 gij (GNU libgcj) version 4.3.2
Copyright (C) 2007 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions. There is  		NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR  		PURPOSE.
 hadoop@ubuntu:~$ ls -l /usr/bin/java
 lrwxrwxrwx 1 root root 22 2010-01-21 00:07 /usr/bin/java ->  		/etc/alternatives/java
 hadoop@ubuntu:~$ ls -l /usr/bin/javac
 lrwxrwxrwx 1 root root 23 2010-01-21 00:07 /usr/bin/javac ->  		/etc/alternatives/javac
 hadoop@ubuntu:~$ ls /usr/lib/jvm/ -l
 total 8
 drwxr-xr-x 7 root root 4096 2010-01-21 12:19 java-1.5.0-gcj-4.3-1.5.0.0
 lrwxrwxrwx 1 root root 19 2010-01-21 00:06 java-6-sun ->  		java-6-sun-1.6.0.14
 drwxr-xr-x 8 root root 4096 2010-01-21 00:06 java-6-sun-1.6.0.14
 lrwxrwxrwx 1 root root 26 2010-01-21 12:18 java-gcj ->  		java-1.5.0-gcj-4.3-1.5.0.0

hadoop@ubuntu:~$ sudo rm /usr/bin/javac
 hadoop@ubuntu:~$ sudo rm /usr/bin/java
 hadoop@ubuntu:~$ sudo ln -s /usr/lib/jvm/java-6-sun/bin/javac /usr/bin/javac
 hadoop@ubuntu:~$ sudo ln -s /usr/lib/jvm/java-6-sun/bin/java /usr/bin/java
 hadoop@ubuntu:~$ java -version
 java version "1.6.0_14"
 Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
 Java HotSpot(TM) Client VM (build 14.0-b16, mixed mode)

Install Eclipse 3.5

You can directly install Eclipse 3.2 version from “Synaptic Package Management”, but this does not match the plug-in requirement. Manually download 3.5 from [1] and check [2] on how to install it. In my case, I install Eclipse 3.5 under “/home/hadoop/bin/packages/eclipse3.5”.

If you are using Ubuntu 9, you can directly install 3.5 from “Synaptic Package Management”.

Install Eclipse plug-in

Please check the official document [3] to have a brief idea. Then download the plug-in from [4]. I download it to the desktop. Copy the file into Eclipse plgu-in directory using command terminal:

Type: cp /home/hadoop/Desktop/hadoop-0.20.1-eclipse-plugin.jar /home/hadoop/bin/packages/eclipse3.5/plugins/

Set up Eclipse

Open eclipse, go to manu “Window -> Preferences”

  • Set up the “Hadoop map/Reduce” to know where is the hadoop installation.

  • set up “Installed JREs” to use the Java 1.6

Now you can start hadoop manually or it starts automatically.

go to “Window -> open perspective -> Other..”,  choose “Map/Reduce”.

Right click in the perspective, and choose “New Hadoop Location…”. The port number is coming from the configuration file.

You also can directly operate on hadoop FS through “DFS Locations”.

There are several updates for that post:

  • I am using Ubuntu 10.04, Eclipse 3.5.2 and hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar now
  • if you install the eclipse from package management, there is no directory “/home/hadoop/bin/packages/eclipse3.5/plugins/“. Instead, you should copy the jar file downloaded to “/usr/lib/eclipse/plugins/” with sudo
  • check out the hfs and map/reduce ports number from “conf/core-site.xml” and “conf/mapred-site.xml”, where are 54310 and 54311 respectively.
Advertisements

2 thoughts on “[Revisit] Eclipse Hadoop plug-in under Ubuntu Linux

  1. Hi,

    I am trying to run Hadoop Eclipse Plug-in that ships with Hadoop 0.21.0 on Eclipse Helios.
    I did pretty much what you have described in your post, but I get several java.lang.NoClassDefFoundErrors.

    Am I missing any Environment variables or classpath settings?

    org.osgi.framework.BundleException: Exception in org.apache.hadoop.eclipse.Activator.stop() of bundle org.apache.hadoop.eclipse.
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:863)
    at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:501)
    at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:550)
    at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1097)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:597)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
    at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:266)
    at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:690)
    at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:588)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.shutdown(EclipseStarter.java:415)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:198)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:620)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1384)
    Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.eclipse.servers.ServerRegistry
    at org.apache.hadoop.eclipse.Activator.stop(Activator.java:63)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:843)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:836)
    … 19 more
    Root exception:
    java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.eclipse.servers.ServerRegistry
    at org.apache.hadoop.eclipse.Activator.stop(Activator.java:63)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:843)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:836)
    at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:501)
    at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:550)
    at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1097)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:597)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
    at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:266)
    at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:690)
    at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:588)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.shutdown(EclipseStarter.java:415)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:198)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:620)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1384)
    !SESSION 2011-10-28 02:19:43.500 ———————————————–
    eclipse.buildId=M20110210-1200
    java.version=1.6.0_26
    java.vendor=Sun Microsystems Inc.
    BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
    Framework arguments: -product org.eclipse.epp.package.java.product
    Command-line arguments: -os linux -ws gtk -arch x86_64 -product org.eclipse.epp.package.java.product -clean

    !ENTRY org.eclipse.ui.navigator 4 2 2011-10-28 02:19:57.000
    !MESSAGE Problems occurred when invoking code from plug-in: “org.eclipse.ui.navigator”.
    !STACK 1
    org.eclipse.core.runtime.CoreException: Plug-in “org.apache.hadoop.eclipse” was unable to instantiate class “org.apache.hadoop.eclipse.dfs.DFSContentProvider”.
    at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.throwException(RegistryStrategyOSGI.java:194)
    at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:188)
    at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:904)
    at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
    at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:55)
    at org.eclipse.ui.internal.navigator.extensions.NavigatorContentDescriptor.createContentProvider(NavigatorContentDescriptor.java:377)
    at org.eclipse.ui.internal.navigator.extensions.NavigatorContentExtension$1.run(NavigatorContentExtension.java:129)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.ui.internal.navigator.extensions.NavigatorContentExtension.internalGetContentProvider(NavigatorContentExtension.java:125)
    at org.eclipse.ui.internal.navigator.NavigatorContentService.extractContentProviders(NavigatorContentService.java:1214)
    at org.eclipse.ui.internal.navigator.NavigatorContentService.updateService(NavigatorContentService.java:391)
    at org.eclipse.ui.internal.navigator.NavigatorContentServiceContentProvider.inputChanged(NavigatorContentServiceContentProvider.java:107)
    at org.eclipse.jface.viewers.ContentViewer.setInput(ContentViewer.java:270)
    at org.eclipse.jface.viewers.StructuredViewer.setInput(StructuredViewer.java:1664)
    at org.eclipse.ui.navigator.CommonNavigator.createPartControl(CommonNavigator.java:212)
    at org.eclipse.ui.navigator.resources.ProjectExplorer.createPartControl(ProjectExplorer.java:75)
    at org.eclipse.ui.internal.ViewReference.createPartHelper(ViewReference.java:375)
    at org.eclipse.ui.internal.ViewReference.createPart(ViewReference.java:229)
    at org.eclipse.ui.internal.WorkbenchPartReference.getPart(WorkbenchPartReference.java:595)
    at org.eclipse.ui.internal.PartPane.setVisible(PartPane.java:313)
    at org.eclipse.ui.internal.ViewPane.setVisible(ViewPane.java:529)
    at org.eclipse.ui.internal.presentations.PresentablePart.setVisible(PresentablePart.java:180)
    at org.eclipse.ui.internal.presentations.util.PresentablePartFolder.select(PresentablePartFolder.java:270)
    at org.eclipse.ui.internal.presentations.util.LeftToRightTabOrder.select(LeftToRightTabOrder.java:65)
    at org.eclipse.ui.internal.presentations.util.TabbedStackPresentation.selectPart(TabbedStackPresentation.java:473)
    at org.eclipse.ui.internal.PartStack.refreshPresentationSelection(PartStack.java:1254)
    at org.eclipse.ui.internal.PartStack.createControl(PartStack.java:666)
    at org.eclipse.ui.internal.PartStack.createControl(PartStack.java:574)
    at org.eclipse.ui.internal.PartSashContainer.createControl(PartSashContainer.java:568)
    at org.eclipse.ui.internal.PerspectiveHelper.activate(PerspectiveHelper.java:272)
    at org.eclipse.ui.internal.Perspective.onActivate(Perspective.java:981)
    at org.eclipse.ui.internal.WorkbenchPage.onActivate(WorkbenchPage.java:2632)
    at org.eclipse.ui.internal.WorkbenchWindow$27.run(WorkbenchWindow.java:2992)
    at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
    at org.eclipse.ui.internal.WorkbenchWindow.setActivePage(WorkbenchWindow.java:2973)
    at org.eclipse.ui.internal.WorkbenchWindow$21.runWithException(WorkbenchWindow.java:2290)
    at org.eclipse.ui.internal.StartupThreading$StartupRunnable.run(StartupThreading.java:31)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3515)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3164)
    at org.eclipse.ui.application.WorkbenchAdvisor.openWindows(WorkbenchAdvisor.java:803)
    at org.eclipse.ui.internal.Workbench$31.runWithException(Workbench.java:1567)
    at org.eclipse.ui.internal.StartupThreading$StartupRunnable.run(StartupThreading.java:31)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3515)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3164)
    at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2548)
    at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2438)
    at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:671)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:664)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:115)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:620)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1384)
    Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
    at org.apache.hadoop.eclipse.server.HadoopServer.(HadoopServer.java:237)
    at org.apache.hadoop.eclipse.servers.ServerRegistry.load(ServerRegistry.java:91)
    at org.apache.hadoop.eclipse.servers.ServerRegistry.(ServerRegistry.java:68)
    at org.apache.hadoop.eclipse.servers.ServerRegistry.(ServerRegistry.java:49)
    at org.apache.hadoop.eclipse.dfs.DFSLocationsRoot.(DFSLocationsRoot.java:52)
    at org.apache.hadoop.eclipse.dfs.DFSContentProvider.(DFSContentProvider.java:73)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:355)
    at java.lang.Class.newInstance(Class.java:308)
    at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:184)
    … 66 more
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:506)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    … 79 more

    Appreciate your help.

    Thank you very much.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s