Migrate Java+MySQL to hadoop

The problem is:  we have an existing java project with the back end storage as MySQL. The data is too big to process in memory now. The solution is to use Hadoop.

How to in a nutshell:

  • load data from MySQL into Hive using Sqoop
  • perform JOIN on different tables using Hive
  • write Java code to process join result using Hadoop
  • analyze the output using other tools like R
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s