Find records contianing IP address using MapReduce Streaming

The python code is named as “GetIP.py”, which outputs every line containing IP address. The delimiter of each line is a single space.

#!/usr/bin/python

import sys
import socket

for line in sys.stdin:
 fields = line.strip().split(" ")
 value = fields[len(fields)-1]
 try:
 socket.inet_aton(value)
 print line.strip()
 except socket.error:
 continue

Run the code:

hadoop@fxu-t60:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-0.20.2-streaming.jar -D mapred.reduce.tasks=1 -input /home/hadoop/tld -output /home/hadoop/tld-ipĀ  -mapper 'GetIP.py' -file ~/Desktop/data/TLD/GetIP.py
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s