maven-hadoop-plugin

Submitting hadoop job on remote machine is not a complicated process but it takes a lot time, it could be 10 minutes or sometime 15. There are a lot steps to do to get the final result of map reduce and download it to local file system.

-compile map reduce code and build jar file
-upload jar to remote server via ftp
-connect to server via ssh client
-prepare input data in HDFS (usually only once)
-submit job using ‘hadoop jar…’ command
-copy map reduce output from HDFS to remote server local file system
-download output to local machine

Like I mentioned, doing those entire steps manually takes time. I was seeking for some ways to do it faster and easier. After short research, because I didn’t find any tool or solution, so I state the easier and the best way is to develop something myself. I was wondering how to do this. I could code some standalone application for doing this but it would not be enough comfortable again and require few steps from the user like switching from java IDE to another window and finding jar in file system. I thought maybe it will be better to write eclipse plugin. Everything would be in one place but would have some weakness also – no usage outside of eclipse.  Next thought was Maven. Integrated with … everything could be used in the console and what important plugin development for it is easy and pleasant, so I took this idea started development.

The result of my work looks really well. Now I submit my job by ‘one button click’. I have my maven execution configured in eclipse (hadoop:execute)

 My sample wordcount need to have maven-hadoop-plugin configured in its pom.xmll file

<build>
 <plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-hadoop-plugin</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <configuration>
    <host>ipAddress</host>
    <login>login</login>
    <password>password</password>
    <outputDir>output30</outputDir>
    <hdfsOutputDir>
      /books/users/gkolpu/output30
    </hdfsOutputDir>
    <hdfsInputDir>
      /books/input
    </hdfsInputDir>
    <className>
      org.gkolpu.hadoop.BookWordCounter
    </className>
    <jarName>WordCounter.jar</jarName>
   </configuration>
  </plugin>
 </plugins>
</build>

Running my maven hadoop:execute goal from eclipse I see all logs coloured in IDE console

When the job finished on remote server map reduce output is downloaded automatically to target directory

Output files can be easily opened in IDE editors

You can find this plugin with source code on my git-hub repository. There is one only one goal but this project is still under development and other maven goals are planned.

https://github.com/gkolpuc/maven-hadoop-plugin

Map Reduce implementation with Hadoop

Hadoop is an open source framework which supports big data distributed application. One od main features is MapReduce algorithm implementation. Hadoop gives us opportunity to use it's API to implement our own Maping and Reducing. To run your first hadoop job you will need to implement generic Mapper and Reducer classes. See examples below.

public class Map extends MapReduceBase implements
  Mapper {

 private final IntWritable one = new IntWritable(1);
 private Text word = new Text();

 public void map(LongWritable key, Text value,
   OutputCollector output, Reporter reporter)
   throws IOException {

  String line = value.toString();
  StringTokenizer tokenizer = new StringTokenizer(line);
  while (tokenizer.hasMoreTokens()) {
   word.set(tokenizer.nextToken());
   output.collect(word, one);
  }
 }
}

public class Reduce extends MapReduceBase implements
  Reducer {
 public void reduce(Text key, Iterator values,
   OutputCollector output, Reporter reporter)
   throws IOException {
  int sum = 0;

  while (values.hasNext()) {
   sum += values.next().get();
  }

  output.collect(key, new IntWritable(sum));
 }
}

Last thing you need to do is clip them together using Job Configuration

public class Job {

 public static final void main(String[] args) throws Exception {

  JobConf conf = new JobConf(Job.class);
  conf.setJobName("Hadoop-Workshop-GKOLPU");

  conf.setOutputKeyClass(Text.class);
  conf.setOutputValueClass(IntWritable.class);

  conf.setMapperClass(Map.class);
  conf.setCombinerClass(Reduce.class);
  conf.setReducerClass(Reduce.class);

  conf.setInputFormat(TextInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);

 FileInputFormat.setInputPaths(conf,new Path(args[1]));
 FileOutputFormat.setOutputPath(conf,new Path(args[2]));

  JobClient.runJob(conf);
 }
}

Hadoop. Using HDFS.

One of Hadoop concepts is having consistent file system for all provided cluster nodes. For Hadoop Distributed File System avilable are unix-based commands set. You can read full syntax calling hadoop dsf -help, nevertheless I atached part of it below. -fs [local | <file system URI>]: Specify the file system to use.

-ls <path>: List the contents that match the specified file pattern.

-lsr <path>: Recursively list the contents that match the specified file pattern.

-mv <src> <dst>: Move files that match the specified file pattern <src>

to a destination <dst>. -cp <src> <dst>: Copy files that match the file pattern <src> to a destination.

-rm [-skipTrash] <src>: Delete all files that match the specified file pattern. Equivalent to the Unix command "rm <src>"

-rmr [-skipTrash] <src>: Remove all directories which match the specified file pattern. Equivalent to the Unix command "rm -rf <src>"

-put <localsrc> ... <dst>: Copy files from the local file system into fs.

-copyFromLocal <localsrc> ... <dst>: Identical to the -put command.

-moveFromLocal <localsrc> ... <dst>: Same as -put, except that the source is deleted after it's copied.

-get [-ignoreCrc] [-crc] <src> <localdst>: Copy files that match the file pattern <src> to the local name. <src> is kept.

-cat <src>: Fetch all files that match the file pattern <src> and display their content on stdout.

-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>: Identical to the -get command.

-mkdir <path>: Create a directory in specified location.

-tail [-f] <file>: Show the last 1KB of the file. The -f option shows apended data as the file grows.

Hadoop. MapReduce File InputFormat

In hadoop it is very important how we read data for Map Reduce. Standard input is a file set in HDFS. I will try to explain how to define your own file input format. First step is implementing InputFormat interface or extend one of it's implementations like in the example below.

public class EmailInputFormat extends FileInputFormat {

 @Override
 public RecordReader getRecordReader(InputSplit split,
  JobConf job, Reporter reporter) throws IOException {
  reporter.setStatus(split.toString());
  return new EmailRecordReader(job, (FileSplit) split);
 }

}

getRecordReader method have to return RecordReader implementation, so let's create one more class.

public class EmailRecordReader implements RecordReader {
 private LineRecordReader lineReader;
 private LongWritable lineKey;
 private Text lineValue;

 public EmailRecordReader(JobConf job, FileSplit split) throws 
                 IOException {
  lineReader = new LineRecordReader(job, split);

  lineKey = lineReader.createKey();
  lineValue = lineReader.createValue();
 }

 public boolean next(Text key, Email value) throws 
                 IOException {
  // TODO Auto-generated method stub
  // put your code here

  // --
  return false;
 }

 public Text createKey() {
  return new Text("");
 }

 public Email createValue() {
  return new Email();
 }

 public long getPos() throws IOException {
  return lineReader.getPos();
 }

 public void close() throws IOException {
  lineReader.close();
 }

 public float getProgress() throws IOException {
  return lineReader.getProgress();
 }

}

We need to implement constructor to utilize FileSpit but the most important part of this code is next method. Hadoop enginne will be running this method until returning false. So we can use it to produce how much input records we want.

Hadoop. MapReduce File OutputFormat.

Hadoop provide much API for MapReduce customizing. Almost ever we want to get results in some specific format. We just nedd to do two things. Implement OutputFormat interface or extend it's implementation. See the code below.

public class EmailXmlOutputFormat extends FileOutputFormat {

 public RecordWriter getRecordWriter(FileSystem ignored, 
                  JobConf job,String name, Progressable progress)
                  throws IOException {
  Path file = FileOutputFormat
                         .getTaskOutputPath(job, name);
  FileSystem fs = file.getFileSystem(job);
  FSDataOutputStream fileOut=fs.create(file, progress);
  return new EmailXmlRecordWriter(fileOut);
 }
}

To make code complete implementing RecordWriter...

public class EmailXmlRecordWriter implements RecordWriter {
 private static final String utf8 = "UTF-8";

 private DataOutputStream out;

 public EmailXmlRecordWriter(DataOutputStream out) 
    throws IOException {
  this.out = out;
  out.writeBytes("\n");
 }

 public synchronized void write(Text key, Text value) 
    throws IOException {

  boolean nullKey = key == null;
  boolean nullValue = value == null;

  if (nullKey && nullValue) {
   return;
  }
 
  //TODO
  //put your code here 
  //write to out stream

 }

 public synchronized void close(Reporter reporter) 
    throws IOException {
  try {
   out.writeBytes("\n");
  } finally {
   out.close();
  }
 }
}

Firstable coding constructor and close method. Then putting some logic into write. Done.

Digitals Days 2012. Hadoop Workshop.

Two weeks ago I attended Digital Days conference. I was IT event organized by students from Warsaw University of Technology.

Me and my buddy Kuba were representing our company Acxiom and performing Hadoop workshop. We prepared 45 minutes lecture and 6 practical exercises for students. Interest of workshop surprised us a lot, the classroom was almost full.

At the beginning we had a lot technical problems with hadoop environment. Organisation of conference leaved a lot to be desired. At the morning we were really shocked, there was nothing. I have prepared student's stations and Kuba did his best to setup hadoop. Unfortunately we had not enough time and were not able to accomplish whole workshop plan.

Nevertheless, it was my first time being presenter and have very good impression about that. A few students contact me via email next days, so there was some feedback 

Due this event I've created 4 post for workshop attendies to help them with excercises.