0

How-to: Use Eclipse with MapReduce in Cloudera’s QuickStart VM

One of the common questions I get from students and developers in my classes relates to IDEs and MapReduce: How do you create a MapReduce project in Eclipse and then debug it?

To answer that question, I have created a screencast showing you how, using Cloudera’s QuickStart VM. The QuickStart VM helps developers get started writing MapReduce code without having to worry about software installs and configuration. Everything is installed and ready to go. You can download the image type that corresponds to your preferred virtualization platform.

Eclipse is installed on the VM and there is a link on the desktop to start it.

MapReduce and Eclipse

You can run and debug MapReduce code in Eclipse just like any other Java program. However, there are a few differences between running MapReduce in a distributed cluster and in an IDE like Eclipse. When you run MapReduce code in Eclipse, Hadoop runs in a special mode called LocalJobRunner, under which all the Hadoop daemons are run in a single JVM (Java Virtual Machine) instead of several different JVMs. Another difference is that all file paths default to local file paths, not HDFS ones.

With those caveats in mind, you can start putting in your breakpoints and debug your MapReduce code like any other Java program.

If you want to clone the same Git project as I do in the screencast, you can find it here. From the terminal, type in:

git clone https://github.com/eljefe6a/UnoExample.git

The project will be cloned to the current directory as a subdirectory.

Note that creating Eclipse projects manually is the easy way to get started.  If you are going to have Hadoop as part of an automated build process, you will want to do this in Maven. In Maven, you can create Eclipse projects — this blog post tells you how. If you want to compile Hadoop from source using Eclipse, this post shows you how.

Conclusion

Whether you to start writing some MapReduce code or debug existing code, the QuickStart VM will help you do it quickly and easily. This screencast walks you through it and gets you coding on your favorite IDE.

Further reading:

Jesse Anderson is an instructor with Cloudera University.

(Jesse just released a series of screencasts about Hadoop MapReduce. It’s published again by the good people at Pragmatic Programmers. These screencasts are the best way for a beginner to learn about Hadoop — unless they’re sitting in his Cloudera University class!)

keven

Leave a Reply

Your email address will not be published. Required fields are marked *