Set up environment for mapreduce developing on Hadoop

Outline Hadoop server information Create user for running on the server Setup jdk Get your own Hadoop Get eclipse and the mapreduce plugin Something Important Experiences

Hadoop server information Server ip: 166.111.68.153. Account: last three number of your student number (password the same, ask TA to change it). The account is used to run mapreduce on the server. If you want use it to ssh to the server, please don’t remove or modify any files on the server.

Create user for running on the server To run mapreduce on the Hadoop, you must have the same user on your machine as on the server. Use useradd to create. Then work as the created user.

Setup jdk You can get jdk by: scp user@166.111.68.153:/course_files/ download/jdk-1_5_0_12-linux-i586.bin local_path Install it on your machine and find out where it is.

Get your own Hadoop You can get Hadoop by: scp user@166.111.68.153:/course_files/ download/hadoop.tar.gz local_path Extract the file Run scripts/make_dirs.sh Modify the JAVA_HOME in hadoop/conf/hadoop-env.sh

Get your own Hadoop Now you can access the Hadoop on the server by run command on your own Hadoop. Find files on the HDFS hadoop/bin/hadoop fs –ls /user/root/ Run MapReduce on the server hadoop/bin/hadoop jar hadoop/hadoop-0.13.1-examples.jar wordcount /user/root/lab0 test

Get eclipse and the mapreduce plugin You can get eclipse and the plugin by: scp user@166.111.68.153:/course_files/ download/eclipse-SDK-3.2.2-linux-gtk.tar.gz local_path scp user@166.111.68.153:/course_files/ download/mapreduce_tools.zip local_path

Get eclipse and the mapreduce plugin Extract the eclipse and the plugin Copy the com.ibm.hipods.mapreduce_1.0.4 folder to the eclispe/plugins Run eclipse, if you can see Window->Preferences->Hadoop Home Directory, your plugin run successfully.

Get eclipse and the mapreduce plugin Information of the plugin: http://www.alphaworks.ibm.com/tech/mapreducetools I found I can’t pass arguments to Hadoop when running mapreduce. So I just use the plugin to write code, then make jar package by ant and run it in the term. If you can resolve the problem, please tell me, thanks.

Something Important Don’t ssh to the server.If you have to do this, please don’t remove or modify any files. Access the HDFS or run MapReduce on the server by run command on your own Hadoop.

Experiences When you write MapReduce on Hadoop, you can get help from: http://wiki.apache.org/lucene-hadoop/ http://lucene.apache.org/hadoop/api/index.html hadoop exmaple hadoop api source

That’s all, Enjoy our labs.