Hadoop Joshua Nester, Garrison Vaughan, Calvin Sauerbier, Jonathan Pingilley, and Adam Albertson

Operating System and Network Configuration  The first thing we did was install Ubuntu 10.4 LTS on every machine.  After all of the nodes were up and running Ubuntu, we connected all of them to the switch

Basic Configuration - Java  After we got all of the machines connected to the switch, we had to install some of the packages we needed for Hadoop.  The Ubuntu installation did not come with Java, so we installed Java on each machine and then configured the PATH variable for each machine so they would be able to discover the Java binary.

Basic Configuration - Hadoop  After getting the Java Development Kit installed, we installed the Hadoop files and setup the PATH variable for HADOOP_HOME.  We then had to create a Hadoop user account and group on each node and change the ownership of the Hadoop files over to that new user.

Basic Configuration - SSH  After setting up the Hadoop accounts on each node, we had to setup the authorized_keys for the master node so it could shell into the Hadoop accounts on the other nodes.

File System Configuration  On each node, we had to configure the XML files that were used for the distributed file system configuration.  After setting up the DFS configuration, we had to format the namenode (master node).  Once all configuration was done, we started the distributed file system and tasktracker scripts and got the datanodes and jobtrackers running on all of the slaves.

Test Run  For our test run, we gave Hadoop seven different books to run against the word counting program provided with the installation.  The first time we ran the test, the cluster successfully mapped all of the work, but failed to reduce.  The problem ended up being caused by an error in the /etc/hosts configuration.

Test Run  When the node running the reducer went to look for the output of its own maps, it would reference its own IP address to communicate with the task tracker it was running.  What we did not realize was that the nodes were referencing themselves using an entry in /etc/hosts that was setup by the Ubuntu installation which pointed to (nodeName-desktop)  We changed the IP of this entry, on each node, to that specific node’s static IP address. This resolved the fetch failure issue we were having with the maps.

Test Run  Once the problem was resolved, our Hadoop cluster successfully counted the occurrence of each word in the input files.