Cloud Computing Mapreduce (2) Keke Chen

Outline  Hadoop streaming example  Hadoop java API Framework important APIs  Mini-project

A nice book  Hadoop: The definitive Guide You can read it online from campus network - ohiolink  ebook center  safari online

Hadoop streaming  Simple and powerful interface for programming Application developers do not need to learn hadoop java APIs Good for simple, adhoc tasks

Note:  Map/Reduce uses the local linux file system for processing and hosting temporary data  HDFS is used to host application data HDFS Node Local file system

Hadoop streamining  /current/streaming.html /current/streaming.html /usr/local/hadoop/bin/hadoop jar \ /usr/local/hadoop/hadoop-streaming jar \ -input myInputDirs -output myOutputDir \ -mapper myMapper -reducer myReducer Reducer can be empty: -reducer None myMapper and myReducer can be any executable Mapper/reducer will take stdin and output to stdout  Files in myInputDirs are fed into mapper as stdin  Mapper’s output will be the input of reducer

Packaging files with job submission  /usr/local/hadoop/bin/hadoop jar \ /usr/local/hadoop/hadoop-streaming jar \ -input “/user/hadoop/inputdata” \ -output “/user/hadoop/outputdata” \ -mapper “python myPythonScript.py myDictionary.txt” \ -reducer “/bin/wc” \ -file myPythonScript.py \ -file myDictionary.txt -file is good for small files Input parameter for the script

Using hadoop library classes hadoop jar $HADOOP_HOME/hadoop-streaming.jar \ -D mapred.reduce.tasks=12 \ -input myInputDirs \ -output myOutputDir \ -mapper org.apache.hadoop.mapred.lib.IdentityMapper \ -reducer org.apache.hadoop.mapred.lib.IdentityReducer \ -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner

Large files and archives  Upload large files to HDFS first  Use –files option in streaming, which will download files to local working directory -files hdfs://host:fs_port/user/testfile.txt#testlink -archives hdfs://host:fs_port/user/testfile.jar#testlink  Cache1.txt, cache2.txt are in testfile.jar  Then, locally testlink/cache1.txt, textlink/cache2.txt

Wordcount  Problem: counting frequencies of words for a large document collection.  Implement mapper and reducer respectively, using python Some good python tutorials at

Mapper.py import sys for line in sys.stdin: line = line.strip() words = line.split() for word in words: print ‘%s\t1’ % (word)

Reducer.py import sys word2count={} for line in sys.stdin: line = line.strip() word, count = line.split(‘\t’, 1) try: count = int(count) word2count[word] = word2count.get(word, 0)+ count except ValueError: pass for word in word2count: print ‘%s\t%s’% (word, word2count[word])

Running wordcount hadoop jar $HADOOP_HOME/hadoop- streaming.jar \ -mapper "python mapper.py" \ -reducer "python reducer.py" \ -input text -output output2 \ -file /localpath/mapper.py -file /localpath/reducer.py

Running wordcount hadoop jar $HADOOP_HOME/hadoop- streaming.jar \ -mapper "python mapper.py" \ -reducer "python reducer.py" \ -input text -output output2 \ -file mapper.py -file reducer.py \ -jobconf mapred.reduce.tasks=2 \ -jobconf mapred.map.tasks=4

 If mapper/reducer takes files as parameters hadoop jar $HADOOP_HOME/hadoop- streaming.jar \ -mapper "python mapper.py" \ -reducer "python reducer.py myfile" \ -input text -output output2 \ -file /localpath/mapper.py -file /localpath/reducer.py -file /localpath/myfile

Hadoop Java APIs  hadoop.apache.org/common/docs/curre nt/api/  benefits Jave code is more efficient than streaming More parameters for control and tuning Better for iterative MR programs

Important base classes  Mapper Function map(Object, Writable, Context)  Reducer Function reduce(WritableComparable, Iterator, Context)  Combiner  Partitioner

The framework public class Wordcount{ public static class MapClass extends Mapper { public void setup(Mapper.Context context){…} public void map(Object key, Text value, Context context) throws IOException {…} } public static class ReduceClass Reducer { public void setup(Reducer.Context context){…} public void reduce(Text key, Iterator values, Context context) throws IOException{…} } public static void main(String[] args) throws Exception{} }

The wordcount example in java  /current/mapred_tutorial.html#Example %3A+WordCount+v1.0 /current/mapred_tutorial.html#Example %3A+WordCount+v1.0  Old/New framework Old framework for version prior to 0.20

Mapper of wordcount public static class WCMapper extends Mapper { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); }

WordCount Reducer public static class WCReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); }

Function parameters  Define map/reduce parameters according to your application Have to use writable classes in org.apache.hadoop.io  E.g. Text, LongWritable, IntWritable etc. Template parameters and the function parameters should be matched Map’s output and reduce’s input parameters should be matched.

Configuring map/reduce  Passing global parameter settings to each map/reduce process In main function, set parameters in a Configuration object Configuration conf = new Configuration(); Job job = new Job(conf, "cloudvista"); job.setJarByClass(Wordcount.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); job.setMapperClass(WCMapper.class); //job.setCombinerClass(WCReducer.class); job.setReducerClass(WCReducer.class); //job.setPartitionerClass(WCPartitioner.class); job.setNumReduceTasks(num_reduce); FileInputFormat.setInputPaths (job, input); FileOutputFormat.setOutputPath (job, new Path(output_path )); System.exit(job.waitForCompletion(true)?0:1);

How to run your app 1. Compile to jar file 2. Command line hadoop jar your_jar your_parameters Normally you need to pass in  Number of reducers  Input files  Output directory  Any other application specific parameters

Access Files in HDFS? Example: In map function Public void setup(Mapper.Context context){ Configuration conf = context.getConfiguration(); string filename = conf.get(“yourfile"); Path p = new Path(filename); // Path is used for opening the file. FileSystem fs = FileSystem.get(conf);//determines local or HDFS FSInputStream file = fs.open(p); while (file.available() > 0){ … } file.close(); }

Combiner  Apply reduce function to the intermediate results locally after the map generates the result Map1 key1 Key n combineKey1, value1 Key2, value2 … Keyn, valueN reduces Map’s local

Partitioner  If map’s output will generate N keys (N>R, R:# of reduces) By default, N keys are randomly distributed to R reduces You can use partitioner to define how the keys are distributed to the reduces.

Mini project 1 1.Learn to use HDFS 2.Read and run wordcount example 2/mapred_tutorial.html 3.Write a MR program for inverted-index /user/hadoop/prj1.txt Implement two versions  Script/exe + streaming  Hadoop Java API The file has “docID \t docContent” per line Generating inverted index Word \t a list of “DocID:position”