Map reduce use case Giuseppe Andronico INFN Sez. CT & Consorzio COMETA Workshops Grids vs. Clouds Beijing, 18.05.2011

Introduction Classical data intensive and data parallel applications Huge amount of data to be analyzed Analysis can be done: splitting data in a great number of small chunks applying analysis procedure to each chunk independently applying a specific procedure to the set of previous results to obtain the final goal In this case can be used the Map & Reduce method It fits nicely cloud computing

What is MapReduce? Simple data-parallel programming model designed for scalability and fault-tolerance Pioneered by Google Processes 20 petabytes of data per day Popularized by open-source Hadoop project Used at Yahoo!, Facebook, Amazon, …

What is MapReduce used for? At Google: Index construction for Google Search Article clustering for Google News Statistical machine translation At Yahoo!: “Web map” powering Yahoo! Search Spam detection for Yahoo! Mail At Facebook: Data mining Ad optimization Spam detection

What is MapReduce used for? In research: Astronomical image analysis (Washington) Bioinformatics (Maryland) Analyzing Wikipedia conflicts (PARC) Natural language processing (CMU) Particle physics (Nebraska) Ocean climate simulation (Washington)

Example Given: snapshot of the internet To find: The top N pages with the highest number of incoming links Also given: Infinite computing and storage resources Goal: Optimize it and make it simple enough so that anybody knowing a computer language can do this. CS591CLD Cloud Computing Seminar

Map Reduce Automatic parallelization & distribution Fault-tolerant Provides status and monitoring tools Clean abstraction for programmers CS591CLD Cloud Computing Seminar

Programming Model Borrows from functional programming Users implement interface of two functions: map (in_key, in_value) -> (out_key, intermediate_value) list reduce (out_key, intermediate_value list) -> out_value list CS591CLD Cloud Computing Seminar

Map Records from the data source (lines out of files, rows of a database, etc) are fed into the map function as key*value pairs: e.g., (filename, line). map() produces one or more intermediate values along with an output key from the input of the form <key, value> CS591CLD Cloud Computing Seminar

Reduce Reduce After the map phase is over, all the intermediate values for a given output key are combined together into a list reduce() combines those intermediate values into one or more final values for that same output key (in practice, usually only one final value per key) CS591CLD Cloud Computing Seminar

Map Reduce CS591CLD Cloud Computing Seminar

Parallelism map() functions run in parallel, creating different intermediate values from different input data sets reduce() functions also run in parallel, each working on a different output key All values are processed independently Bottleneck: reduce phase can’t start until map phase is completely finished. CS591CLD Cloud Computing Seminar

Example: Count word occurrences map(String input_key, String input_value): // input_key: document name // input_value: document contents for each word w in input_value: EmitIntermediate(w, "1"); reduce(String output_key, Iterator intermediate_values): // output_key: a word // output_values: a list of counts int result = 0; for each v in intermediate_values: result += ParseInt(v); Emit(AsString(result)); CS591CLD Cloud Computing Seminar

Example vs. Actual Source Code Example is written in pseudo-code Actual implementation is in Java, using Hadoop True code is somewhat more involved (defines how the input key/values are divided up and accessed, etc.) CS591CLD Cloud Computing Seminar

Locality Master program divides up tasks based on location of data: tries to have map() tasks on same machine as physical file data, or at least same rack map() task inputs are divided into 64 MB blocks: same size as Google File System chunks CS591CLD Cloud Computing Seminar

Fault Tolerance Master detects worker failures Re-executes completed & in-progress map() tasks Re-executes in-progress reduce() tasks Master notices particular input key/values cause crashes in map(), and skips those values on re-execution. Effect: Can work around bugs in third-party libraries! CS591CLD Cloud Computing Seminar

Optimizations No reduce can start until map is complete: A single slow disk controller can rate-limit the whole process Master redundantly executes “slow-moving” map tasks; uses results of first copy to finish CS591CLD Cloud Computing Seminar

Sort Backup tasks reduce job completion time a lot! Normal No backup tasks 200 processes killed M = 15000 R = 4000 Backup tasks reduce job completion time a lot! System deals well with failures CS591CLD Cloud Computing Seminar

Optimizations “Combiner” functions can run on same machine as a mapper Causes a mini-reduce phase to occur before the real reduce phase, to save bandwidth CS591CLD Cloud Computing Seminar

Some Applications Count of URL access frequency Reverse Web-Link Graph Distributed Grep: Map - Emits a line if it matches the supplied pattern Reduce - Copies the the intermediate data to output Count of URL access frequency Map – Process web log and outputs <URL, 1> Reduce - Emits <URL, total count> Reverse Web-Link Graph Map – process web log and outputs <target, source> Reduce - emits <target, list(source)> CS591CLD Cloud Computing Seminar

MapReduce Conclusions MapReduce has proven to be a useful abstraction Greatly simplifies large-scale computations at Google Functional programming paradigm can be applied to large-scale applications Fun to use: focus on problem, let library deal with messy details CS591CLD Cloud Computing Seminar

What is Hadoop? At Google MapReduce operation are run on a special file system called Google File System (GFS) that is highly optimized for this purpose. GFS is not open source. Doug Cutting and Yahoo! reverse engineered the GFS and called it Hadoop Distributed File System (HDFS). The software framework that supports HDFS, MapReduce and other related entities is called the project Hadoop or simply Hadoop. This is open source and distributed by Apache. 11/22/2018

Resources “MapReduce: Simplified Data Processing on Large Clusters”, Jeffrey Dean and Sanjay Ghemawat GFS, BigTable, Sawzall, Chubby, Protocol Buffers Google Map Reduce Lecture Series HDFS Architecture CS591CLD Cloud Computing Seminar