1. Lecture 3: Bringing it all together
Except where otherwise noted all portions of this work are Copyright (c) 2007 Google
and are licensed under the Creative Commons Attribution 3.0 License

2. Context and Review
Data dependencies determine whether a problem can be formulated in MapReduce
The properties of fold() and map() determine how to formulate a problem in MapReduce
How do you parallelize fold()? map()?

3. MapReduce Introduction
MapReduce is both a programming model and a clustered computing system
A specific way of formulating a problem, which yields good parallelizability
A system which takes a MapReduce-formulated problem and executes it on a large cluster
Hides implementation details, such as hardware failures, grouping and sorting, scheduling …
Previous lectures have focused on MapReduce- the-problem-formulation
Today will mostly focus on MapReduce-the- system

4. MR Problem Formulation: Formal Definition
MapReduce:
mapreduce fm fr l =
map (reducePerKey fr) (group (map fm l))
reducePerKey fr (k,v_list) =
(k, (foldl (fr k) [] v_list))
Assume map here is actually concatMap.
Argument l is a list of documents
The result of first map is a list of key-value pairs
The function fr takes 3 arguments key, context, current.
With currying, this allows for locking the value of “key” for each list during the fold.
MapReduce maps a fold over the sorted result of a map!

5. MR System Overview (1 of 2)
Map:
Preprocesses a set of files to generate intermediate key-value pairs
As parallelized as you want
Group:
Partitions intermediate key-value pairs by unique key, generating a list of all associated values
Reduce:
For each key, iterates over value list
Performs computation that requires context between iterations
Parallelizable amongst different keys, but not within one key
A common question is: how is map reduce different from cloning the input data N ways, and using N workers to process the data using the original non-MR formulation/

6. MR System Overview (2 of 2)
Shamelessly stolen from Jeff Dean’s OSDI ‘04 presentation

7. Example: MapReduce DocInfo (1 of 2)
mapreduce fm fr l =
map (reducePerKey fr) (group (map fm l))
reducePerKey fr (k,v_list) =
(k, (foldl (fr k) [] v_list)
Pseudocode for fm
fm contents =
concat [
[(“spaces”, (count_spaces contents))],
(map (emit “raw”) (split contents)),
(map (emit “scrub”) (scrub (split contents)))]
emit label value = (label, (value, 1))

8. Example: MapReduce DocInfo (2 of 2)
mapreduce fm fr l =
map (reducePerKey fr) (group (map fm l))
reducePerKey fr (k,v_list) =
(k, (foldl (fr k) [] v_list)
Pseudocode for fr
fr ‘spaces’ count (total:xs) =
(total+count:xs)
fr ‘raw’ (word,count) (result) =
(update_result (word,count) result)
fr ‘scrub’ (word,count) (result) =

9. Group Exercise Formulate the following as map reduces:
Find the set of unique words in a document
Input: a bunch of words
Output: all the unique words (no repeats)
Calculate per-employee taxes
Input: a list of (employee, salary, month) tuples
Output: a list of (employee, taxes due) pairs
Randomly reorder sentences
Input: a bunch of documents
Output: all sentences in random order (may include duplicates)
Compute the minesweeper grid/map
Input: coordinates for the location of mines
Output: coordinate/value pairs for all non-zero cells
Can you think generalized techniques for decomposing problems?

10. MapReduce Parallelization: Execution
Shamelessly stolen from Jeff Dean’s OSDI ‘04 presentation

11. MapReduce Parallelization: Pipelining
Finely granular tasks: many more map tasks than machines
Better dynamic load balancing
Minimizes time for fault recovery
Can pipeline the shuffling/grouping while maps are still running
Example: 2000 machines -> 200,000 map reduce tasks
Shamelessly stolen from Jeff Dean’s OSDI ‘04 presentation

12. Example: MR DocInfo, revisited
Do MapReduce DocInfo in 2 passes (instead of 1), performing all the work in the “group” step
Map1:
Tokenize document
For each token output:
(“raw:<word>”,1)
(“scrubbed:<scrubbed_word>”, 1)
Reduce1:
For each key, ignore value list and output (key,1)
Map2:
For each token “type:value”, output (type,1)
Reduce 2:
For each key, output (key, (sum values))

13. Example: MR DocInfo, revisited
Mapper
Reducer
Mapper
Reducer
Key:
Connections are network links
GFS is a cluster of storage machines
Mapper
GFS
Of the 2 DocInfo MapReduce implementations, which is better?
Define “better”. What resources are you considering?
Dev time? CPU? Network? Disk? Complexity? Reusability?

14. Hadoop takes the generated class files and manages running them
HaDoop-as-MapReduce
mapreduce fm fr l =
map (reducePerKey fr) (group (map fm l))
reducePerKey fr (k,v_list) =
(k, (foldl (fr k) [] v_list)
Hadoop:
The fm and fr are function objects (classes)
Class for fm implements the Mapper interface
Map(WritableComparable key, Writable value,
OutputCollector output, Reporter reporter)
Class for fr implements the Reducer interface
reduce(WritableComparable key, Iterator values,
Hadoop takes the generated class files and manages running them

15. Bonus Materials: MR Runtime
The following slides illustrate an example run of MapReduce on a Google cluster
A sample job from the indexing pipeline, processes ~900 GB of crawled pages

16. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (1 of 9)
From Jeff Dean’s OSDI ‘04 presentation

17. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (2 of 9)
From Jeff Dean’s OSDI ‘04 presentation

18. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (3 of 9)
From Jeff Dean’s OSDI ‘04 presentation

19. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (4 of 9)
From Jeff Dean’s OSDI ‘04 presentation

20. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (5 of 9)
From Jeff Dean’s OSDI ‘04 presentation

21. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (6 of 9)
From Jeff Dean’s OSDI ‘04 presentation

22. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (7 of 9)
From Jeff Dean’s OSDI ‘04 presentation

23. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (8 of 9)
From Jeff Dean’s OSDI ‘04 presentation

24. From Jeff Dean’s OSDI ‘04 presentation
MR Runtime (9 of 9)
From Jeff Dean’s OSDI ‘04 presentation