MapReduce Design Patterns CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook

Agenda Summarization Patterns Filtering Patterns Data Organization Patterns Joins Patterns Metapatterns I/O Patterns Bloom Filters

SUMMARIZATION PATTERNS Numerical Summarizations, Inverted Index, Counting with Counters

Overview Top-down summarization of large data sets Most straightforward patterns Calculate aggregates over entire data set or groups Build indexes

Numerical Summarizations Group records together by a field or set of fields and calculate a numerical aggregate per group Build histograms or calculate statistics from numerical values

Known Uses Word Count Record Count Min/Max/Count Average/Median/Standard Deviation

Structure

Performance Perform well, especially when combiner is used Need to be concerned about data skew with from the key

Example Discover the first time a StackOverflow user posted, the last time a user posted, and the number of posts in between User ID, Min Date, Max Date, Count

public class MinMaxCountTuple implements Writable { private Date min = new Date(); private Date max = new Date(); private long count = 0; private final static SimpleDateFormat frmt = new SimpleDateFormat( "yyyy-MM-dd'T'HH:mm:ss.SSS"); public Date getMin() { return min; } public void setMin(Date min) { this.min = min; } public Date getMax() { return max; } public void setMax(Date max) { this.max = max; } public long getCount() { return count; } public void setCount(long count) { this.count = count; } public void readFields(DataInput in) { min = new Date(in.readLong()); max = new Date(in.readLong()); count = in.readLong(); } public void write(DataOutput out) { out.writeLong(min.getTime()); out.writeLong(max.getTime()); out.writeLong(count); } public String toString() { return frmt.format(min) + "\t" + frmt.format(max) + "\t" + count; }

public static class MinMaxCountMapper extends Mapper { private Text outUserId = new Text(); private MinMaxCountTuple outTuple = new MinMaxCountTuple(); private final static SimpleDateFormat frmt = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS"); public void map(Object key, Text value, Context context) { Map parsed = xmlToMap(value.toString()); String strDate = parsed.get("CreationDate"); String userId = parsed.get("UserId"); Date creationDate = frmt.parse(strDate); outTuple.setMin(creationDate); outTuple.setMax(creationDate) outTuple.setCount(1); outUserId.set(userId); context.write(outUserId, outTuple); }

public static class MinMaxCountReducer extends Reducer { private MinMaxCountTuple result = new MinMaxCountTuple(); public void reduce(Text key, Iterable values, Context context) { result.setMin(null); result.setMax(null); result.setCount(0); int sum=0; for (MinMaxCountTuple val : values) { if (result.getMin() == null || val.getMin().compareTo(result.getMin()) < 0) { result.setMin(val.getMin()); } if (result.getMax() == null || val.getMax().compareTo(result.getMax()) > 0) { result.setMax(val.getMax()); } sum += val.getCount(); } result.setCount(sum); context.write(key, result); }

public static void main(String[] args) { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: MinMaxCountDriver "); System.exit(2); } Job job = new Job(conf, "Comment Date Min Max Count"); job.setJarByClass(MinMaxCountDriver.class); job.setMapperClass(MinMaxCountMapper.class); job.setCombinerClass(MinMaxCountReducer.class); job.setReducerClass(MinMaxCountReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(MinMaxCountTuple.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }

Inverted Index Generate an index from a data set to enable fast searches or data enrichment Building an index takes time, but can greatly reduce the amount of time to search for something Output can be ingested into key/value store

Structure

Counting with Counters Use MapReduce framework’s counter utility to calculate global sum entirely on the map side, producing no output Small number of counters only!!

Known Uses Count number of records Count a small number of unique field instances Sum fields of data together

Structure

FILTERING PATTERNS Filtering, Bloom Filtering, Top Ten, Distinct

Filtering Discard records that are not of interest Create subsets of your big data sets that you want to further analyze

Known Uses Closer view of the data Tracking a thread of events Distributed grep Data cleansing Simple random sampling

Structure

Bloom Filtering Keep records that are a member of a large predefined set of values Inherent possibility of false positives

Known Uses Removing most of the non-watched values Pre-filtering a data set prior to expensive membership test

Structure

Top Ten Retrieve a relatively small number of top K records based on a ranking scheme Find the outliers or most interesting records

Known Uses Outlier analysis Selecting interesting data Catchy dashboards

Structure

Distinct Remove duplicate entries of your data, either full records or a subset of fields That fourth V nobody talks about that much

Known Uses Deduplicate data Get distinct values Protect from inner join explosion

Structure

DATA ORGANIZATION PATTERNS Structured to Hierarchical, Partitioning, Binning, Total Order Sorting, Shuffling

Structured to Hierarchical Transformed row-based data to a hierarchical format Reformatting RDBMS data to a more conducive structure

Known Uses Pre-joining data Prepare data for HBase or MongoDB

Structure

Partitioning Partition records into smaller data sets Enables faster future query times due to partition pruning

Known Uses Partition pruning by continuous value Partition pruning by category Sharding

Structure

Binning File records into one or more categories – Similar to partitioning, but the implementation is different Can be used to solve similar problems to Partitioning

Known Uses Pruning for follow-on analytics Categorizing data

Structure

Total Order Sorting Sort your data set in parallel Difficult to apply “divide and conquer” technique of MapReduce

Known Uses Sorting

Structure

Shuffling Set of records that you want to completely randomize Instill some anonymity or create some repeatable random sampling

Known Uses Anonymize the order of the data set Repeatable random sampling after shuffled

Structure

JOIN PATTERNS Join Refresher, Reduce-Side Join w/ and w/o Bloom Filter, Replicated Join, Composite Join, Cartesian Product

Join Refresher A join is an operation that combines records from two or more data sets based on a field or set of fields, known as a foreign key Let’s go over the different types of joins before talking about how to do it in MapReduce

A Tale of Two Tables

Inner Join

Left Outer Join

Right Outer Join

Full Outer Join

Antijoin

Cartesian Product

How to implement? Reduce-Side Join w/ and w/o Bloom Filter Replicated Join Composite Join Cartesian Product stands alone

Reduce Side Join Two or more data sets are joined in the reduce phase Covers all join types we have discussed – Exception: Mr. Cartesian All data is sent over the network – If applicable, filter using Bloom filter

Structure

Performance Need to be concerned about data skew 2 PB joined on 2 PB means 4 PB of network traffic

Replicated Join Inner and Left Outer Joins Removes need to shuffle any data to the reduce phase Very useful, but requires one large data set and the remaining data sets to be able to fit into memory of each map task

Structure

Performance Fastest type of join Map-only Limited based on how much data you can safely store inside JVM Need to be concerned about growing data sets Could optionally use a Bloom filter

Composite Join Leverages built-in Hadoop utilities to join the data Requires the data to be already organized and prepared in a specific way Really only useful if you have one large data set that you are using a lot

Data Structure

Structure

Performance Good performance, join operation is done on the map side Requires the data to have the same number of partitions, partitioned in the same way, and each partition must be sorted

Cartesian Product Pair up and compare every single record with every other record in a data set Allows relationships between many different data sets to be uncovered at a fine-grain level

Known Uses Document or image comparisons Math stuff or something

Structure

Performance Massive data explosion! Can use many map slots for a long time Effectively creates a data set size O(n 2 ) – Need to make sure your cluster can fit what you are doing

METAPATTERNS Job Chaining, Chain Folding, Job Merging

Job Chaining One job is often not enough Need a combination of patterns discussed to do your workflow Sequential vs Parallel

Methodologies In the Driver In a Bash run script With the JobControl utility

Chain Folding Each record can be submitted to multiple mappers, then a reducer, then a mapper Reduces amount of data movement in the pipeline

Structure

Methodologies Just do it ChainMapper/ChainReducer

Job Merging Merge unrelated jobs together into the same pipeline

Structure

Methodologies Tag map output records Use MultipleOutputs

I/O PATTERNS Generating Data, External Source Output, External Source Input, Partition Pruning

Customizing I/O Unstructured and semi-structured data often calls for a custom input format to be developed

Generating Data Generate lots of data in parallel from nothing Random or representative big data sets for you to test your analytics with

Known Uses Benchmarking your new cluster Making more data to represent a sample you were given

Structure

External Source Output You want to write MapReduce output to some non-native location Direct loading into a system instead of using HDFS as a staging area

Known Uses Write directly out to some non-HDFS solution – Key/Value Store – RDBMS – In-Memory Store Many of these are already written

Structure

External Source Input You want to load data in parallel from some other source Hook other systems into the MapReduce framework

Known Uses Skip the staging area and load directly into MapReduce Key/Value store RDBMS In-Memory store

Structure

Partition Pruning Abstract away how the data is stored to load what data is needed based on the query

Known Uses Discard unneeded files based on the query Abstract data storage from query, allowing for powerful middleware to be built

Structure

References “MapReduce Design Patterns” – O’Reilly erns erns