1. COMP9313: Big Data Management Lecturer: Xin Cao Course web site: http://www.cse.unsw.edu.au/~cs9313/

2. Chapter 12: Revision and Exam Preparation

3. Learning outcomes After completing this course, you are expected to:
elaborate the important characteristics of Big Data (Chapter 1)
develop an appropriate storage structure for a Big Data repository (Chapter 5)
utilize the map/reduce paradigm and the Spark platform to manipulate Big Data (Chapters 2, 3, 4, and 7)
use a high-level query language to manipulate Big Data (Chapter 6)
develop efficient solutions for analytical problems involving Big Data (Chapters 8-11)

4. Final exam Final written exam (100 pts)
Six questions in total on six topics
Three hours
You can bring the printed lecture notes
If you are ill on the day of the exam, do not attend the exam – I will not accept any medical special consideration claims from people who already attempted the exam.

5. Topic 1： MapReduce (Chapters 2 and 3)

6. Data Structures in MapReduce
Key-value pairs are the basic data structure in MapReduce
Keys and values can be: integers, float, strings, raw bytes
They can also be arbitrary data structures
The design of MapReduce algorithms involves:
Imposing the key-value structure on arbitrary datasets
E.g.: for a collection of Web pages, input keys may be URLs and values may be the HTML content
In some algorithms, input keys are not used, in others they uniquely identify a record
Keys can be combined in complex ways to design various algorithms

7. Map and Reduce Functions
Programmers specify two functions:
map (k1, v1) → list [<k2, v2>]
Map transforms the input into key-value pairs to process
reduce (k2, [v2]) → [<k3, v3>]
Reduce aggregates the list of values for each key
All values with the same key are sent to the same reducer
Optionally, also:
combine (k2, [v2]) → [<k3, v3>]
Mini-reducers that run in memory after the map phase
Used as an optimization to reduce network traffic
partition (k2, number of partitions) → partition for k2
Often a simple hash of the key, e.g., hash(k2) mod n
Divides up key space for parallel reduce operations
The execution framework handles everything else…

8. Combiners
Often a Map task will produce many pairs of the form (k,v1), (k,v2), … for the same key k
E.g., popular words in the word count example
Combiners are a general mechanism to reduce the amount of intermediate data, thus saving network time
They could be thought of as “mini-reducers”
Warning!
The use of combiners must be thought carefully
Optional in Hadoop: the correctness of the algorithm cannot depend on computation (or even execution) of the combiners
A combiner operates on each map output key. It must have the same output key-value types as the Mapper class.
A combiner can produce summary information from a large dataset because it replaces the original Map output
Works only if reduce function is commutative and associative
In general, reducer and combiner are not interchangeable

9. Partitioner
Partitioner controls the partitioning of the keys of the intermediate map-outputs.
The key (or a subset of the key) is used to derive the partition, typically by a hash function.
The total number of partitions is the same as the number of reduce tasks for the job.
This controls which of the m reduce tasks the intermediate key (and hence the record) is sent to for reduction.
System uses HashPartitioner by default:
hash(key) mod R
Sometimes useful to override the hash function:
E.g., hash(hostname(URL)) mod R ensures URLs from a host end up in the same output file
Job sets Partitioner implementation (in Main)

10. A Brief View of MapReduce

11. MapReduce Data Flow

12. MapReduce Data Flow

13. Design Patter 1: Local Aggregation
Programming Control:
In mapper combining provides control over
when local aggregation occurs
how it exactly takes place
Hadoop makes no guarantees on how many times the combiner is applied, or that it is even applied at all.
More efficient:
The mappers will generate only those key-value pairs that need to be shuffled across the network to the reducers
There is no additional overhead due to the materialization of key-value pairs
Combiners don't actually reduce the number of key-value pairs that are emitted by the mappers in the first place
Scalability issue (not suitable for huge data) :
More memory required for a mapper to store intermediate results

14. Design Patter 2: Pairs vs Strips
The pairs approach
Keep track of each team co-occurrence separately
Generates a large number of key-value pairs (also intermediate)
The benefit from combiners is limited, as it is less likely for a mapper to process multiple occurrences of a word
The stripe approach
Keep track of all terms that co-occur with the same term
Generates fewer and shorted intermediate keys
The framework has less sorting to do
Greatly benefits from combiners, as the key space is the vocabulary
More efficient, but may suffer from memory problem
These two design patterns are broadly useful and frequently observed in a variety of applications
Text processing, data mining, and bioinformatics

15. Design Pattern 3: Order Inversion
Common design pattern
Computing relative frequencies requires marginal counts
But marginal cannot be computed until you see all counts
Buffering is a bad idea!
Trick: getting the marginal counts to arrive at the reducer before the joint counts
Caution:
You need to guarantee that all key-value pairs relevant to the same term are sent to the same reducer

16. Design Pattern 4: Value-to-key Conversion
Put the value as part of the key, make Hadoop do sorting for us
Provides a scalable solution for secondary sorting.
Caution:
You need to guarantee that all key-value pairs relevant to the same term are sent to the same reducer

17. Topic 2： Spark (Chapter 7)

18. Data Sharing in MapReduce
Slow due to replication, serialization, and disk IO
Complex apps, streaming, and interactive queries all need one thing that MapReduce lacks:
Efficient primitives for data sharing

19. Data Sharing in Spark Using RDD
10-100× faster than network and disk

20. What is RDD
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei Zaharia, et al. NSDI’12
RDD is a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.
Resilient
Fault-tolerant, is able to recompute missing or damaged partitions due to node failures.
Distributed 
Data residing on multiple nodes in a cluster.
Dataset 
A collection of partitioned elements, e.g. tuples or other objects (that represent records of the data you work with).
RDD is the primary data abstraction in Apache Spark and the core of Spark. It enables operations on collection of elements in parallel.

21. RDD Operations Transformation: returns a new RDD.
Nothing gets evaluated when you call a Transformation function, it just takes an RDD and return a new RDD.
Transformation functions include map, filter, flatMap, groupByKey, reduceByKey, aggregateByKey, filter, join, etc.
Action: evaluates and returns a new value.
When an Action function is called on a RDD object, all the data processing queries are computed at that time and the result value is returned.
Action operations include reduce, collect, count, first, take, countByKey, foreach, saveAsTextFile, etc.

22. Working with RDDs Create an RDD from a data source
by parallelizing existing Python collections (lists)
by transforming an existing RDDs
from files in HDFS or any other storage system
Apply transformations to an RDD: e.g., map, filter
Apply actions to an RDD: e.g., collect, count
Users can control two other aspects:
Persistence
Partitioning

23. RDD Operations

24. More Examples on Pair RDD
Create a pair RDD from existing RDDs
Output?
reduceByKey() function: reduce key-value pairs by key using give func
mapValues() function: work on values only
groupByKey() function: When called on a dataset of (K, V) pairs, returns a dataset of (K, Iterable<V>) pairs
val pairs = sc.parallelize( List( (“This”, 2), (“is”, 3), (“Spark”, 5), (“is”, 3) ) )
pairs.collect().foreach(println)
val pair1 = pairs.reduceByKey((x,y) => x + y)
pairs1.collect().foreach(println)
val pair2 = pairs.mapValues( x => x -1 )
pairs2.collect().foreach(println)
pairs.groupByKey().collect().foreach(println)

25. Topic 3： Link Analysis (Chapter 8)

26. PageRank: The “Flow” Model
A “vote” from an important page is worth more
A page is important if it is pointed to by other important pages
Define a “rank” rj for page j y m a
a/2
y/2
“Flow” equations:
ry = ry /2 + ra /2
ra = ry /2 + rm
rm = ra /2

27. Google’s Solution: Random Teleports
di … out-degree of node i

28. The Google Matrix

29. Random Teleports ( = 0.8) y a m M [1/N]NxN 1/2 1/2 0 1/2 0 0 0 1/2 1
7/15 y
1/2 1/2 0 1/
0 1/2 1
1/3 1/3 1/3
0.8
+ 0.2
7/15
1/15
7/15
1/15
y 7/15 7/15 1/15
a 7/15 1/15 1/15
m 1/15 7/15 13/15
13/15 a
7/15 m
1/15
1/15 A y
a = m
1/3
0.33
0.20
0.46
0.24
0.20
0.52
0.26
0.18
0.56
7/33
5/33
21/33
. . .

30. Topic-Specific PageRank
Random walker has a small probability of teleporting at any step
Teleport can go to:
Standard PageRank: Any page with equal probability
To avoid dead-end and spider-trap problems
Topic Specific PageRank: A topic-specific set of “relevant” pages (teleport set)
Idea: Bias the random walk
When walker teleports, she pick a page from a set S
S contains only pages that are relevant to the topic
E.g., Open Directory (DMOZ) pages for a given topic/query
For each teleport set S, we get a different vector rS

31. Matrix Formulation

32. Hubs and Authorities j1 j2 j3 j4 i i j1 j2 j3 j4

33. Hubs and Authorities

34. Hubs and Authorities
Convergence criterion:
Repeated matrix powering

35. Example of HITS 1 1 1 A = 1 0 1 0 1 0 1 1 0 AT = 1 0 1 . . . h(yahoo)
M’soft
Amazon
. . .
h(yahoo)
h(amazon)
h(m’soft) =
.58
.80
.53
.27
.80
.53
.27
.79
.57
.23
.788
.577
.211
a(yahoo) =
a(amazon) =
a(m’soft) =
.58
.62
.49
. . .
.62
.49
.628
.459

36. Topic 4： Graph Data Processing

37. From Intuition to Algorithm
Data representation:
Key: node n
Value: d (distance from start), adjacency list (list of nodes reachable from n)
Initialization: for all nodes except for start node, d = 
Mapper:
m  adjacency list: emit (m, d + 1)
Sort/Shuffle
Groups distances by reachable nodes
Reducer:
Selects minimum distance path for each reachable node
Additional bookkeeping needed to keep track of actual path

38. Multiple Iterations Needed
Each MapReduce iteration advances the “known frontier” by one hop
Subsequent iterations include more and more reachable nodes as frontier expands
The input of Mapper is the output of Reducer in the previous iteration
Multiple iterations are needed to explore entire graph
Preserving graph structure:
Problem: Where did the adjacency list go?
Solution: mapper emits (n, adjacency list) as well

39. BFS Pseudo-Code Equal Edge Weights (how to deal with weighted edges?)
Only distances, no paths stored (how to obtain paths?)
class Mapper     method Map(nid n, node N)     d ← N.Distance     Emit(nid n,N)                                  //Pass along graph structure     for all nodeid m ∈ N.AdjacencyList do         Emit(nid m, d+w)                        //Emit distances to reachable nodes
class Reducer     method Reduce(nid m, [d1, d2, . . .])     dmin←∞     M ← ∅     for all d ∈ counts [d1, d2, . . .] do         if IsNode(d) then             M ← d            //Recover graph structure         else if d < dmin then                  //Look for shorter distance             dmin ← d     M.Distance ← dmin                        //Update shortest distance     Emit(nid m, node M)

40. Stopping Criterion
How many iterations are needed in parallel BFS (equal edge weight case)?
Convince yourself: when a node is first “discovered”, we’ve found the shortest path
Now answer the question...
The diameter of the graph, or the greatest distance between any pair of nodes
Six degrees of separation?
If this is indeed true, then parallel breadth-first search on the global social network would take at most six MapReduce iterations.

41. BFS Pseudo-Code (Weighted Edges)
The adjacency lists, which were previously lists of node ids, must now encode the edge distances as well
Positive weights!
In line 6 of the mapper code, instead of emitting d + 1 as the value, we must now emit d + w, where w is the edge distance
The termination behaviour is very different!
How many iterations are needed in parallel BFS (positive edge weight case)?
Convince yourself: when a node is first “discovered”, we’ve found the shortest path
Not true!

42. Additional Complexities
Assume that p is the current processed node
In the current iteration, we just “discovered” node r for the very first time.
We've already discovered the shortest distance to node p, and that the shortest distance to r so far goes through p
Is s->p->r the shortest path from s to r?
The shortest path from source s to node r may go outside the current search frontier
It is possible that p->q->r is shorter than p->r!
We will not find the shortest distance to r until the search frontier expands to cover q.
search frontier r s q p

43. Computing PageRank Properties of PageRank Can be computed iteratively
Effects at each iteration are local
Sketch of algorithm:
Start with seed ri values
Each page distributes ri “credit” to all pages it links to
Each target page tj adds up “credit” from multiple in-bound links to compute rj
Iterate until values converge

44. Simplified PageRank First, tackle the simple case: No teleport
No dangling nodes (dead ends)
Then, factor in these complexities…
How to deal with the teleport probability?
How to deal with dangling nodes?

45. Sample PageRank Iteration (1)

46. Sample PageRank Iteration (2)

47. PageRank Pseudo-Code

48. Complete PageRank Two additional complexities
What is the proper treatment of dangling nodes?
How do we factor in the random jump factor?
Solution:
If a node’s adjacency list is empty, distribute its value to all nodes evenly.
In mapper, for such a node i, emit (nid m, ri/N) for each node m in the graph
Add the teleport value
In reducer, M.PageRank =  * s + (1- ) / N

49. Topic 5： Streaming Data Processing
Types of queries one wants on answer on a data stream: (we’ll learn these today)
Sampling data from a stream
Construct a random sample
Queries over sliding windows
Number of items of type x in the last k elements of the stream
Filtering a data stream
Select elements with property x from the stream
Counting distinct elements (Not required)
Number of distinct elements in the last k elements of the stream

50. Topic 6： Machine Learning
Recommender systems
User-user collaborative filtering
Item-item collaborative filtering
SVM
Given you the training dataset, find out the optimal solution

51. CATEI and myExperience Survey

52. End of Chapter 12