1. Advanced Database Systems: DBS CB, 2nd Edition
Advanced Topics of Interest: “MapReduce and SQL” & “SSD and DB”

2. Outline
MapReduce and SQL
SSD and SQL

3. MapReduce and SQL 3 3 3

4. Introduction It is all about divide and conquer Partition Combine
“Work” w1 w2 w3 r1 r2 r3
“Result”
“worker”
Partition
Combine

5. Introduction Different workers: Parallelization Problems:
Different threads in the same core
Different cores in the same CPU
Different CPUs in a multi-processor system
Different machines in a distributed system
Parallelization Problems:
How do we assign work units to workers?
What if we have more work units than workers?
What if workers need to share partial results?
How do we aggregate partial results?
How do we know all the workers have finished?
What if workers die?

6. Introduction General Themes: Parallelization problems arise from:
Communication between workers
Access to shared resources (e.g., data)
Thus, we need a synchronization system!
This is tricky:
Finding bugs is hard
Solving bugs is even harder

7. Introduction Patterns for Parallelism: Master/Workers
Producer/Consumer Flow
Work Queues C P C P
shared queue

8. Introduction: Evolution
Functional Programming
MapReduce
Google File System (GFS)

9. Introduction Functional Programming:
MapReduce = functional programming meets distributed processing on steroids
Not a new idea… dates back to the 50’s (or even 30’s)
What is functional programming?
Computation as application of functions
Theoretical foundation provided by lambda calculus
How is it different?
Traditional notions of “data” and “instructions” are not applicable
Data flows are implicit in program
Different orders of execution are possible
Exemplified by LISP and ML

10. Introduction: Lisp  MapReduce?
What does this have to do with MapReduce?
After all, Lisp is about processing lists
Two important concepts in functional programming
Map: do something to everything in a list
Fold: combine results of a list in some way

11. Introduction: Map Map is a higher-order function How map works:
Function is applied to every element in a list
Result is a new list f

12. Introduction: Fold Fold is also a higher-order function
How fold works:
Accumulator set to initial value
Function applied to list element and the accumulator
Result stored in the accumulator
Repeated for every item in the list
Result is the final value in the accumulator f
final value
Initial value

13. Lisp  MapReduce Let’s assume a long list of records: imagine if...
We can distribute the execution of map operations to multiple nodes
We have a mechanism for bringing map results back together in the fold operation
That’s MapReduce! (and Hadoop)
Implicit parallelism:
We can parallelize execution of map operations since they are isolated
We can reorder folding if the fold function is commutative and associative

14. Typical Problem Iterate over a large number of records
Map: extract something of interest from each
Shuffle and sort intermediate results
Reduce: aggregate intermediate results
Generate final output
Key idea: provide an abstraction at the point of these two operations

15. MapReduce Programmers specify two functions:
map (k, v) → <k’, v’>*
reduce (k’, v’) → <k’, v’>*
All v’ with the same k’ are reduced together
Usually, programmers also specify:
partition (k’, number of partitions ) → partition for k’
Often a simple hash of the key, e.g. hash(k’) mod n
Allows reduce operations for different keys in parallel

16. It’s just divide and conquer!
Data Store
Initial kv pairs
map
k1, values…
k2, values…
k3, values…
Barrier: aggregate values by keys
reduce
final k1 values
final k2 values
final k3 values

17. Recall these problems? How do we assign work units to workers?
What if we have more work units than workers?
What if workers need to share partial results?
How do we aggregate partial results?
How do we know all the workers have finished?
What if workers die?

18. MapReduce Runtime Handles data distribution
Gets initial data to map workers
Shuffles intermediate key-value pairs to reduce workers
Optimizes for locality whenever possible
Handles scheduling
Assigns workers to map and reduce tasks
Handles faults
Detects worker failures and restarts
Everything happens on top of GFS (later)

19. “Hello World”: Word Count
Map(String input_key, String input_value):
// input_key: document name
// input_value: document contents
for each word w in input_values:
EmitIntermediate(w, "1");
Reduce(String key, Iterator intermediate_values):
// key: a word, same for input and output
// intermediate_values: a list of counts
int result = 0;
for each v in intermediate_values:
result += ParseInt(v);
Emit(AsString(result));

20. Behind the scenes…

21. Bandwidth Optimizations
Take advantage of locality
Move the process to where the data is!
Use “Combiner” functions
Executed on same machine as mapper
Results in a “mini-reduce” right after the map phase
Reduces key-value pairs to save bandwidth
When can you use combiners?

22. Skew Problem Issue: reduce is only as fast as the slowest map
Solution: redundantly execute map operations, use results of first to finish
Addresses hardware problems...
But not issues related to inherent distribution of data
Data, Data, More Data
All of this depends on a storage system for managing all the data…
That’s where GFS (Google File System), and by extension HDFS in Hadoop

23. Assumptions High component failure rates “Modest” number of HUGE files
Inexpensive commodity components fail all the time
“Modest” number of HUGE files
Just a few million (!!!)
Each is 100MB or larger; multi-GB files typical
Files are write-once, mostly appended to
Perhaps concurrently
Large streaming reads
High sustained throughput favoured over low latency

24. GFS Design Decisions Files stored as chunks Fixed size (64MB)
Reliability through replication
Each chunk replicated across 3+ chunk servers
Single master to coordinate access, keep metadata
Simple centralized management
No data caching
Little benefit due to large data sets, streaming reads
Familiar interface, but customize the API
Simplify the problem; focus on Google apps
Add snapshot and record append operations

25. GFS Architecture Single master Multiple chunk servers
Can anyone see a potential weakness in this design?

26. Single master From distributed systems we know this is a
Single point of failure
Scalability bottleneck
GFS solutions:
Shadow masters
Minimize master involvement
Never move data through it, use only for metadata (and cache metadata at clients)
Large chunk size
Master delegates authority to primary replicas in data mutations (chunk leases)
Simple, and good enough!

27. Metadata Global metadata is stored on the master
File and chunk namespaces
Mapping from files to chunks
Locations of each chunk’s replicas
All in memory (64 bytes / chunk)
Fast
Easily accessible
Master has an operation log for persistent logging of critical metadata updates
Persistent on local disk
Replicated
Checkpoints for faster recovery

28. Mutations Mutation = write or append Must be done for all replicas
Goal: minimize master involvement
Lease mechanism:
Master picks one replica as primary; gives it a “lease” for mutations
Primary defines a serial order of mutations
All replicas follow this order
Data flow decoupled from control flow

29. Relaxed Consistency Model
“Consistent” = all replicas have the same value
“Defined” = replica reflects the mutation, consistent
Some properties:
Concurrent writes leave region consistent, but possibly undefined
Failed writes leave the region inconsistent
Some work has moved into the applications:
E.g., self-validating, self-identifying records
Google apps can live with it
What about other apps?

30. Master’s Responsibilities (1/2)
Metadata storage
Namespace management/locking
Periodic communication with chunk servers
Give instructions, collect state, track cluster health
Chunk creation, re-replication, rebalancing
Balance space utilization and access speed
Spread replicas across racks to reduce correlated failures
Re-replicate data if redundancy falls below threshold
Rebalance data to smooth out storage and request load

31. Master’s Responsibilities (2/2)
Garbage Collection
Simpler, more reliable than traditional file delete
Master logs the deletion, renames the file to a hidden name
Lazily garbage collects hidden files
Stale replica deletion
Detect “stale” replicas using chunk version numbers

32. Fault Tolerance High availability Data integrity
Fast recovery: master and chunk servers restartable in a few seconds
Chunk replication: default 3 replicas
Shadow masters
Data integrity
Checksum every 64KB block in each chunk

33. Parallelization Problems
How do we assign work units to workers?
What if we have more work units than workers?
What if workers need to share partial results?
How do we aggregate partial results?
How do we know all the workers have finished?
What if workers die?

34. Managing Dependencies
Remember: Mappers run in isolation
You have no idea in what order the mappers run
You have no idea on what node the mappers run
You have no idea when each mapper finishes
Question: what if your computation is a non-commutative operation on mapper results?
Answer: Cleverly “hide” dependencies in the reduce stage
The reducer can hold state across multiple map operations
Careful choice of partition function
Careful choice of sorting function
Example: computing conditional probabilities

35. Other things to beware of…
Object creation overhead
Reading in external resources is tricky
Possibility of creating hotspots in underlying file system

36. M/R Application: Cost Measures for Algorithms
Communication cost = total I/O of all processes.
Elapsed communication cost = max of I/O along any path.
(Elapsed ) computation costs analogous, but count only running time of processes.

37. M/R Application: Example: Cost Measures
For a map-reduce algorithm:
Communication cost = input file size + 2  (sum of the sizes of all files passed from Map processes to Reduce processes) + the sum of the output sizes of the Reduce processes
Elapsed communication cost is the sum of the largest input + output for any map process, plus the same for any reduce process

38. M/R Application: What Cost Measures Mean
Either the I/O (communication) or processing (computation) cost dominates.
Ignore one or the other.
Total costs tell what you pay in rent from your friendly neighborhood cloud.
Elapsed costs are wall-clock time using parallelism

39. Join By Map-Reduce
Our first example of an algorithm in this framework is a map-reduce example
Compute the natural join R(A,B) ⋈ S(B,C)
R and S each are stored in files
Tuples are pairs (a,b) or (b,c)
Use a hash function h from B-values to 1..k.
A Map process turns input tuple R(a,b) into key-value pair (b,(a,R)) and each input tuple S(b,c) into (b,(c,S))

40. Map-Reduce Join – (2)
Map processes send each key-value pair with key b to Reduce process h(b)
Hadoop does this automatically; just tell it what k is
Each Reduce process matches all the pairs (b,(a,R)) with all (b,(c,S)) and outputs (a,b,c)

41. Cost of Map-Reduce Join
Total communication cost = O(|R|+|S|+|R ⋈ S|)
Elapsed communication cost = O(s)
We’re going to pick k and the number of Map processes so I/O limit s is respected
With proper indexes, computation cost is linear in the input + output size
So computation costs are like comm. costs

42. Three-Way Join
We shall consider a simple join of three relations, the natural join R(A,B) ⋈ S(B,C) ⋈ T(C,D)
One way: cascade of two 2-way joins, each implemented by map-reduce
Fine, unless the 2-way joins produce large intermediate relations

43. Example: 3-Way Join
Reduce processes use hash values of entire S(B,C) tuples as key
Choose a hash function h that maps B- and C-values to k buckets
There are k 2 Reduce processes, one for each (B-bucket, C-bucket) pair

44. Mapping for 3-Way Join
Aside: even normal
map-reduce allows
inputs to map to
several key-value
pairs.
We map each tuple S(b,c) to ((h(b), h(c)), (S, b, c))
We map each R(a,b) tuple to ((h(b), y), (R, a, b)) for all y = 1, 2,…,k
We map each T(c,d) tuple to ((x, h(c)), (T, c, d)) for all x = 1, 2,…,k.
Keys
Values

45. Assigning Tuples to Reducers
h(b) = 0 1 2 3
h(c) =
S(b,c) where
h(b)=1; h(c)=2
R(a,b), where
h(b)=2
T(c,d), where
h(c)=3

46. Job of the Reducers
Each reducer gets, for certain B-values b and C-values c:
All tuples from R with B = b,
All tuples from T with C = c, and
The tuple S(b,c) if it exists
Thus it can create every tuple of the form (a, b, c, d) in the join

47. Semijoin Option
A possible solution: first semijoin – find all the C-values in S(B,C)
Feed these to the Map processes for R(A,B), so they produce only keys (b, y) such that y is in C(S)
Similarly, compute B(S), and have the Map processes for T(C,D) produce only keys (x, c) such that x is in B(S)

48. Semijoin Option – (2)
Problem: while this approach works, it is not a map-reduce process
Rather, it requires three layers of processes:
Map S to B(S), C(S), and S itself (for join)
Map R and B(S) to key-value pairs and do the same for T and C(S)
Reduce (join) the mapped R, S, and T tuples

49. SSD and SQL 49 49 49

50. Problem Definition
SSD has the potential of displacing disk; experiments by Oracle/HP and Teradata indicates the great performance potential
But placing the database on SSD instead of disk today is still a very expensive proposition (storage is ~60% of BI system cost, and Enterprise SSD cost = 10-15X disk  cost of BI system based on SSD = 7-8X BI cost using disk)!
Wanted to experiment:
Is it possible to explore using SSD instead of disk in the DW internals in specific areas where we can get maximum ROI with minimal $$ increase to the overall system?
Assume that SSD will take over in few years, what are the implications on the DW architecture, specially with indices?

51. Areas of Interest Explore multiple SSD vendors and chose Fusion-io
Explore impact of using Fusion-io on the following DW areas:
Overflow handling
Secondary indices
Audit-trail
As a DB secondary cache
Possible SSD connectivity:
Dedicated SSD card per DB server
SSD pool on IB using SRP protocol to be shared among blades within an enclosure

52. Presentation Title
Overflow Handling
SORT> select [last 0] * from lineitem where L_ORDERKEY < 15,000, order by L_ORDERKEY;
Lineitem table is 60 million rows
We run 1,2,4,and 8 streams using SSD or disk for overflow
Measurements indicate 5-8X performance gain over disk-based SQL overflow
Copyright © 2003 HP corporate presentation. All rights reserved.

53. Presentation Title
Indexes
Secondary Index> select [last 0] * x.l_linenumber, y.l_linenumber from lineitem x, lineitem y where x.l_linenumber = y.l_linenumber and
x.l_orderkey = y.l_orderkey and
x.l_orderkey < 30,000,000;
Lineitem table is 60 million rows, primary key <ShipDate:OrderKey:LineNumber>. Secondary index is <OrderKey:LineNumber>
Without reducing the outer table rows in the nested join; it takes forever (60 million * 60 million rows); we reduced it to 30,000,000 rows
We run the above query where the base table and main index are on disk and all secondary indexes are either on SSD or on disk
Measurements indicate few 100% performance gain over indexes on disk
Copyright © 2003 HP corporate presentation. All rights reserved. 53

54. Presentation Title
Audit Trail
Six tables, single partition and 200,000 rows each. Single Audit Trail process for the whole configuration. Each row is 1012 bytes. Each of the 2 blades has 3 tables
Run Audit Trail as both disk- or SSD-based (direct attached, IB/IPC, IB/disk) as follows:
M updaters do update (900 bytes each) each record in a single partition in a single transaction, 50 times – 50 transactions, where M = 1, 2, , 4, 6
Measurements indicates order of magnitude improvement over disk-based audit trail
Copyright © 2003 HP corporate presentation. All rights reserved. 54

55. DW Cache Run TPC-H SF10 benchmark (due to limited SSD)
Presentation Title
DW Cache
Run TPC-H SF10 benchmark (due to limited SSD)
Run both database and index on SSD with TSE cache equal 700 MB and 2 MB – overflow is not relevant with TPC-H
Run both database and index on disk with TSE cache equal 700 MB and 2 MB
Using SSD for DBMS with very small TSE cache / Disk for database with large buffer cache = 7X times
Copyright © 2003 HP corporate presentation. All rights reserved. 55

56. Summary Clearly SSD is a “game changing” technology
Presentation Title
Summary
Clearly SSD is a “game changing” technology
Instead of putting the database on SSD you can leverage SSD in specific subsystems in the DB, i.e., till price comes down
Copyright © 2003 HP corporate presentation. All rights reserved. 56

57. END

58. M/R: Overview of Lisp Lisp ≠ Lost In Silly Parentheses
We’ll focus on particular a dialect: “Scheme”
Lists are primitive data types
Functions written in prefix notation
'( )
'((a 1) (b 2) (c 3))
(+ 1 2)  3
(* 3 4)  12
(sqrt (+ (* 3 3) (* 4 4)))  5
(define x 3)  x
(* x 5)  15

59. M/R: Overview of Lisp
Functions are defined by binding lambda expressions to variables
Syntactic sugar for defining functions
Above expressions is equivalent to:
Once defined, function can be applied:
(define foo (lambda (x y) (sqrt (+ (* x x) (* y y)))))
(define (foo x y) (sqrt (+ (* x x) (* y y))))
(foo 3 4)  5

60. Recursion Simple factorial example
Even iteration is written with recursive calls!
(define (factorial n) (if (= n 1) (* n (factorial (- n 1)))))
(factorial 6)  720
(define (factorial-iter n) (define (aux n top product) (if (= n top) (* n product) (aux (+ n 1) top (* n product))))
(aux 1 n 1))
(factorial-iter 6)  720