On the role of Interactivity and Data Placement in Big Data Analytics Srini Parthasarathy OSU

The Data Deluge: Data Data Everywhere 22

600$ to buy a disk drive that can store all of the worlds music 3 [McKinsey Global Institute Special Report, June 11] Data Storage is Cheap

Data does not exist in isolation. 4

Data almost always exists in connection with other data – integral part of the value proposition. 5

6 Social networks Protein InteractionsInternet VLSI networks Data dependencies Neighborhood graphs

7 Big Data Problem: All this data is only useful if we can scalably extract useful knowledge from such complex data

THIS TALK THE ROLE OF DATA PLACEMENT IN BIG DATA SYSTEMS THE ROLE OF VISUALIZATION AND INTERACTION IN BIG DATA ANALYSIS

GLOBAL GRAPHS

What? – System for deploying applications processing complex data Why? – Seeks balance between high productivity and high performance How? – Built on top of PNLs GlobalArrays – Trees (GlobalTrees, GlobalForests) – Relational Arrays (ArrayDB-GA) – Graphs (GlobalGraphs) Data Placement is key to high performance

Importance of Data Placement Locality – Placing related items close to each other so they may be processed together Mitigating Impact of Data Skew – Reducing load imbalance in a parallel setting – Reducing variance in partition samples Generating Stratified Samples – Improving interactive performance

Key Ideas Pivotization – Convert data with complex structure into sets – Each element of set captures features of local topology Hashing into Strata: Hash related sets into similar bins – Can employ a sketch-clustering algorithm Partitioning: Place Strata into partitions for Locality Mitigating Data Skew Samples

SKETCHSORT or SKETCHCLUSTER S-1 : S-4 (Δ1, SK-1) (Δ5, SK-5) (Δ12,SK-12) (Δ25,SK-25) : S-5 : S-128 : PARTITIONING & REPLICATION P-1 : P-2 S-4 S-7 S-8 S-12 : S-128 P-3 : P-8 S-3 S-4 S-9 S-12 : S- 127 PIVOT TRANSFORMATIONS A B C L E A B C L E F Δ1 Δ25 DATA ( Δ ) A B C A F C A E C A F L B E F A E L A B L A B C A E C A E L A B L (PS-1) (PS-25) PIVOT SETS (PS) MINWISE HASHING on PIVOT SETS {1050, 2020, 3130,1800} (SK-1) {1050, 2020, 7225, 2020} (SK-25) SKETCHES(SK) Strata (S)

Frequent Tree Mining Our proposed approaches shows 100X gains

WebGraph Compression Linear Scaleup with no loss in compression ratio

PRISM-HD - PRobing the Intrinsic Structure and Makeup of High-dimensional Data HD

Visualization and Interactivity are key to discovery 17

PRISM-HD What? – A novel mechanism for exploring complex data Why? – User is often overwhelmed with characteristics of data – Befuddled on where to start How? – Given, similarity measure-of-interest – Compute similarity graph at threshold (t) Key: Graphs are dimensionless – Provide user graph visualization cues User determines next threshold and repeats HD

HIGH THRESHOLD MODERATE THRESHOLD LOW THRESHOLD

Benefits of Knowledge Caching HD

Benefits of Incremental Processing on Twitter Incremental estimates on Twitter t 1 = 0.95 HD

PRISM-HD and Global Graphs in Context: Leveraging Social Media in Emergency Response HD

Concluding Remarks Data is everywhere Data is fraught with complexities – Dimensionality, dynamics, structure, massive… Both data placement and data interactivity have an important role to play in big data analytics – PRISM-HD and GlobalGraphs can help! HD

Thanks for your attention Contact: Mining Simulation Data Medical Image Analysis Protein Interaction Network (yeast) Acknowledgements: Various NSF, NIH, DOE and industry grants