Wook-Shin Han, Sangyeon Lee POSTECH, DGIST

Slides:



Advertisements
Similar presentations
1 Copyright © 2012 Oracle and/or its affiliates. All rights reserved. Convergence of HPC, Databases, and Analytics Tirthankar Lahiri Senior Director, Oracle.
Advertisements

Dynamic Data Partitioning for Distributed Graph Databases Xavier Martínez Palau David Domínguez Sal Josep Lluís Larriba Pey.
Piccolo: Building fast distributed programs with partitioned tables Russell Power Jinyang Li New York University.
CS 245Notes 71 CS 245: Database System Principles Notes 7: Query Optimization Hector Garcia-Molina.
Scalable Multi-Cache Simulation Using GPUs Michael Moeng Sangyeun Cho Rami Melhem University of Pittsburgh.
Query Processing and Optimizing on SSDs Flash Group Qingling Cao
Out-of core Streamline Generation Using Flow-guided File Layout Chun-Ming Chen 788 Project 1.
A Scalable, Predictable Join Operator for Highly Concurrent Data Warehouses George Candea (EPFL & Aster Data) Neoklis Polyzotis (UC Santa Cruz) Radek Vingralek.
Distributed Graph Processing Abhishek Verma CS425.
Spark: Cluster Computing with Working Sets
GraphChi: Big Data – small machine
FAWN: A Fast Array of Wimpy Nodes Presented by: Aditi Bose & Hyma Chilukuri.
1 SINA: Scalable Incremental Processing of Continuous Queries in Spatio-temporal Databases Mohamed F. Mokbel, Xiaopeng Xiong, Walid G. Aref Presented by.
Presented by Ozgur D. Sahin. Outline Introduction Neighborhood Functions ANF Algorithm Modifications Experimental Results Data Mining using ANF Conclusions.
Query Execution :Nested-Loop Joins Rohit Deshmukh ID 120 CS-257 Rohit Deshmukh ID 120 CS-257.
1 An Empirical Study on Large-Scale Content-Based Image Retrieval Group Meeting Presented by Wyman
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
Introduction to Database Systems 1 Join Algorithms Query Processing: Lecture 1.
Virtual Memory By: Dinouje Fahih. Definition of Virtual Memory Virtual memory is a concept that, allows a computer and its operating system, to use a.
Memory Management Last Update: July 31, 2014 Memory Management1.
ICOM 6005 – Database Management Systems Design Dr. Manuel Rodríguez-Martínez Electrical and Computer Engineering Department Lecture 14 – Join Processing.
X-Stream: Edge-Centric Graph Processing using Streaming Partitions
GRAPH PROCESSING Hi, I am Mayank and the second presenter for today is Shadi. We will be talking about Graph Processing.
CSE 451: Operating Systems Section 10 Project 3 wrap-up, final exam review.
Architectural Support for Fine-Grained Parallelism on Multi-core Architectures Sanjeev Kumar, Corporate Technology Group, Intel Corporation Christopher.
DANIEL J. ABADI, ADAM MARCUS, SAMUEL R. MADDEN, AND KATE HOLLENBACH THE VLDB JOURNAL. SW-Store: a vertically partitioned DBMS for Semantic Web data.
MonetDB/X100 hyper-pipelining query execution Peter Boncz, Marcin Zukowski, Niels Nes.
Speaker: 吳晋賢 (Chin-Hsien Wu) Embedded Computing and Applications Lab Department of Electronic Engineering National Taiwan University of Science and Technology,
©Silberschatz, Korth and Sudarshan13.1Database System Concepts Chapter 13: Query Processing Overview Measures of Query Cost Selection Operation Sorting.
File System Implementation Chapter 12. File system Organization Application programs Application programs Logical file system Logical file system manages.
Porting Irregular Reductions on Heterogeneous CPU-GPU Configurations Xin Huo, Vignesh T. Ravi, Gagan Agrawal Department of Computer Science and Engineering.
Collective Buffering: Improving Parallel I/O Performance By Bill Nitzberg and Virginia Lo.
Towards a Billion Routing Lookups per Second in Software  Author: Marko Zec, Luigi, Rizzo Miljenko Mikuc  Publisher: SIGCOMM Computer Communication Review,
The Vesta Parallel File System Peter F. Corbett Dror G. Feithlson.
Damian Gordon. HARD DISK (MAIN MEMORY) (SECONDARY MEMORY) 2 CACHE 1.
Autonomic scheduling of tasks from data parallel patterns to CPU/GPU core mixes Published in: High Performance Computing and Simulation (HPCS), 2013 International.
Leonardo Guerreiro Azevedo Geraldo Zimbrão Jano Moreira de Souza Approximate Query Processing in Spatial Databases Using Raster Signatures Federal University.
Memory Management during Run Generation in External Sorting – Larson & Graefe.
Adaptive Multi-Threading for Dynamic Workloads in Embedded Multiprocessors 林鼎原 Department of Electrical Engineering National Cheng Kung University Tainan,
By Teacher Asma Aleisa Year 1433 H.   Goals of memory management  To provide a convenient abstraction for programming.  To allocate scarce memory.
Lecture 24 Query Execution Monday, November 28, 2005.
Variant Indexes. Specialized Indexes? Data warehouses are large databases with data integrated from many independent sources. Queries are often complex.
Advanced Science and Technology Letters Vol.30 (ICCA 2013), pp BPP: Large Graph Storage for Efficient.
Data Structures and Algorithms in Parallel Computing
Page 1 A Platform for Scalable One-pass Analytics using MapReduce Boduo Li, E. Mazur, Y. Diao, A. McGregor, P. Shenoy SIGMOD 2011 IDS Fall Seminar 2011.
ICOM 5016 – Introduction to Database Systems Lecture 13- File Structures Dr. Bienvenido Vélez Electrical and Computer Engineering Department Slides by.
Introduction to Database Systems1 External Sorting Query Processing: Topic 0.
ICOM 6005 – Database Management Systems Design Dr. Manuel Rodríguez-Martínez Electrical and Computer Engineering Department Lecture 7 – Buffer Management.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
CS422 Principles of Database Systems Buffer Management Chengyu Sun California State University, Los Angeles.
DATABASE OPERATORS AND SOLID STATE DRIVES Geetali Tyagi ( ) Mahima Malik ( ) Shrey Gupta ( ) Vedanshi Kataria ( )
Journal of Computational and Applied Mathematics Volume 253, 1 December 2013, Pages 14–25 Reporter : Zong-Dian Lee A hybrid quantum inspired harmony search.
Oracle Announced New In- Memory Database G1 Emre Eftelioglu, Fen Liu [09/27/13] 1 [1]
1 Munther Abualkibash University of Bridgeport, CT.
A Study of Data Partitioning on OpenCL-based FPGAs Zeke Wang (NTU Singapore), Bingsheng He (NTU Singapore), Wei Zhang (HKUST) 1.
Today topics: File System Implementation
Module 11: File Structure
Pagerank and Betweenness centrality on Big Taxi Trajectory Graph
Conception of parallel algorithms
Database Applications (15-415) DBMS Internals- Part VII Lecture 16, October 25, 2016 Mohammad Hammoud.
Parallel Data Laboratory, Carnegie Mellon University
Advanced Science and Technology Letters Vol.30 (ICCA 2013), pp
Introduction to Spark.
On Efficient Graph Substructure Selection
External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.
Operating Systems.
CSE451 Virtual Memory Paging Autumn 2002
Lecture 22: Query Execution
Implementation of a De-blocking Filter and Optimization in PLX
Automatic and Efficient Data Virtualization System on Scientific Datasets Li Weng.
Presentation transcript:

Wook-Shin Han, Sangyeon Lee POSTECH, DGIST TurboGraph: A Fast Parallel Graph Engine Handling Billion-scale Graphs in a Single PC Wook-Shin Han, Sangyeon Lee POSTECH, DGIST

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

about TurboGraph TurboGraph is the first truly parallel graph engine that exploits : 1) full parallelism including multi-core parallelism and FlashSSD IO parallelismand 2) full overlap of CPU processing and I/O processing as much as possible. 3) Specifically, we propose a novel parallel execution model, called pin-and-slide. 4) TurboGraph also provides engine-level operators such as BFS which are implemented under the pin-and-slide model.

problems of PSW from Graphchi 1) In order to start updating vertices/edges in a shard file, their in-edges must be fully loaded in memory. 2) All edges in the shard file whose source and target vertices are in the same execution interval are processed in sequential order, which hinders full parallelism. 3) At each iteration, a significant number of updated edges can be flushed to disk. 4) Even if a query needs to access a small portion of the data graph, it reads the whole graph at the first iteration.

contributions 1) a general and scalable graph engine on a single machine 2) efficient disk and memory structures for representing billion-scale graphs 3) fast and scalable core graph operations which implement the pin-and-slide model. 4) TurboGraph consistently and significantly outperforms the state-of-the-art methods by up to four orders of magnitude!

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

Disk-based Graph Representation

In-memory Data Structures The buffer manager of TurboGraphmaintains a buffer pool, actually, an array offrames, each of which consists of the page-sized sequences of main memory bytes and some meta information such as pin count, reference bit, and dirty bit.

Core Operations PINCOMPUTEUNPIN(PageID pid, list<RID> RIDList, UserObject uo), in order to allow asynchronous I/Os to the FlashSSD.

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

The Pin-and-Slide Model(based on bit) a buffer pool a graph database two types of threads: execution threads callback threads.

The Pin-and-Slide Model(based on bit) the size = the number of the total LA pages −the number of its PL pages maximize the benefit of all items in the 0-1 knapsack.

Handling General Vectors The main idea is to adopt the concept of the block-based nested loop join, which is well-known in the database area. We regard the set of pages pinned in the current buffer as a blockand also assume that a general vector is partitioned into multiplechunkssuch that each chunk fits in memory.

Handling General Vectors---PageRank

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

Targeted Queries neighborhood, induced subgraph, egonet, K-core, and crossedges

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

outlines INTRODUCTION RELATED WORK EFFICIENT GRAPH STORAGE DISK-BASED PARALLEL GRAPH COMPUTATION PROCESSING GRAPH QUERIES EXPERIMENTS CONCLUSION

Through extensive experiments on large, real graphs, including billion-nodegraphs, we showed that TurboGraphoutperforms the state-of-the-art algorithms by up to four orders of magnitude. Overall, we believe we provide comprehensive insight and a substantial framework for future research. thx @hh's