Presentation is loading. Please wait.

Presentation is loading. Please wait.

Query-Friendly Compression of Graph Streams

Published byΟνησίφορος Ζέρβας Modified over 6 years ago

Similar presentations

Presentation on theme: "Query-Friendly Compression of Graph Streams"— Presentation transcript:

1 Query-Friendly Compression of Graph Streams
Arijit Khan Charu C. Aggarwal Nanyang Technical University Singapore IBM T. J. Watson Research Lab NY, USA

2 Graph Streams Graph Stream: Continuous stream of graph edges  Telephone network, communication network, social media data, IP traffic e1 e2 e4 e3 …. e2 e1 e5 e9 e5 e11 e6 e8 e11 e9 e10 e15 e13 Edge Stream e12 e16 Graph Structure With Edge Frequency 1/23 A. Khan, C. Aggarwal

3 Graph Streams Graph Stream: Continuous stream of graph edges  Telephone network, communication network, social media data, IP traffic Massive volume and high speed Construct summary to support future queries e1 e2 e4 e3 …. e2 e1 e5 e9 e5 e11 e6 e8 e11 e9 e10 e15 e13 Edge Stream e12 e16 Graph Structure With Edge Frequency 1/23

4 Challenges in Data Streams Querying
Trade-off among Space, Accuracy, and Efficiency: -- Increasing space increases accuracy, but reduces throughput Other requirements: -- Build summary in one pass over the stream -- Incremental updates in summary 2/23 A. Khan, C. Aggarwal

5 Additional Challenges in Graph Streams Querying: Query Expressibility
Compute reachability formed by heavy-hitter edges e1 e2 e4 e3 …. e2 e1 e5 e9 e5 e11 e6 e8 e11 e9 e10 e15 e13 Edge Stream e12 e16 Graph Data: Red Edges are heavy-hitter edges 3/23 A. Khan, C. Aggarwal

6 Additional Challenges in Graph Streams Querying: Query Expressibility
Compute reachability formed by heavy-hitter edges e1 V1 e2 e4 e3 …. e2 e1 e5 e9 e5 e11 e6 e8 e11 e9 e10 e15 e13 Edge Stream V2 e12 e16 Graph Data: Red Edges are heavy-hitter edges 3/23 A. Khan, C. Aggarwal

7 Additional Challenges in Graph Streams Querying: Query Expressibility
Compute reachability formed by heavy-hitter edges e1 V1 e2 e4 e3 …. e2 e1 e5 e9 e5 e11 e6 e8 e11 e9 e10 e15 e13 Edge Stream V2 e12 e16 Graph Data: Red Edges are heavy-hitter edges Need to preserve connectivity information of the edges in the graph data 3/23

8 Related Work Graph Summarization:
- Query Preserving Graph Compression (SIGMOD 2012) - Graph Summarization with Bounded Error (SIGMOD 2008) - Representing Web Graphs (ICDE 2003) - The Transitive Reduction of a Directed Graph (SIGCOMP 1972) Data Stream Summarization: - Sketches (SIGMOD 2002, VLDB 2002, SIGMOD 2004) - Histograms (SIGMOD 1996, VLDB 1998) - Wavelets (SIAM Rev. 1996) - Space Saving (ICDT 2005) Graph Streams Querying: - gSketches (VLDB 2012) - Analyzing Graph Structure via Linear Measurements (SODA 2012) - Graph Sketches: Sparsification, Spanners, and Subgraphs (PODS 2012) - TCM Sketch (SIGMOD 2016) 4/23

9 Related Work Graph Summarization: Not for stream setting
- Query Preserving Graph Compression (SIGMOD 2012) - Graph Summarization with Bounded Error (SIGMOD 2008) - Representing Web Graphs (ICDE 2003) - The Transitive Reduction of a Directed Graph (SIGCOMP 1972) Data Stream Summarization: - Sketches (SIGMOD 2002, VLDB 2002, SIGMOD 2004) - Histograms (SIGMOD 1996) - Wavelets (SIAM Rev. 1996) - Space Saving (ICDT 2005) Graph Streams Querying: - gSketches (VLDB 2012) - Analyzing Graph Structure via Linear Measurements (SODA 2012) - Graph Sketches: Sparsification, Spanners, and Subgraphs (PODS 2012) - TCM Sketch (SIGMOD 2016) Not for stream setting Does not preserve graph structural information Cannot answer a combination of frequency and structure-based queries, e.g., find all connected components defined by heavy-hitter edges 5/23

10 Related Work TCM Sketch (SIGMOD 2016):
- Does not provide theoretical error bounds - Difficult to answer reachability over heavy-hitter edges 6/23 A. Khan, C. Aggarwal

11 h H1(e) ( e, f ) w Hw(e) Count-Min Sketch + f + f + f
“h” much smaller than total no of edges Estimate frequency of an edge, find heavy-hitter edges Cannot answer structural queries: are these two nodes connected by only high-frequency edges? 7/23

12 Our Solution: GMatrix Synopsis
incoming edge: e = (i,j) H4(.) H3(.) w “h” much smaller than total no of nodes H2(.) H1(.) h k-th Hash Function hashes into ( Hk(i), Hk(j)) h (H1(i), H1(j)) 8/23 A. Khan, C. Aggarwal

13 GMatrix Compression Contract nodes into a total of h super-nodes
Different hash functions create different contractions ⇒ Holds key to effective query processing A graph with 108 nodes, 1010 edges ⇒ Storage 40 GB GMatrix with h = 103 and w = 10 ⇒ Storage 40 MB 9/23 A. Khan, C. Aggarwal

14 Choice of Hash Functions
Pair-wise independent, e.g., modular hash function P is a prime number larger than any node id: (1, 2, … , n) a, b chosen uniformly from (1, P-1) 10/23 A. Khan, C. Aggarwal

15 Reverse Hash Mapping 7x mod 9 = 1 x= 4 7*4 = 3*9 + 1
Reverse hash mapping ⇒ small size and computed efficiently Modular hash function: reverse hash mapping size ⌊P/h⌋ Can be computed in time O(⌊P/h⌋ log P) using extended Euclidean algorithm 11/23 A. Khan, C. Aggarwal

16 Other Synopsis Options with Same Functionality as GMatrix
H1(ij) ( ij, f ) + f w Hw(ij) + f Reverse hash mapping computes w . n2/h2 intersections In GMatrix, reverse hash mapping computes 2. w . n/h intersections 12/23 A. Khan, C. Aggarwal

17 Queries supported by GMatrix (not a comprehensive list)
Edge Frequency Query Heavy-hitter Edge Query Node Frequency Query Sub-graph Edge Frequency Query Heavy-hitter Node Query Reachability Query over High-frequency Edges 13/23 A. Khan, C. Aggarwal

18 Queries supported by GMatrix (not a comprehensive list)
Edge Frequency Query Heavy-hitter Edge Query Node Frequency Query Sub-graph Edge Frequency Query Heavy-hitter Node Query Reachability Query over High-frequency Edges • Last four queries combine graph structure with edge frequency • Possible to define analogous graph mining algorithms, e.g., frequent sub-graphs mining 13/23

19 Edge-Frequency Estimation Query
For edge (i, j), compute the frequencies of w different cells: (Hk(i), Hk(j), k) The minimum of these values is returned as the estimated frequency Estimation is good for high-frequency edges If true frequency is significant fraction of total stream size, then relative error is small 14/23 A. Khan, C. Aggarwal

20 Heavy-Hitter Edge Query
Find all edges with frequency greater than F No false negative, but false positive Find all hash-edges with frequency at least F Reverse hash mapping to find real edges Intersection of edge sets 15/23 A. Khan, C. Aggarwal

21 Heavy-Hitter Edge Query: Optimization
First Optimization If a node does not appear as the source node of some potential frequent edge in at least one of the w hash functions, that node and its outgoing edges can be safely eliminated. Second Optimization 16/23 A. Khan, C. Aggarwal

22 Heavy-Hitter Edge Query: Time Complexity
17/23 A. Khan, C. Aggarwal

23 Reachability Query Find if two query nodes are connected by a path with edges having frequency at least F Determine all edges for which frequency is at least F using heavy-hitter edge query Answer reachability query with these edges 18/23 A. Khan, C. Aggarwal

24 Friendster Stream (Zipf Frequency Distribution with Varying Skew)
Experimental Results #Nodes #Edges Agg. Edge Freq. Max. Edge Freq. Flat Stream Size Compressed Stream Size 66M 3612M 1010 4.43 × 108 80GB 16.47 GB 2.37 GB 1.81 × 109 250 MB 3.22 × 109 Skew 1.0 Skew 1.2 Skew 1.4 Friendster Stream (Zipf Frequency Distribution with Varying Skew) GMatrix Size 40MB (h=1000, w=10) GMatrix Update Time 10-6 sec Experiments were performed on a single core of 10GB, 2.4GHz Xeon server 19/23 A. Khan, C. Aggarwal

25 Edge Frequency Estimation Query
Query over top-500 frequent edges 20/23 A. Khan, C. Aggarwal

26 Heavy Hitter Edge Query
Frequency Threshold = 0.01% of Total Stream Size Frequency Threshold (% of Total Stream Size) GMatrix Count-Min Sketch 1 28 sec 1 sec 0.1 149 sec 2 sec 0.01 771 sec 7 sec Query Answering Time 21/23

27 Reachability Query Frequency Threshold = 0.01% of Total Stream Size
Skew (ZipF) Reachability Error 1.0 0.012 1.2 0.008 1.4 0.004 Frequency Threshold = 0.01% of Total Stream Size Each reachability query can be processed in 0.1 sec 22/23 A. Khan, C. Aggarwal

28 Conclusions GMatrix synopsis for summarizing rapid graph streams
Can be leveraged for a variety of frequency and structural queries Future Work: Improving accuracy by hashing high- and low-frequency edges separately? 23/23 A. Khan, C. Aggarwal

Download ppt "Query-Friendly Compression of Graph Streams"

Similar presentations

Online Mining of Frequent Query Trees over XML Data Streams Hua-Fu Li*, Man-Kwan Shan and Suh-Yin Lee Department of Computer Science.

Online Mining of Frequent Query Trees over XML Data Streams Hua-Fu Li*, Man-Kwan Shan and Suh-Yin Lee Department of Computer Science.
ADAPTIVE FASTEST PATH COMPUTATION ON A ROAD NETWORK: A TRAFFIC MINING APPROACH Hector Gonzalez, Jiawei Han, Xiaolei Li, Margaret Myslinska, John Paul Sondag.

ADAPTIVE FASTEST PATH COMPUTATION ON A ROAD NETWORK: A TRAFFIC MINING APPROACH Hector Gonzalez, Jiawei Han, Xiaolei Li, Margaret Myslinska, John Paul Sondag.
Reachability Querying: An Independent Permutation Labeling Approach (published in VLDB 2014) Presenter: WEI, Hao.

Reachability Querying: An Independent Permutation Labeling Approach (published in VLDB 2014) Presenter: WEI, Hao.
Size-estimation framework with applications to transitive closure and reachability Presented by Maxim Kalaev Edith Cohen AT&T Bell Labs 1996.

Size-estimation framework with applications to transitive closure and reachability Presented by Maxim Kalaev Edith Cohen AT&T Bell Labs 1996.
Evaluating “find a path” reachability queries P. Bouros 1, T. Dalamagas 2, S.Skiadopoulos 3, T. Sellis 1,2 1 National Technical University of Athens 2.

Evaluating “find a path” reachability queries P. Bouros 1, T. Dalamagas 2, S.Skiadopoulos 3, T. Sellis 1,2 1 National Technical University of Athens 2.
Probabilistic Skyline Operator over Sliding Windows Wenjie Zhang University of New South Wales & NICTA, Australia Joint work: Xuemin Lin, Ying Zhang, Wei.

Probabilistic Skyline Operator over Sliding Windows Wenjie Zhang University of New South Wales & NICTA, Australia Joint work: Xuemin Lin, Ying Zhang, Wei.
Fast Algorithms For Hierarchical Range Histogram Constructions

Fast Algorithms For Hierarchical Range Histogram Constructions
Summarizing Distributed Data Ke Yi HKUST += ?. Small summaries for BIG data  Allow approximate computation with guarantees and small space – save space,

Summarizing Distributed Data Ke Yi HKUST += ?. Small summaries for BIG data  Allow approximate computation with guarantees and small space – save space,
Towards Estimating the Number of Distinct Value Combinations for a Set of Attributes Xiaohui Yu 1, Calisto Zuzarte 2, Ken Sevcik 1 1 University of Toronto.

Towards Estimating the Number of Distinct Value Combinations for a Set of Attributes Xiaohui Yu 1, Calisto Zuzarte 2, Ken Sevcik 1 1 University of Toronto.
A Fast and Compact Method for Unveiling Significant Patterns in High-Speed Networks Tian Bu 1, Jin Cao 1, Aiyou Chen 1, Patrick P. C. Lee 2 Bell Labs,

A Fast and Compact Method for Unveiling Significant Patterns in High-Speed Networks Tian Bu 1, Jin Cao 1, Aiyou Chen 1, Patrick P. C. Lee 2 Bell Labs,
Streaming Algorithms for Robust, Real- Time Detection of DDoS Attacks S. Ganguly, M. Garofalakis, R. Rastogi, K. Sabnani Krishan Sabnani Bell Labs Research.

Streaming Algorithms for Robust, Real- Time Detection of DDoS Attacks S. Ganguly, M. Garofalakis, R. Rastogi, K. Sabnani Krishan Sabnani Bell Labs Research.
1 Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams Robert Schweller Ashish Gupta Elliot Parsons Yan Chen Computer.

1 Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams Robert Schweller Ashish Gupta Elliot Parsons Yan Chen Computer.
Graph-Based Synopses for Relational Selectivity Estimation Joshua Spiegel and Neoklis Polyzotis University of California, Santa Cruz.

Graph-Based Synopses for Relational Selectivity Estimation Joshua Spiegel and Neoklis Polyzotis University of California, Santa Cruz.
Ph.D. DefenceUniversity of Alberta1 Approximation Algorithms for Frequency Related Query Processing on Streaming Data Presented by Fan Deng Supervisor:

Ph.D. DefenceUniversity of Alberta1 Approximation Algorithms for Frequency Related Query Processing on Streaming Data Presented by Fan Deng Supervisor:
CS728-2008 Lecture 9 Storeing and Querying Large Web Graphs.

CS Lecture 9 Storeing and Querying Large Web Graphs.
Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications Robert Schweller 1, Zhichun Li 1, Yan Chen 1, Yan Gao 1, Ashish.

Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications Robert Schweller 1, Zhichun Li 1, Yan Chen 1, Yan Gao 1, Ashish.
CS728 Lecture 16 Web indexes II. Last Time Indexes for answering text queries –given term produce all URLs containing –Compact representations for postings.

CS728 Lecture 16 Web indexes II. Last Time Indexes for answering text queries –given term produce all URLs containing –Compact representations for postings.
Reverse Hashing for Sketch Based Change Detection in High Speed Networks Ashish Gupta Elliot Parsons with Robert Schweller, Theory Group Advisor: Yan Chen.

Reverse Hashing for Sketch Based Change Detection in High Speed Networks Ashish Gupta Elliot Parsons with Robert Schweller, Theory Group Advisor: Yan Chen.
Presented by Ozgur D. Sahin. Outline Introduction Neighborhood Functions ANF Algorithm Modifications Experimental Results Data Mining using ANF Conclusions.

Presented by Ozgur D. Sahin. Outline Introduction Neighborhood Functions ANF Algorithm Modifications Experimental Results Data Mining using ANF Conclusions.
Dream Slides Courtesy of Minlan Yu (USC) 1. Challenges in Flow-based Measurement 2 Controller Configure resources1Fetch statistics2(Re)Configure resources1.

Dream Slides Courtesy of Minlan Yu (USC) 1. Challenges in Flow-based Measurement 2 Controller Configure resources1Fetch statistics2(Re)Configure resources1.

Similar presentations

Ads by Google