Download presentation
Presentation is loading. Please wait.
Published byEustacia Newman Modified over 9 years ago
1
49221052 施賀傑 69521041 何承恩 TelegraphCQ
2
Outline Introduction Data Movement Implies Adaptivity Telegraph - an Ancestor of TelegraphCQ Adaptive Building Blocks Initial CQ Approaches TelegraphCQ Conclusion and Future Work
3
Introduction TelegraphCQ is an extension to the Telegraph project Handling large streams of continuous queries over high- volume, highly-variable data streams Traditional data processing environment is not suitable for motion data Large scale Unpredictability of the environment Need for close interaction with users
4
Data Movement Implies Adaptivity Traditional database are inappropriate for dataflow processing Streaming data Pushing, instead of pulling Have to be processed on the fly Continuous Queries (CQ) Queries are continuously active Data initiates access to queries
5
Data Movement Implies Adaptivity Shared processing Avoid blocking or interrupt dataflow Processing each query individually can be slow and wasteful of resources Queries should have some commonalities Other Sources of Unpredictability Deeply networked environment User may need to adjust the query on the fly based on the previous result
6
Telegraph - an Ancestor of TelegraphCQ Designed to provide adaptability to individual dataflow graphs Two new prototypes to extend Telegraph to support shared processing over streams CACQ PSoup
7
Adaptive Building Blocks Telegraph consist a set of modules Module Types Ingress and Caching Query Processing Adaptive Routing
8
Adaptive Building Blocks
9
Ingress and Caching Interface with external data sources HTML/XML screen scraper (TeSS) Proxy for fetching data from peer-to-peer networks (TeleNap) Query Processing Routing tuples through query modules on a tuple-by-tuple basis A special type of module known as a State Module (SteM)
10
Adaptive Building Blocks Adaptive Routing Construct a query plan that contains adaptive routing modules Be able to re-optimize the plan while a query is running Eddy : route data to other query operators Juggle : perform online reordering FLuX : route tupples to support parallelism with load-balancing and fault-tolerance
11
Eddy Continuously route tuples among a set of other modules according to a routing policy
12
Eddy Routing policy Naive Eddy: Handle only operators with different costs but equal selectivity Deliver tuples to the two selection equally Fast Eddy: Improve the Naive Eddy with Lottery Scheduling Tuple to operator → costs a “ticket” Operator return a tuple → a “ticket” is debited Benefit: nearly optimal performance with less effort
13
Eddy Our query
14
Eddy Suppose s1 and s2 have the same selectivity Set s2 cost 5 delay units
15
Eddy Suppose s1 and s2 have the same cost Set the selectivity of s2 fixed at 50%
16
Eddy
17
SteMs A temporary repository of tuples
18
Fjords An inter-module communications API Allow query plans to use a mixture of push and pull connections between modules
19
Initial CQ Approaches CACQ PSoup Limitation of CACQ and PSoup Restricted their processing to data that could fit in memory Did not investigate scheduling and resource management issues for queries with little or no overlap Did not explicitly deal with the notion of QoS for adapting to resource limitation Did not explore opportunities for varying the degree of adaptivity to tradeoff flexibility and overhead
20
CACQ First continuous query engine to exploit the adaptive query processing framework of Telegraph Modify Eddies to execute multiple queries simultaneously Use grouped filters to optimize selections in the shared execution of the individual queries.
21
CACQ
22
PSoup Extend the mechanisms developed in CACQ in two main ways Allow queries to access historical data Support disconnected operation New queries can be applied to old data New data can be applied to old queries Accomplished by creating a query SteM
23
PSoup
24
For example: add a new query
25
PSoup
26
Exercise: add a new data using PSoup with example step by step R.a=3 R.b=6
27
PSoup
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.