10/5/ Geometric Approach Geometric Interpretation: Geometric Interpretation: Each node holds a statistics vector Each node holds a statistics vector Coloring the vector space Coloring the vector space Grey:: function > threshold Grey:: function > threshold White:: function <= threshold White:: function <= threshold Goal: determine color of global data vector (average). Goal: determine color of global data vector (average).

10/5/ Bounding the Convex Hull Observation: average is in the convex hull  Observation: average is in the convex hull  If convex hull monochromatic then average too If convex hull monochromatic then average too But – convex hull may become large But – convex hull may become large

10/5/ Drift Vectors Periodically calculate an estimate vector - the current global Periodically calculate an estimate vector - the current global Each node maintains a drift vector – the change in the local statistics vector since the last time the estimate vector was calculated Each node maintains a drift vector – the change in the local statistics vector since the last time the estimate vector was calculated Global average statistics vector is also the average of the drift vectors Global average statistics vector is also the average of the drift vectors

10/5/ The Bounding Theorem [SIGMOD’06] A reference point is known to all nodes A reference point is known to all nodes Each vertex constructs a sphere Each vertex constructs a sphere Theorem: convex hull is bounded by the union of spheres Theorem: convex hull is bounded by the union of spheres  Local constraints!  Local constraints!

10/5/ Basic Algorithm Basic Algorithm An initial estimate vector is calculated An initial estimate vector is calculated Nodes check color of drift spheres Nodes check color of drift spheres Drift vector is the diameter of the drift sphere Drift vector is the diameter of the drift sphere If any sphere non monochromatic: node triggers re-calculation of estimate vector If any sphere non monochromatic: node triggers re-calculation of estimate vector

10/5/ Reuters Corpus (RCV1-v2) 800,000+ news stories 800,000+ news stories Aug Aug Aug Aug Corporate/Industrial tagging Corporate/Industrial tagging n=10 10 nodes, random data distribution

10/5/ Trade-off: Accuracy vs. Performance Inefficiency: value of function on average is close to the threshold Inefficiency: value of function on average is close to the threshold Performance can be enhanced at the cost of less accurate result: Performance can be enhanced at the cost of less accurate result: Set error margin around the threshold value Set error margin around the threshold value

10/5/ Performance Analysis

10/5/ Performance Analysis (cntd.)

10/5/ Balancing Globally calculating average is costly Globally calculating average is costly Often possible to average only some of the data vectors. Often possible to average only some of the data vectors.

SRDC /5/ Shape Sensitivity [PODS’08] Fitting cover to Data Fitting cover to Data Fitting cover to threshold surface Fitting cover to threshold surface Specific function classes Specific function classes

SRDC /5/ Fitting Cover to Data (using the covariance matrix)

10/5/ Fitting Cover to Threshold Surface -- Reference Vector Selection

10/5/ Distance Fields Skeleton, Medial Axis

10/5/ Results – Shape Sensitivity

e ΔV1ΔV1 ΔV2ΔV2 ΔV3ΔV3 ΔV4ΔV4 ΔV5ΔV5 f(v(t))  T epep ΔVp1ΔVp1 ΔVp2ΔVp2 ΔVp3ΔVp3 ΔVp4ΔVp4 ΔVp5ΔVp5 v(t) Stricter local constraints if local predictions remain accurate Keeping up with v(t) movement Prediction-Based Geometric Monitoring [SIGMOD’12]

SRDC 2013 Local Constraints 17 Let the nodes communicate only when “something happens” Tell me only if your measurement is larger than 50! Send me your current measurements! Safe Zones!

SRDC 2013 Local Distributions Reasonable to assume future data will behave similarly… These Safe Zones save more communication!

SRDC 2013 Optimal Safe Zones Legal / Safe 2. Large: Minimize Communication

SRDC 2013 Example: Air quality monitoring 20 What are the optimal Safe Zones…?

SRDC 2013 The Optimization Problem 21 Is this Convex? Is this Linear? How many constraints are these? BAD NEWS: This problem is NP-hard.

SRDC 2013 The Optimization Problem Step 3: Use non-convex optimization toolboxes (e.g. Matlab’s “fmincon”).  These toolboxes use sophisticated Gradient Descent algorithms and return close-to-optimal results. X

SRDC Data Set How the data looks like

SRDC 2013 Ratio Queries 24 Example of triangular Safe Zones

SRDC 2013 Improvement over convex-hull cover method 25 Why do we improve so much? Up to 200 nodes were involved in the experiment. The average improvement was by a factor of 17.5 Up to 200 nodes were involved in the experiment. The average improvement was by a factor of ’000 hours

26 Higher Dimensions

SRDC 2013 Chi-Square Monitoring (5D) 27 Examples of axis aligned boxes as Safe Zones

SRDC 2013 Improvement over GM The improvement over the Geometric Method gets more substantial in higher dimensions. 28 1’000 hours 90 nodes

SRDC Safe Zones - Example

SRDC 2013 Biclique: Non-Convex Safe Zones 30 Safe Zone Algorithm (for 2 nodes): Take the data points, build a bipartite graph(how?), find the maximal Biclique, these are your Safe Zones!

SRDC 2013 Conclusions Local filtering for large-scale distributed data systems Local filtering for large-scale distributed data systems Saving in communication is unlimited Saving in communication is unlimited Bounded only by the aggregate over system lifetime Bounded only by the aggregate over system lifetime Saving bandwidth, central resources, power. Saving bandwidth, central resources, power. Not necessary to sacrifice precision and latency Not necessary to sacrifice precision and latency Less communication  more Privacy Less communication  more Privacy 10/5/

SRDC 2013