Enabling Real Time Alerting through streaming pattern discovery Chengyang Zhang Computer Science Department University of North Texas 11/21/2016 CRI Group Meeting

Motivation Real Time Event Detection  Flood Detection  Sudden change of water quality System Malfunction Detection  Some sensors are dead  Some sensors give incorrect readings In any case, it is likely there are some anomalous pattern changes in the data! 11/21/2016 CRI Group Meeting

How to automatically identify such changes Threshold-based approach  E.g. Temp; relative humidity; barometric pressure; sun radiation ; ; ; ; We can easily set the temperature threshold to be say, between 0F, and 150F  Problems:  Has to be manually assigned  Cannot reflect temporal patterns Temp; relative humidity; barometric pressure; sun radiation, time 50; ; ; ; :00:00  Cannot reflect correlations between sensor readings Time; UVB; Temp :00:00; 0.727; :00:00; 0.827; :00: ; 73.4 System Malfunction Detection  Some sensors are dead  Some sensors give incorrect readings In any case, it is likely there are some pattern changes in the data! 11/21/2016 CRI Group Meeting

How to automatically identify such changes Simple Statistics-based approach  Historical Mean Temp; relative humidity; barometric pressure; sun radiation, time 50; ; ; ; :00:00  Mean of Multiple Sensors  Problems:  Still not able to reflect correlations among data 11/21/2016 CRI Group Meeting

Objective 11/21/2016 CRI Group Meeting Objective  Find hidden trend from the correlate data  On the fly(real time) detection  Limited memory

Slides 7 to 17 are extracted from reference 2 11/21/2016 CRI Group Meeting

1. How to capture correlations 1 ? 20 o C 30 o C Temperature T 1 First sensor time

1. How to capture correlations? First sensor Second sensor 20 o C 30 o C Temperature T 2 time

20 o C30 o C 1. How to capture correlations 20 o C 30 o C Temperature T 2 Temperature T 1 First three lie (almost) on a line in the space of value- pairs…  O(n) numbers for the slope, and  One number for each value- pair (offset on line) Offset “ hidden variable ” time=1 time=2 time=3

1. How to capture correlations 20 o C30 o C 20 o C 30 o C Temperature T 2 Temperature T 1 Other pairs also follow the same pattern: they lie (approximately) on this line

2. Incremental update error 20 o C30 o C 20 o C 30 o C Temperature T 2 Temperature T 1 For each new point Project onto current line Estimate error New value

2. Incremental update 20 o C 30 o C 20 o C30 o C Temperature T 2 Temperature T 1 For each new point Project onto current line Estimate error Rotate line in the direction of the error and in proportion to its magnitude

Stream correlations Principal Component Analysis (PCA) The “line” is the first principal component (PC) vector This line is optimal: it minimizes the sum of squared projection errors

T3T3 3. Number of hidden variables If we had three sensors with similar measurements Again: points would lie on a line (i.e., one hidden variable, k=1), but in 3-D space T1T1 T2T2 value-tuple space

T3T3 3. Number of hidden variables Assume one sensor intermittently gets stuck Now, no line can give a good approximation T1T1 T2T2 value-tuple space

T3T3 3. Number of hidden variables Assume one sensor intermittently gets stuck Now, no line can give a good approximation But a plane will do (two hidden variables, k = 2) T1T1 T2T2 value-tuple space

Number of hidden variables (PCs) Keep track of energy maintained by approximation with k variables (PCs): Reconstruction accuracy, w.r.t. total squared error Increment (or decrement) k if fraction of energy maintained goes below (or above) a threshold If below 95%, k  k  1 If above 98%, k  k  1

11/21/2016 References 1.Spiros Papadimitriou, Jimeng Sun, Christos Faloutsos. Streaming pattern discovery in multiple time- series. In VLDB’05. (Part of the slides are also borrowed from their vldb presentation) CRI Group Meeting

11/21/2016 Questions? CRI Group Meeting