Watchdog Confident Event Detection in Heterogeneous Sensor Networks Matthew Keally 1, Gang Zhou 1, Guoliang Xing 2 1 College of William and Mary, 2 Michigan State University

Overview Problem Statement Challenges Related Work Contributions Design Evaluation 2

Confident Event Detection Many applications for event detection have stringent accuracy requirements and demand long system lifetimes  Vehicular traffic monitoring  Falls in elderly patients  Military/intrusion detection Perform confident event detection  Meet user-defined false positive and false negative rates in the presence of in-situ sensing reality  Reduce energy usage to extend system lifetime 3

Challenges of Confident Event Detection How to cluster the right sensors to meet user accuracy requirements?  Learn the detection capabilities of individual sensors and clusters  Use part of the detection capability to meet user requirements and save energy How to efficiently perform collaboration between heterogeneous sensors to meet user requirements?  Difficult for modality-specific models and data fusion  Need a generic solution How to adapt detection capability to runtime observations?  Easier observations and harder observations need different detection capabilities 4

Related Work Sensing Coverage  Do not address user accuracy requirements  Do not explore detection capability of deployment Modality-specific Sensing Models and Data Fusion  User requirements not met in reality  Difficult to perform heterogeneous sensor fusion  Do not cluster the right sensors to meet user requirements Machine Learning  Do not address user accuracy requirements  Do not adapt sensing capability to runtime observations 5

Motivation: Related Work Shortfalls Vehicle Detection: sensing irregularity  Same distance, different accuracies  Accuracy can increase with distance Sensing Coverage may overdetect or underdetect events Theoretical sensing models assume all sensors are identical 6

Motivation: Related Work Shortfalls Different clusters (C 1,C 2,C 3 ) have the same accuracy, 100%, better than individual sensors  Difficult to capture for existing works: Due to lack of knowledge of detection capability of different sensors and clusters 7

Watchdog Contributions A confident and energy efficient event detection framework  Choose the right sensors to meet user requirements  Generic framework that provides heterogeneous sensor fusion Adapt detection capability to runtime observations  Easy observations: low-power sentinel sensors  Hard observations: higher-power reinforcement sensors Performance evaluation: two scenarios  Monitor traffic entering and leaving computer science building  Vehicle detection using Wisconsin trace data  Compare against sensing coverage and signal attenuation model 8

Watchdog Design Overview Efficient heterogeneous collaboration Explore detection capability of a deployment Cluster the right sensors to meet user requirements Adapt detection capability to runtime observations Node Local Aggregation Runtime Event Detection Request Reinforcement Data Cluster Generation Sentinel and Reinforcement Selection Sensor Aggregator Training Results Observations  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 9

Cluster Generation Goal: determine detection capability of  Individual sensors and sensor clusters  A specific deployment Method  Randomly generate up to M clusters for each cluster size  For each generated cluster  Step 1: Train a Hidden Markov Model for the cluster HMM is good for heterogeneous sensor fusion HMM captures time and space correlation of sensor data  Step 2: Determine cluster FP/FN based on the HMM decision and ground truth at each time interval  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 10

Step 2: Determine cluster FP/FN based on the HMM decision and ground truth At each aggregation interval:  Determine event detection decision with trained HMM  Compare cluster detection decision with ground truth Get the cluster FP/FN (accuracy)  Determine FP/FN for each possible event probability  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 11

Sentinel and Reinforcement Selection Choose sentinel cluster: low detection capability – Meets user's FN requirement – Makes easy detection decisions Choose reinforcement cluster: higher detection capability – Meets both FP and FN requirements – Used to make more difficult detection decisions All other sensors go to sleep  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 12

Runtime Event Detection Goal: adapt detection capability to runtime observations – Easier observations and harder observations need different detection capabilities Method: – Sentinels and reinforcements form local observations at each aggregation interval – Sentinels report non-default observations to the aggregator to make detection decisions – Reinforcements requested when sentinel event probability false positive rate exceeds user requirements – Reinforcements return non-default observation data and aggregator makes a confident decision  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 13

Runtime Event Detection User requirements: u.FN = u.FP = 0.05  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection Acoustic Seismic Sentinels Reinforcements Aggregator Time interval t=1: No Event, s.FN =.01 < u.FN t=2: Event, s.FP =.02 < u.FP t=3: No Event, s.FN =.01 < u.FN t=4 :Undecided, s.FP =.45 > u.FP t=4 :Event, r.FP = 0.3 < u.FP t=5: No Event, s.FP = 0.2 < u.FP

Evaluation App1: Wisconsin SensIT trace data – Vehicle detection at a fixed location – 75 nodes with acoustic, seismic, and infrared sensors – 100ms aggregation interval App2: Computer Science Building Traffic Monitor – Five IRIS motes mounted on main entrance door – MTS 310: 2-axis accelerometer, 2-axis magnetometer, acoustic, and light sensors – Define event as when someone opens the door and walks through – 4s aggregation interval Compare with a modality-specific sensing model – Distance-based signal attenuation – Data fusion for event decisions Compare with V-SAM, a state of the art protocol for handling sensing irregularity – Measure data similarity between sensors – Keep awake only sensors with dissimilar readings 15

Only a limited & discrete number of FP/FN rates supported by the deployment For a specific FP/FN rate, a large number of clusters may be available During runtime detection, Watchdog meets FP/FN explored during training Exploring Detection Capability & Meeting Requirements 16

Compare with V-SAM: Accuracy V-SAM with k-coverage and similarity coverage Watchdog outperforms all with near perfect accuracy 17

Compare with Modality-Specific Sensing Model: Accuracy Vehicle detection with acoustic sensors – Select clusters with two different ranges to target location: near ( 40m) Watchdog always meets user requirements Modality-specific model ignores in-situ sensing reality 18

Compare with Modality-Specific Sensing Model: Energy Watchdog clusters the right sensors to meet user requirements – Meets requirements with reduced energy Watchdog adapts its capability to runtime observations to save energy Modality-specific sensing model uses all sensors in the cluster 19

Adapting Detection Capability to Runtime Observations Experimental setting – Vehicle trace data and sensors from <25m – User requires 0% false positives and false negatives Watchdog clusters the right sensors to meet user requirements Neither V-SAM nor the modality-specific sensing model adapts detection capability to runtime observations Sentinel FP/FN (%) Reinforcement FP/FN (%) Reinforcement Requests (%) 9.5/0.00.0/

Conclusions and Future Work Existing works do not provide event detection with confidence, we need to – Cluster the right sensors to meet user requirements – Provide a generic approach for heterogeneous deployments – Adapt detection capability to runtime observations Watchdog: a confident event detection framework – Meets user accuracy requirements – Exceeds accuracy of existing approaches – Uses knowledge of detection capability to save energy Future Work – Online and distributed detection 21

Thanks to NSF grants ECCS and CNS

Compare with V-SAM: Training Length Watchdog achieves maximum performance with a short training V-SAM requires little training, but is less accurate 23

Local Aggregation Allows for heterogeneous sensor fusion Raw data is combined to form a single observation – Use a common aggregation technique Discrete, finite number of possible observations – Same number for each sensor and modality – Allow for comparison between sensors of all modalities – We use two discrete observations  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 24

Event Probability Discussion Differentiate the accuracy between different event probabilities – Some observations are more reliable than others – Probabilities near 0.5 are more inaccurate Determine FP and FN for each of p probability ranges (p=10) – Probability between.1 and.2 has zero false negatives – Probability between.9 and 1.0 has 6% false positive rate – Ranges with no events have 100% false positive or false negative rates  Local Aggregation  Cluster Generation  Sentinel & Rein. Selection  Runtime Event Detection 25