Exploiting Semantics for Sensor Re-Calibration in Event Detection Systems Ronen Vaisenberg, Shengyue Ji, Bijit Hore, Sharad Mehrotra and Nalini Venkatasubramanian School of Information and Computer Sciences University of California - Irvine

Outline  Motivation – Self Managed Sensors  Event detection systems  Task  Challenge  Our approach  Exploiting semantics in Event Detection Systems  Algorithms  Analysis  Experiments and results

Motivation – Self Managed Sensors Managing Networks of Sensors can be very Challenging when sensors are: Distant and isolated Deployed in Large numbers Human intervention is not desirable.

Example 1 – Isolated/Distant Sensors Microwave radio-meters are used to provide continuous measurements of integrated water vapor (IWV) and integrated liquid water (ILW). Deployed in isolation. (Rural Oklahoma and Kansas, the north slope of Alaska). Physical changes in the sensor: Operates Based on mirrors that tend to slip as much as 1 degree on their stepper motor shafts due to continuous use. Human intervention is not desirable, Too far for frequent visits

The Task – Geographically Distributed Sensors Traffic sensors are to collect accurate reads of the current traffic on state roads.

Example 2 – Correct Functioning is a Matter of Life or Death Collect accurate reads of the pressure under the bridge and in strategic points to detect dangerous pressure levels. Pressure sensors need to accurately reset “zero” pressure. Human intervention is not desirable, can’t wait until technician becomes available

Example 3 – Network with Large Number of Video Sensors Video surveillance systems are being used in a variety of public spaces such as metro stations, airports, shipping docks, etc.

The Challenge: Inner-Sensor Change Physical Changes in the System The detection algorithm in the ARM microwave radio-meter has to adjust to changes in the physical location of its mirrors that tend to slip as much as 1 degree on their stepper motor shafts due to continuous use.

Example 3 – Network with Large Number of Video Sensors (Cont.) Changes in the field of view Human intervention is not desirable, Too many cameras

Outline  Motivation – Self Managed Sensors  Event detection systems  Task  Challenge  Our approach  Exploiting semantics in Event Detection Systems  Algorithms  Analysis  Experiments and results

The System Monitored – The context: Event Detection Systems (EDS) The fundamental concept underlying EDS’s is the gathering, managing, analyzing, and interpreting of different information streams in a timely manner to recognize potential incidents early enough to respond effectively. In most cases, event detection systems gather streams from sensors. Each sensor monitors a system of interest. This system can be modeled.

The System Monitored – The Modeling: Finite State Machine Many systems of interest and observed environments can be modeled as Finite State Machines. For example: Level of pressure (“Critical", ”High", “Medium", “Low") on the foundations of a bridge are of crucial importance to maintenance authorities. Traffic light states (“Red”, “Orange”, ”Green”)

The Sensor’s Task Event Detection Task Given an FSM representation of a system, containing states S={s 1..s n } generate an accurate time-series:

Coffee-Level Example – A More Complicated System Application: detect when there is fresh coffee. Can be modeled as a finite state machine Challenge: changes in view.

System Evolution – Change of Camera Example of Small Change of View State A State B State C State D Timeline System Evolution Time State A State D

The Challenge: Recover from “System Evolution” State A State B State C State D Timeline System Evolution Time State A State B State D State C

The Challenge Formally How can we perform autonomous re- calibration under a generic setting assumption? Video data: General purpose object recognition proves to be very limited... And even if you detect the object, how would you translate sensed features to different states. Other streams: What is an object? How to search for a system of interest.

Semantics to the rescue Semantics (one of the definitions) The meanings assigned to symbols and sets of symbols in a language. If a computer understands the semantics of a system, it understands the meaning, rather than just interpreting a set of features.

The Proposed Solution State A State B State C State D Timeline System Evolution Time State A State B State D State C

Semantic Model States of the FSM: Red, Orange, Green) State transition: {G->O,O->R,R->G} Semantics used for re-calibration: transition times between states should be in [avg-2std, avg+2std]. AvgStdavg-2stdavg+2std G->O O->R R->G

Fireflies – Exploiting Semantics in Nature Semantics are used by fireflies to search for the best mate. Source: BBC Transition time between “on” and “off” should be in: (0,maxTime]

Directory  Motivation – Self Managed Sensors  Event detection systems  Task  Challenge  Our approach  Exploiting semantics in Event Detection Systems  Algorithms  Analysis  Experiments and results

Information Flow – Regular Detection Process Perform State Detection User Specified Prediction Parameters Time Series State Data Real Time Image Input OutputSensingAuxiliary Services Extract Features Parameterized Prediction State A State B State C State D ….. t i =“Red”

Information Flow – Semantics Based Detection Process Perform State Detection User Specified Prediction Parameters Time Series State Data Real Time Image Input OutputSensingAuxiliary Services Extract Features Parameterized Prediction State A State B State C State D Sliding Window of Images ….. t i =“Red” Manage Semantic Module System Evolution Occurred Detect System Evolution No Perform Recalibration Yes Auto Generated Parameters Semantic State A State B State D State C

Parameter Recalibration Search in the space of parameters for new parameters that optimize consistency of the stream generated with the model. Forall p 1,p 2,..,p n do Use p 1, p 2,..,p n to generate a new stream based on features in the buffer Evaluate the previous stream against the semantic model return p 1,p 2,…,p n if consistent

DEMO: Searching for the correct field of view Using Semantics Only Search in the space of parameters for new parameters that optimize consistency of the stream generated with the model. Forall p 1,p 2,..,p n do Use p 1, p 2,..,p n to generate a new stream based on features in the buffer Evaluate the previous stream against the semantic model return p 1,p 2,…,p n if consistent Forall different fields of view do Extract features from the field of view from all images in buffer. Cluster features extracted to m clusters (m is the number of states). Forall possible m! state assignments to the different states do Use labeled centroids of clusters to generate a new state stream Evaluate the previous stream against the semantic model return p 1,p 2,…,p n if consistent

Detect System Evolution Evaluate the consistency of the last M time series data elements generated using current prediction model. The consistency value found in the previous step is compared to a predefined threshold. Forall (M) State Transitions (s i →s j ) determined based on features in buffer do avg ← get average transition time from model for ( s i →s j ) stdv ← get standard deviation time from model for (si→sj) If transition time for (s i →s j ) is not in [avg-2*stdv, avg+2*stdv] faults++ if faults/(number of State Transitions)>threshold declare “system evolution”

Probabilistic Analysis – Probability to avoid re-calibration under normal operation (No System Evolution) Probability of the observations collected in the WindowBuffer to have less than C=ThresholdConsistent percent of deviations observing a system with p = 4.4% probability to generate a transition outside the 2 stdv range.

Analysis After a system evolution is declared at timedetected, we know evolution happened before timedetected. The problem is we don't know exactly when. Since we want to search for new parameters only for the evolved state, we propose waiting until WindowBuffer contains only features that were recorded after timedetected before engaging in the re-calibration phase. System at New State Actual System Evolution Time System at Old State Sliding Window Time of Evolution Detection Sliding Window

Experimental Results - Coffee-Level Detection Data Collection Two cameras: From June 11 th to June 13 th Record an image when changes are detected anywhere in view 1174 images (60MB) were collected Number of actual transition transitions: 203 Bounding box adjusted correctly in both cameras when changed to make the prediction algorithm confuse two states Empty↔Off Empty↔Half-Full

Experimental Results - Coffee-Level Detection After semantic based calibration, detection is much more accurate

Future Work Multi sensor systems Incorporating better models for anomaly detection Incorporating features in the process of re- calibration

Questions/Comments?