Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse UC Berkeley EWSN, Feb 13, 2006 Joint with Feng Zhao and Jie Liu, Microsoft Research

fact Data Processing Data Processing fact Data Processing Motivation Imaging that sensors cover the globe Can we automatically query for world facts? No: each fact is an interpretation of data Processing must consider –Sensor context –Sensor fusion –Data formats –Calibration –Signal transforms –Etc. World Sensor DB fact Data Processing fact Data Processing

Semantic Streams Goal: to allow direct querying of facts Example query: “Is a vehicle in the parking lot” Instantiates a graph of inference units to derive the desired fact Output is a stream of all vehicles detected Fact: vehicle Size Inference Vehicle Inference Space Inference Fact: empty space Sensor.data=72Sensor.data=56Sensor.data=98obj.size=23 Sensor.data=49 obj.type=vehicle

Usage Model Fixed Sensor infrastructure –Many simultaneous, unrelated users –Short-term queries repeatedly over long periods –Queries are often similar (though not identical) When programmer poses new query, system will use existing sensors and inference units, if possible. Otherwise, system may give actionable error messages, –need to add 2 more sensors in area XYZ –need to add new inference unit to the system Semantic values produced by infrastructure grows organically as it is used for new purposes

Complete System Mobile Application Query Processor Inference Graph Embedding Engine MSTML Execution Engine Sensors Planning time User Output Stream Inference Unit Instances Run time

Background: Prolog and FOL george, jane, carA X, Y, Z, StudentA female(jane), parent(X,Y) parent(george,jane). male(george). father(X) :- parent(X,Y),male(X). male(X). > X = george father(X). > X = george father(jane). > false Constants Variables Predicates Facts Rules Queries Query processing

Markup Language sensor(magnetometer, [[60,0,0][70,10,10]]). sensor(camera, [[40,0,0][55,15,15]]). sensor(breakBeam, [[10,0,0][12,10,2]]). inference( speedDetector, needs( sensor(magnetometer, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R,region), property(X,S,speed) ) ). Sensors are declared as logical facts –All regions are appx’d by 3D cubes Inference units are logical rules –Antecedents: the fact streams that it needs –Consequents: the fact streams that it creates –Spatial relationship encoded in variable name

Stream Relationships subregion( A, B ) inference(speedDetector, needs( sensor(magnetometer, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), subregion(R2,R), property(X,S,speed) ) ). More sophisticated spatial relationships –Subregions Sensor coverage (R) Queried area (R2)

Stream Relationships inference( speedDetector, needs( sensor(breakBeam, R1), sensor(breakBeam, R2), sensor(breakBeam, R3), subregion(R1,R), subregion(R2,R), subregion(R3,R), \+ intersecttion(_,R1,R2), \+ intersecttion(_,R1,R3), \+ intersecttion(_,R2,R3) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R,region) ) ). More sophisticated spatial relationships –Subregions –Non-intersection Queried area Sensor coverage

Stream Relationships inference( vehicleDetector, needs( sensor(magnetometer, R), stream(X), isa(X,object), property(X,S,speed) ), creates( stream(X), isa(X,vehicle) ) ). More sophisticated spatial relationships –Subregions –Non-intersection Stream identity

Query processing 1.Choose first predicate in query 2.Search KB for sensors that match 3.Search output of inference units for streams that match 4.Add new input streams to query 5.Back to step 1 Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(X,vehicle) stream(X), property(X,speed) Sensor(mag,…) Vehicle Inference stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Speed Inference

Query processing Actionable Error Messages –When missing inference unit –When missing sensor Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Speed Inference

Query processing When too many choices –Multiple graphs provide logically equivalent results Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Speed Inference stream(Y), property(Y,speed) Camera Speed Inference

Quality of Service Constraints subregion( A, B ) inference(speedDetector, needs( sensor(camera, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), property(X,S,speed), property(X,C,confidence), {C > 99}, property(X,L,latency), {L < 450} ) ). Inference unit may specify QoS of output –Confidence –Latency

Quality of Service Constraints subregion( A, B ) inference(speedDetector, needs( sensor(breakBeam, R1) ), sensor(breakBeam, R2) ), sensor(breakBeam, R3) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), property(X,S,speed) ) ). property(X,C,confidence), {C > 80}, property(X,L,latency), {L < 75} ) ). Inference unit may specify QoS of output –Confidence –Latency

Quality of Service Constraints subregion( A, B ) inference(vehicleDetector, needs( sensor(magnetometer, R1), stream(X), property(X,S,speed), property(X,C2,confidence), property(X,L2,latency), creates( stream(X), isa(X,vehicle), {C > 0.9 * C2}, {L = L } ) ). Inference unit may specify QoS of output –Confidence –Latency May be function of input QoS

Query processing Query can declare QoS parameters to create ordering of inference graphs Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Size Inference stream(Y), property(Y,speed) Camera Size Inference {C>70, L <300}

Example Infrastructure

Goal Application: Vehicle Detection

Query and Inference Graph isa(Y,histogram),property(Y,T,value),property(X,T,time), isa(X,vehicle),region(X,R,[[10,10,0][40,30,12]]), Isa(Z,photo),property(Z,X,triggerStream) Vehicle Inference Speed Inference Direction Inference Size Inference Histogram Service Photo Service Break Beams Magnetometers Camera

Screenshot

Final Application Camera image Report (histogram) Break beam plot Micro ServerCamera Server Magnetometer plot

Conclusion Semantic Streams –Infers facts from data –Intended for sensor infrastructure –Can select inference graph based on QoS constraints Limitations –Must be a large supply of inference units –Stream operators are difficult to program Different programming language/execution model Programmer must deal with temporal values, implement appropriate buffering, etc.

Future Work: Streaming KB’s Relational Database : Knowledge Base :: Streaming Database : Streaming KB SQL operators: Select, Join, Max Operate on Tables Logical rules: c(X) :- a(x),b(x). Operate on Facts Infinite tablesInfinite facts

Video: Vehicle Detection

Video: Human Filtering

Traffic Histogram