Design and Implementation of a Middleware for Sentient Spaces Bijit Hore, Hojjat Jafarpour, Ramesh Jain, Shengyue Ji, Daniel Massaguer Sharad Mehrotra, Nalini Venkatasubramanian,Utz Westermann School of Information and Computer Science University of California, Irvine

SATware: Middleware for Sentient Systems 2 Outline Rescue project & Responsphere Multimodal sensor data processing SATware architecture  Stream model  SATRuntime: Stream processing runtime  SATLite: Stream query and transformation language  SATDeployer: Operator deployment  Privacy SATware Demo Related work

SATware: Middleware for Sentient Systems 3 Rescue project and Responsphere Exploiting IT for effective response to unexpected events Responsphere: a unique infrastructure for situation monitoring and awareness in the context of emergency response  Includes more than 200 sensors of 10 different types deployed over the geographical space that covers about half of the UCI campus.

SATware: Middleware for Sentient Systems 4 Multimodal sensor data processing: Application SATware applications:  Surveillance and monitoring  Emergency response Situation awareness Drills

SATware: Middleware for Sentient Systems 5 Surveillance Application A building is equipped with multimodal sensors Video cameras. RFID readers, Motes,.. Specific event should be detected and notifications should be generated

SATware: Middleware for Sentient Systems 6 Multimodal sensor data processing: Challenges Abstraction Entities and spaces Virtual sensor concept Multimodal stream processing Heterogeneous stream sources e.g., video cam, RFID, mote,.. Scalability Extensibility Mobility Privacy

SATware: Middleware for Sentient Systems 7 SATware A distributed middleware for capturing semantically meaningful events Provides higher level of semantic abstraction compared to sensor middleware Two level stream processing:  Semantic event stream processing  Sensor data stream processing

SATware: Middleware for Sentient Systems 8 QoS Privacy Stream QoS Privacy Situation model Domain model Uninterpreted streams of sensor event observations Interpreted streams of entity-related events Sensing layer Semantic layer: entities and activities User Application SATware stream processing model

SATware: Middleware for Sentient Systems 9 SATware architecture: Stream model Data model  A stream is a flow of records  t : time stamp  C: content  Confidence: probability of the correctness of tuple content 

SATware: Middleware for Sentient Systems 10 SATware architecture: Stream model Processing model  Operators Input, output buffer  Operator types Image processing operators Join operators Synchronization operators Filtering operators … OiOi

SATware: Middleware for Sentient Systems 11 SATware architecture: Stream model OiOi OiOi Sample image processing operator Sample join operator: OiOi Sample synchronizer operator:

SATware: Middleware for Sentient Systems 12 SATware architecture

SATware: Middleware for Sentient Systems 13 SATware architecture: SATRuntime Main components  Set of processing nodes  SATRuntime component One SATRuntime on each processing node Interface for operators (running, accepting, migrating, connecting,…)  Operators as mobile agents Input stream, output stream buffers Operator connection interface Operator migration

SATware: Middleware for Sentient Systems 14 SATware architecture: SATRuntime SATRuntime infrastructure directory Available streams Sensor and Operator repository  Sensor management  public boolean addSensorInstance(SensorInstance sensorInstance);  Sensor.setLocation( Location l );  Operator management  public boolean addOperatorInstance(OperatorInstance operatorInstance); Deployment status  Operator graph  Operator location

SATware: Middleware for Sentient Systems 15 SATware architecture: SATLite SATLite: Defines the query plan Set of interconnected operators fed by sensors SATLite provides a query plan by translating the query into a graph  Select t, PersonID from DoorStatus and PersonID where Person.t = DoorStatus.t and DoorStatus=‘OPEN’ OiOi OjOj

SATware: Middleware for Sentient Systems 16 SATware architecture: SATDeployer Translating a query plan into a deployment plan  Mapping query plan nodes into processing nodes Minimizing query processing cost  Network cost  Computing cost  Query evaluation time Optimizations Operator reuse Load balancing

SATware: Middleware for Sentient Systems 17 SATware architecture: Privacy Hiding individual’s identity Revealing individual’s identity when a predefined policy is violated Sensors are trusted components Adversary is an inquisitive but non-interfering (passive) entity that can observe all stored data and intercept all communication between two components of the system.

SATware: Middleware for Sentient Systems 18 Sentient building demo A pervasive space equipped with sensors Video cams, RFID readers, Motes, Mobile sensing platforms,… SATRecorder: a building surveillance application Recording video feed based on timed schedule or event detection Coffee room scenario Sensors: 2 video cams, RFID reader Sample policy: if pot is empty and burner is on record video

SATware: Middleware for Sentient Systems 19 Sentient building demo

SATware: Middleware for Sentient Systems 20 Sentient building demo

SATware: Middleware for Sentient Systems 21 Sentient building demo: coffee room Event definition: Select PersonID from KitchenEvent where CoffeeLevel = ‘EMPTY’ and PotInPlace = TRUE and BurnerStatue = ‘ON’ Network model and query plan

SATware: Middleware for Sentient Systems 22 Sentient building demo: coffee room

SATware: Middleware for Sentient Systems 23 Related work Data stream processing  Aurora, STREAM, TelegraphCQ Sensor networks Multimedia stream processing  IBM S3, MedsMan, IrisNet, Smart Camera Network

SATware: Middleware for Sentient Systems 24 Conclusion We proposed:  Challenges in sensor stream processing  SATware: A middleware for sentient spaces  A surveillance application based on SATware Future plan:  Expanding sensor deployment  Sophisticated event detection  Improving SATLite and SATDeployer  Integrating more sophisticated privacy preserving techniques