1. Design and Implementation of a Middleware for Sentient Spaces Bijit Hore, Hojjat Jafarpour, Ramesh Jain, Shengyue Ji, Daniel Massaguer Sharad Mehrotra, Nalini Venkatasubramanian,Utz Westermann School of Information and Computer Science University of California, Irvine

2. SATware: Middleware for Sentient Systems 2 Outline Rescue project & Responsphere Multimodal sensor data processing SATware architecture  Stream model  SATRuntime: Stream processing runtime  SATLite: Stream query and transformation language  SATDeployer: Operator deployment  Privacy SATware Demo Related work

3. SATware: Middleware for Sentient Systems 3 Rescue project and Responsphere Exploiting IT for effective response to unexpected events Responsphere: a unique infrastructure for situation monitoring and awareness in the context of emergency response  Includes more than 200 sensors of 10 different types deployed over the geographical space that covers about half of the UCI campus.

4. SATware: Middleware for Sentient Systems 4 Multimodal sensor data processing: Application SATware applications:  Surveillance and monitoring  Emergency response Situation awareness Drills

5. SATware: Middleware for Sentient Systems 5 Surveillance Application A building is equipped with multimodal sensors Video cameras. RFID readers, Motes,.. Specific event should be detected and notifications should be generated

6. SATware: Middleware for Sentient Systems 6 Multimodal sensor data processing: Challenges Abstraction Entities and spaces Virtual sensor concept Multimodal stream processing Heterogeneous stream sources e.g., video cam, RFID, mote,.. Scalability Extensibility Mobility Privacy

7. SATware: Middleware for Sentient Systems 7 SATware A distributed middleware for capturing semantically meaningful events Provides higher level of semantic abstraction compared to sensor middleware Two level stream processing:  Semantic event stream processing  Sensor data stream processing

8. SATware: Middleware for Sentient Systems 8 QoS Privacy Stream QoS Privacy Situation model Domain model Uninterpreted streams of sensor event observations Interpreted streams of entity-related events Sensing layer Semantic layer: entities and activities User Application SATware stream processing model

9. SATware: Middleware for Sentient Systems 9 SATware architecture: Stream model Data model  A stream is a flow of records  t : time stamp  C: content  Confidence: probability of the correctness of tuple content 

10. SATware: Middleware for Sentient Systems 10 SATware architecture: Stream model Processing model  Operators Input, output buffer  Operator types Image processing operators Join operators Synchronization operators Filtering operators … OiOi

11. SATware: Middleware for Sentient Systems 11 SATware architecture: Stream model OiOi OiOi Sample image processing operator Sample join operator: OiOi Sample synchronizer operator:

12. SATware: Middleware for Sentient Systems 12 SATware architecture

13. SATware: Middleware for Sentient Systems 13 SATware architecture: SATRuntime Main components  Set of processing nodes  SATRuntime component One SATRuntime on each processing node Interface for operators (running, accepting, migrating, connecting,…)  Operators as mobile agents Input stream, output stream buffers Operator connection interface Operator migration

14. SATware: Middleware for Sentient Systems 14 SATware architecture: SATRuntime SATRuntime infrastructure directory Available streams Sensor and Operator repository  Sensor management  public boolean addSensorInstance(SensorInstance sensorInstance);  Sensor.setLocation( Location l );  Operator management  public boolean addOperatorInstance(OperatorInstance operatorInstance); Deployment status  Operator graph  Operator location

15. SATware: Middleware for Sentient Systems 15 SATware architecture: SATLite SATLite: Defines the query plan Set of interconnected operators fed by sensors SATLite provides a query plan by translating the query into a graph  Select t, PersonID from DoorStatus and PersonID where Person.t = DoorStatus.t and DoorStatus=‘OPEN’ OiOi OjOj

16. SATware: Middleware for Sentient Systems 16 SATware architecture: SATDeployer Translating a query plan into a deployment plan  Mapping query plan nodes into processing nodes Minimizing query processing cost  Network cost  Computing cost  Query evaluation time Optimizations Operator reuse Load balancing

17. SATware: Middleware for Sentient Systems 17 SATware architecture: Privacy Hiding individual’s identity Revealing individual’s identity when a predefined policy is violated Sensors are trusted components Adversary is an inquisitive but non-interfering (passive) entity that can observe all stored data and intercept all communication between two components of the system.

18. SATware: Middleware for Sentient Systems 18 Sentient building demo A pervasive space equipped with sensors Video cams, RFID readers, Motes, Mobile sensing platforms,… SATRecorder: a building surveillance application Recording video feed based on timed schedule or event detection Coffee room scenario Sensors: 2 video cams, RFID reader Sample policy: if pot is empty and burner is on record video

19. SATware: Middleware for Sentient Systems 19 Sentient building demo

20. SATware: Middleware for Sentient Systems 20 Sentient building demo

21. SATware: Middleware for Sentient Systems 21 Sentient building demo: coffee room Event definition: Select PersonID from KitchenEvent where CoffeeLevel = ‘EMPTY’ and PotInPlace = TRUE and BurnerStatue = ‘ON’ Network model and query plan

22. SATware: Middleware for Sentient Systems 22 Sentient building demo: coffee room

23. SATware: Middleware for Sentient Systems 23 Related work Data stream processing  Aurora, STREAM, TelegraphCQ Sensor networks Multimedia stream processing  IBM S3, MedsMan, IrisNet, Smart Camera Network

24. SATware: Middleware for Sentient Systems 24 Conclusion We proposed:  Challenges in sensor stream processing  SATware: A middleware for sentient spaces  A surveillance application based on SATware Future plan:  Expanding sensor deployment  Sophisticated event detection  Improving SATLite and SATDeployer  Integrating more sophisticated privacy preserving techniques