Video and Sensor Network Architecture and Displays Douglas Fidaleo Mohan Trivedi (PI) Center for Wireless COMMUNICATIONS

Overview Sensor data can be overwhelming especially with visual (video) sensors Transmission of higher level semantic content rather than low level detail Network architecture in support of this goal

Network Architecture: Goals Data warehouse Flexibility Reconfigurability Easily Scalable Bi-directional communication Simulated sensor-to-sensor communication Sensors 2-3 min Scalability is especially important for sensor networks as there are so many types of sensors: GPS, Text message from PDA, seismic, temperature, fluid flow, video. And they come in totally different packages: Cameras come on cell phones, PDAs, fixed video feeds, controllable PTZ cameras. Bidirectionality: Most sensor networks are designed to collect information from point sources and transmit to a single data warehouse. These architectures don’t allow for adaptaptive sensing of the environment. For example, each local node likely has little analytical power, so most of the analysis is occuring off site…but what if that off-site processor determines that some change in sensing will make the information more useful…we need to be able to communicate that back to the sensor. A simple example of this is the PTZ unit of a video sensor. If the analysis module determines that there is something of interest in the scene, it should be able to send a control signal to the PTZ unit to allow it to focus in on the event. This is also essential for selective capture of events…for example, for high power or bandwidth sensors, we may not want to or be able to consistently capture information…we can hoewever, use a control signal to trigger the capture at times that are most relevant.

Observation Application Network Architecture Clients Database Application Observation Application GPS Cell phone Tablet PC Seismic sensor General framework Thin clients XML data packets Bi-directional communication Independent video server Video Analysis Application Server (2-3 min) Describe general architecture: thin clients, XML packets, bidirectional, Explain how different modules can be instantiated as examples of clients Explain how video must be handled differently as it is streaming data Show direct video server connections. Clients can feed extracted events onto the network.

Mobile Visual Communicator Clients Clients Database Application Observation Application GPS Cell phone Tablet PC Seismic sensor Tablet PC Wireless transmission and visualization of user position Remote mobile video observations snapshots GPS Video Analysis Application Server Server (2-3 min) Describe general architecture: thin clients, XML packets, bidirectional, Explain how different modules can be instantiated as examples of clients Explain how video must be handled differently as it is streaming data Show direct video server connections. Clients can feed extracted events onto the network.

Demo application:

Current Implementation Server built on top of Fortress online game server Secure client authentication Secure communication channels Added sensor network Java API Thin client Data descriptors Message routing and filtering Current implementation uses the Fortress online game server as the glue to provide basic secure client authentication and packet handling. We have developed a layer on top of this to handle client message filters and routing of packets. We have also developed a set of thin clients in Java for interfacing to the server.

Visualization and Interfaces Direct visualization of data streams Temperature gauge Embedded visualization GPS location on a spatial map Video augmented 2D maps Data abstraction and interpretation The NeST is intended to be a flexible backbone for event capture, but there are other challenges such visualization of the data streams. Again there are several different levels at which we can approach this: T A very important area of exploration is how to abstract and make sense of the abundance of sensor data. If we have a mish-mash of thousands of sensors, it is clearly impossible to directly visualize each sensor, whether they are embedded or not. For certain domains it may be possible to simply pick out certain sensors that are problematic with a beacon, but we may require more in-depth understanding.

Data Abstraction: The SimCity Analogy City is populated with sensors Advisors make sense of data Provide feedback to operator (Mayor) An interesting analogy here are the “advisors” in the game SimCity. The game is effectively a set of virtual sensors, and the advisor’s role is to make sense of the information and point out to the Mayor the areas that must be addressed. The advisor’s each have specific roles and goals, but their primary function is to interface between the abundant sensor data and the “operator”.

Future Plans: Short Term Security and Privacy Client-server architecture enhancement Sensor control API Application APIs Interface existing capture and analysis modules Face recognition, Body posture analysis, Body tracking Integrate a wider variety of sensors and processing clients RFIDs, PDAs, Cell phones Security: Can easily secure the connection channels between the clients and the server, but also need a way to restrict access to data Privacy: User selectable tracking is a first step. But the issue is more complicated. Object tracking generally occurs before identification, so there may be an implicit record of where you were for the last 5 minutes, before the system actually knows that this record belongs to you. So you can say “only track those objects for whom you have the identity”, but this excludes a huge portion of potentially useful information.

Future Plans: Longer Term Large scale data abstraction and human interfacing (aka the SimCity model) Sensor data archiving (DB)