1. IrisNet: A Planetary-Scale Architecture for Multimedia Sensors
Phillip B. Gibbons
Intel Research Pittsburgh
January 19, 2007
Slides are © Phillip B. Gibbons

2. What is a Sensor Network?
Tiny sensor nodes with very limited processing power, memory, battery. Scalar sensors (e.g., temperature)
Closely co-located, communicating via an ad hoc low-bandwidth wireless network
Singly tasked
Microservers?
not so tiny, PDA-class processor
Fault-line monitoring?
wide-area, not ad hoc
Broadband?
not low-bandwidth
Webcams?
not scalar, can be multi-tasked
Tanker/Fab monitoring?
powered, wired
© Phillip B. Gibbons

3. Sensor Networks is a Rich Space
Characteristics of sensor network depend on
Requirements of the application
Restrictions on the deployment
Characteristics of sensed data
Sampling the real world
Tied to particular place and time
Often write-dominated: Data Sampled far more than Data Queried
UCLA CENS NIMS
James Reserve
© Phillip B. Gibbons

4. Multimedia Sensor Systems
Rich collection of (cheap) sensors
Cameras, Microphones, RFID readers, vibration sensors, etc
Internet-connected. Potentially Internet-scale
Tens to millions of sensor feeds over wide-area
Pervasive broadband (wired & wireless)
Goal: Unified system for accessing, filtering, processing, querying, and reacting to sensed data
Programmed to provide useful sensing services
© Phillip B. Gibbons

5. Example Multimedia Sensing Services
Consumer services:
Parking Space Finder
Lost & Found / Lost pet
Watch-my-child / Watch-my-parent
Congestion avoidance
© Phillip B. Gibbons

6. Example Multimedia Sensing Services
Health, Security, Commerce, and Science services:
Epidemic Early Warning System
Homeland Security
Low Atmosphere Climate Monitoring
Asset/Supply Chain Tracking
Our prototype
Internet-scale Sensor Observatories
© Phillip B. Gibbons

7. Sensor Network Challenges?
Resource-constrained sensor networks present many unique challenges
Battery limitations => limiting communication, duty cycles, etc
Limited processing power & memory => new OS, languages, algorithms, etc
On the other hand, in multimedia sensor systems
Nodes are powerful, well-connected
Where’s the Challenge?
© Phillip B. Gibbons

8. Data & Query Scaling Challenges
Data scaling
Millions of sensors
Globally-dispersed
High volume feeds
Historical data
Query scaling
May want sophisticated data processing on all sensor feeds
May aggregate over large quantities of data, use historical data, run continuously
Want latest data, NOW
NetRad: 100Mb/s
© Phillip B. Gibbons

9. Further Challenges IrisNet addresses some of these challenges
System Scaling
How to manage a million node system?
How to write services for a million node system?
System Heterogeneity
Sensor types, Nodes, Networks, Admin. Domains
Shared Resource
Privacy & Security
Competing actuation
IrisNet addresses some of these challenges
Many open problems remain…
© Phillip B. Gibbons

10. Outline Multimedia Sensor Systems: Opportunities & Challenges
IrisNet overview
IrisNet key features
Conclusions
© Phillip B. Gibbons

11. IrisNet: Internet-scale Resource-intensive Sensor Network services
General-purpose architecture for wide-area sensor systems
A worldwide sensor web
Key Goal: Ease of service authorship
Provides important functionality for all services
Intel Research Pittsburgh + many CMU collaborators
First prototype in late 2002, in use
In ACM Multimedia, BaseNets, CVPR, DCOSS, DSC, FAST, NSDI(2), Pervasive Computing, PODC, SenSys, Sigmod(2)
© Phillip B. Gibbons

12. Data & Query Scaling in IrisNet
Store sensor feeds locally
Too much data to collect centrally
Push data processing & filtering to sensor nodes
Reduce the raw data to derived info, in parallel near source
Push queries to sensor nodes
Data sampled >> Data queried
Tied to particular place: Queries often local
Exploit logical hierarchy of sensor data
Compute answers in-network
© Phillip B. Gibbons

13. Ease of Service Authorship
What is a minimal service specification?
Application-specific functionality needed
Program sensor system as a whole
Provide a simple, centralized view of the system
Data as a single queriable unit
Declarative (database-centric) approach
Be the Google/Ask Jeeves for live content
IrisNet uses XML: Self-describing tags, Hierarchy
© Phillip B. Gibbons

14. Data Organized as Logical Hierarchy
<State id=“Pennysylvinia”>
<County id=“Allegheny”>
<City id=“Pittsburgh”>
<Neighborhood id=“Oakland”>
<total-spaces>200</total-spaces>
<Block id=“1”>
<GPS>…</GPS>
<pSpace id=“1”>
<in-use>no</in-use>
<metered>yes</metered>
</pSpace>
<pSpace id=“2”> …
</Block>
</Neighborhood>
<Neighborhood id=“Shadyside”>
Example XML Hierarchy …
ID attributes are treated specially.
Hierarchy is automatically
partitioned among the OAs
© Phillip B. Gibbons

15. IrisNet’s Two-Tier Architecture
User
Two components:
SAs: sensor feed processing
OAs: distributed database
Query
Web Server
for the url
. . . OA
XML database
. . . SA
senselet
Sensor SA
senselet
Sensor SA
senselet
. . .
Sensornet
© Phillip B. Gibbons

16. SAs and Senselets: Further Details
Sensor SA
senselet
Each SA runs a C module that provides a common runtime environment for senselets
Senselets are arbitrary Linux binaries
Can leverage OpenCV image processing library
Example senselet: images to parking availability
Send updates to OAs
Multiple senselets can share the same sensor feed
Senselets are uploaded to running SAs
Deployment costs greatly reduced:
Services share same sensors and infrastructure
© Phillip B. Gibbons

17. Creating a New IrisNet Service
Image processing steps
FFFFEFF
Send to OA
Updates DB
Only 500 lines of new code
for Parking Space Finder
Extended code
(application-specific aggregation)
Senselet (program to filter sensor data)
Hierarchy (XML schema)
Front-end
Query with standard
DB language
OAs SA SA
© Phillip B. Gibbons

18. Outline Multimedia Sensor Systems: Opportunities & Challenges
IrisNet overview
IrisNet key features
Conclusions
© Phillip B. Gibbons

19. IrisNet Key Features Distributed data collection and storage
Support for XML queries for distributed DB
Transparently routing queries to data
Automatic data partitioning
Multi-camera calibration & Image stitching
Efficient and protected sharing of sensor nodes
Privacy features
Triggering and actuation
Novel fault tolerance schemes
© Phillip B. Gibbons

20. Blind Men and Elephant Problem
OA’s local DB contains some fragment of the overall service DB
Not a federation – A single XML document
Quickly determining which part of an (XPATH) query answer can be answered from an XML fragment is a challenging task, not previously studied
E.g., can this predicate be correctly evaluated?
Is the result returned from local DB complete?
Where can the missing parts be gathered?
© Phillip B. Gibbons

21. Query Evaluate Gather (QEG)
/NE/PA/Allegheny/Pittsburgh/(Oakland | Shadyside) / rest of query
Pittsburgh OA Q
1. Queries its XML DB
Discovers Shadyside data
is cached, but not Oakland
Does DNS lookup to find
IP addr for Oakland
2. Evaluate the result Q’
Q’: /NE/PA/Allegheny/Pittsburgh/Oakland/ rest of query
3. Gathers the missing data
by sending Q’ to Oakland OA
Oakland OA
QEG
Combines results & returns
© Phillip B. Gibbons

22. Helping the Blind Men Tag the data
[Sigmod’03]
Tag the data
IrisNet tags the nodes in its fragment with status info, indicating various degrees of completeness
Maintains partitioning/tagging invariants
E.g., when gather data, generalize subquery to fetch partitionable units
For each missing part, construct global name from its id chain & do DNS lookup as before
Specialize subquery to avoid duplications & ensure progress
© Phillip B. Gibbons

23. IrisNet’s Query Processing (cont)
XPATH query converted to an XSLT program that walks the local XML document & handles the various tags appropriately
Conversion done without accessing the DB
IrisNet supports all of XPATH 1.0
Supports (distribution transparent) continuous queries
& application-specific aggregates, predicates, states
(Technically, the unordered projection of XPATH)
© Phillip B. Gibbons

24. IrisNet Key Features Distributed data collection and storage
Support for XML queries for distributed DB
Transparently routing queries to data
Automatic data partitioning
Multi-camera calibration & Image stitching
Efficient and protected sharing of sensor nodes
Privacy features
Triggering and actuation
Novel fault tolerance schemes
© Phillip B. Gibbons

25. Adaptive Data Partitioning & Placement
Distributed Storage
Distributed Query Processing
(Internet)
Distributed Storage
Partitioning
Replication
Placement …
For robustness and performance
Goal: Automate these data-management tasks
© Phillip B. Gibbons

26. Desired Behavior Partitioning + Placement 2. Replication + Placement
Query
Query
Query
Partitioning + Placement
2. Replication Placement
Query
Query
Query
Query
© Phillip B. Gibbons

27. Problem Statement Given Hierarchical databases Machine capacities
Data partitioning
Replication factor
Data placement
Determine
Cluster
based on access locality
Write-heavy data => few replicas
Near data
and
query sources
Query latency
Query traffic
Update traffic
Optimize
© Phillip B. Gibbons

28. IrisNet Data Placement algorithm
[FAST’05]
Distributed Algorithm using locally collected stats
E.g., link use frequency (hierarchy), read/write ratio, GNP coordinates of readers & writers
Balances desire to cluster vs. placing near sources
Replica selection probability ≈ inversely proportional to recent response time
Fast response to flash-crowds
Reacts quickly
Accounts for delay between off-load decision & corresponding load decrease
Approximates globally optimal solution
© Phillip B. Gibbons

29. IDP Adapting to Flash Crowds
Real deployment, real workload, synthetic flashcrowd (real workload replayed 100x times faster)
Live PlanetLab deployment with real workload
© Phillip B. Gibbons

30. IrisNet Key Features Distributed data collection and storage
Support for XML queries for distributed DB
Transparently routing queries to data
Automatic data partitioning
Multi-camera calibration & Image stitching
Efficient and protected sharing of sensor nodes
Privacy features
Triggering and actuation
Novel fault tolerance schemes
© Phillip B. Gibbons

31. Auto-Calibrating Cameras
Automatic extrinsic calibration
Many, unattended cameras
External forces => out-of-band camera movement
Goal: Determine physical positions & orientations of cameras wrt the scene
Keypoint algorithm finds correspondences between cameras
Key points: rotation & scaling invariant, plentiful, distinctive
Calculate homographies & combine images
© Phillip B. Gibbons

32. Improved Key Point Algorithm
Threshold set manually to return 10 matches per image.
SIFT (4/10) PCA-SIFT (9/10)
[CVPR’04]
White solid – correct matches
Black dotted – incorrect matches
© Phillip B. Gibbons

33. PCA-SIFT In Action
© Phillip B. Gibbons

34. IrisNet Key Features Distributed data collection and storage
Support for XML queries for distributed DB
Transparently routing queries to data
Automatic data partitioning
Multi-camera calibration & Image stitching
Efficient and protected sharing of sensor nodes
Privacy features
Triggering and actuation
Novel fault tolerance schemes
© Phillip B. Gibbons

35. Network Monitoring using IrisNet
We have deployed IrisNet on the PlanetLab network
450+ nodes, 5 continents
“Sensor” feeds are CPU, memory & network measurements at each node
Demonstrates wide-area scaling, robustness features
© Phillip B. Gibbons

36. IrisNet’s Deployment on PlanetLab: Lessons Learned
Robustness, robustness, robustness
Platform heterogeneity => VM-based?
Load balancing
Processes die; connectivity flaky (on re-connect incorporate into running aggregate query)
Correlated failures:
replication scheme matters: used Signed Quorum Systems (SQS)
subtleties arise (can predict 99% of failure patterns and still not improve availability)
Importance of relative object assignment
© Phillip B. Gibbons

37. IrisNet’s Availability on PlanetLab
Fraction of available data objects in IrisStore
Fraction of
available
PlanetLab
nodes
Say: planetlab is the most unstable before major systems conferences
Naming in IrisStore is done via DNS. IrisStore has about 3500 objects. Objects are grouped into ~300 groups that can be named. Individual objects cannot be named. This means about 300 DNS entries. We maintain two DNS servers locally at CMU. When doing the experiments, the DNS client is at CMU as well.
The results do capture naming failures. However, because the DNS server and client is always up, there were no naming failures during the experiment. -
Hours before SOSP’05 deadline
7 replicas/object with SQS quorum size = 2
[NSDI’06]
© Phillip B. Gibbons

38. Outline Multimedia Sensor Systems: Opportunities & Challenges
IrisNet overview
IrisNet key features
Conclusions
© Phillip B. Gibbons

39. Pervasive Sensing is Emerging
Planetary-Scale Multimedia Sensor Systems
Beyond motes: wide-area, broadband, multimedia, multi-tasked
Many opportunities for new services
Many Challenges
Data & Query Scaling
System scaling (authoring, robustness)
System heterogeneity
Shared resource
© Phillip B. Gibbons

40. A Distributed Systems Nightmare
Planetary-scale Multimedia Sensor Systems are:
Write-intensive, time-critical, high volume, mobile, heterogeneous, shared, subject to harsh conditions, include both sensing & actuation
Continuous/Global/aggregating queries (macroscope)
Programmed by many developers for many applications
…dynamically changing mix
…sensors shared by many applications
Must hide this complexity from application developers
© Phillip B. Gibbons

41. IrisNet: Architecture & Prototype
User
Two components:
SAs: sensor feed processing
OAs: distributed database
Query
Web Server
for the url
. . .
open source code,
papers OA
XML database
. . . SA
senselet
Sensor SA
senselet
Sensor SA
senselet
. . .
Sensornet
© Phillip B. Gibbons

42. IrisNet Team Phil Gibbons [project leader, databases, Feb02- ]
Rahul Sukthankar [computer vision, robotics, Jan03- ]
Babu Pillai [real-time systems, Sep03- ]
Jason Campbell [robotics, Oct03- ]
Haifeng Yu [distributed systems, Dec03- ]
Brad Karp [networking, Nov02-Nov03]
+ many valuable suggestions from M.Satyanarayanan
Srini Seshan [networking, systems, Mar02- ]
Suman Nath [grad student, May02-Aug05 ]
Yan Ke [grad student, Sep02-Apr04 ]
Summer interns: Amol Deshpande [Berkeley], Shimin Chen [CMU], Dilip Sundarraj [Portland St], Amit Manjhi [CMU],…
Intel Research
Pittsburgh
Carnegie Mellon
University
© Phillip B. Gibbons

43. Backup Slides
© Phillip B. Gibbons

44. Coastal Imaging using IrisNet
Working with oceanographers at Oregon State
Process coastline video to detect & analyze sandbar formation and riptides, etc
Images from IrisNet prototype
© Phillip B. Gibbons

45. Query-based Image Stitching
Stitch images from multiple cameras
What does the LA-region coast look like today?
How is “this” rip current going to progress in the next 24 hours?
Use “Key Points” to automatically find
how images fit together (planar scene) + =
© Phillip B. Gibbons

46. Disconnection-Tolerant Query Processing
Intermittent sensor disconnections
Wireless link failures
Internet delays to remote sites
Power-saving duty cycles
Mobile nodes out-of-range
Why is this important ?
Disconnected sensors still collecting data
Data resides only on that node: can’t replicate yet
Weakest availability link in system
Query responses based on stale data can be problematic
For instance: In Oceanography,
Unable to talk to tidal sensor
Unable to get images from SAs involved in image stitching
NETWORK SLEEP
© Phillip B. Gibbons

47. Advantage of Disconnection-Tolerant QP
© Phillip B. Gibbons

48. Relative Object Assignment
ABCD
EFGH
ABCD
EFGH
GHAB
CDEF
NO, Left better
Do they give us the same availability?
How large is the difference?
Over 2 nines for TPC-H
© Phillip B. Gibbons

49. Synopsis of Related Work
Sensor Networks
Specialized local sensor networks , e.g., home security
Satellite-based data collection
Mote-based: local network of sensors with small CPUs
Large-Scale Distributed Databases
E.g., PIER (P2P DB), SQL transactions, XML collections
Multi-Camera Systems & Algorithms
E.g., VSAM – study detecting & tracking objects
E.g., Lee at al [PAMI], Stauffer & Tieu [CVPR] – calibration
© Phillip B. Gibbons

50. Logging & Replay of Queries
Leaf OA detects loss of connection to SA
Queries and partial results are logged locally
Partial results sent based on user preference
Query is replayed only along disconnected branches
Leaf OA
Query log
SA-1
SA-2
Partial results
Disconnected Mode
Leaf OA
Replay
SA-1
SA-2
Query log
Partial results
Reconnected Mode
© Phillip B. Gibbons

51. More recent data available at ‘t’
Temporal Repair
Temporally Consistent Complete Result
QUERY:
Stitch latest
complete set
of images
(arrives at ‘t’) 1
Latest complete
set of images at
‘t-x’ 2 3 3 4 5
Consensus protocol : Fetch timestamp vectors from each leaf OA. At the LCA determine complete set that satisfies an epsilon window
Image at ‘t’
unavailable
At ‘t-x’
At ‘t’
At ‘t-x’
At ‘t-x’
At ‘t’
Most recent
More recent data available at ‘t’
© Phillip B. Gibbons

52. Actuation Challenges Competing actuation: shared resource
Economic models
Multiple cameras, set of requests, fidelity based on resolution & angle
Real-time control
Control Theory
Planning/Coordinating the actuation
Multiple ways to accomplish the same goal
Declarative API
Query-actuated data collection
© Phillip B. Gibbons