Eric Grimson, Chris Stauffer, A Forest of Sensors: Using adaptive tracking to classify and monitor activities MIT AI Lab Eric Grimson, Chris Stauffer, Lily Lee, Raquel Romano, Gideon Stein 11/21/2018 Forest of Sensors

Eric Grimson, Chris Stauffer, A Forest of Sensors: Using adaptive tracking to classify and monitor activities MIT AI Lab Eric Grimson, Chris Stauffer, Lily Lee, Raquel Romano, Gideon Stein 11/21/2018 Forest of Sensors

A Forest of Sensors Given autonomous vision modules (AVMs): low power, low cost, can compute wide range of visual routines Create a forest of disposable AVMs: attached to trees, buildings, vehicles dropped by air Question: Can the forest bootstrap itself to monitor sites for activities? 11/21/2018 Forest of Sensors

Multi-camera version 11/21/2018 Forest of Sensors

Adaptive Background Model Pixel Consistent with Background Model? Video Frames Adaptive Background Model X,Y,Size,Dx,Dy Local Tracking Histories 11/21/2018 Forest of Sensors

Capabilities and components Self calibration building rough site models detecting visibility primitive detection moving object modeling activity detection activity calibration 11/21/2018 Forest of Sensors

Working hypothesis Can achieve all of these capabilities simply by accurately tracking moving objects in the scene. 11/21/2018 Forest of Sensors

A robust, real-time tracker Model each pixel as an independent process Model previous n samples with weighted mixture of Gaussians, using exponentially decaying time window Background defined as set of dominant models that account for T percent of data Update weights and parameters Pixel > 2 sigma from background is moving Select significant blobs as objects 11/21/2018 Forest of Sensors

Adaptive tracking 11/21/2018 Forest of Sensors

Dynamic calibration Track objects in multiple cameras use correspondences to find a homography, assuming planar motion modify homography by fitting image features use non-planar features to solve for epipolar geometry, and refine 11/21/2018 Forest of Sensors

Dynamic calibration 11/21/2018 Forest of Sensors

Site modeling N camera stereo from calibration dynamic tracking updates visibility detection and placement update reconstruction from multiple moving views 11/21/2018 Forest of Sensors

Site models by tracking 11/21/2018 Forest of Sensors

Site models by tracking 11/21/2018 Forest of Sensors

Site modeling from stereo Multi-camera stereo reconstruction 11/21/2018 Forest of Sensors

Stereo reconstruction 11/21/2018 Forest of Sensors

Activity detection & calibration Need to find common patterns in track data 11/21/2018 Forest of Sensors

Pattern tracks 11/21/2018 Forest of Sensors

Detect Regularities & Anomalies? 11/21/2018 Forest of Sensors

Example track patterns Running continuously for almost 1 year during snow, wind, rain, ... one can observe patterns over space and over time need a system to detect automatically 11/21/2018 Forest of Sensors

Classifying objects Simple classifiers based on feature selection Example -- people vs. cars using aspect ratio and size 11/21/2018 Forest of Sensors

Flexible template querying DATABASE QUERY IMAGES IN THE QUERY-CLASS Template match A B C 11/21/2018 Forest of Sensors

The Starting Point Sinha’s Ratio Template 11/21/2018 Forest of Sensors

Using this template 11/21/2018 Forest of Sensors

Build a Spatial Template R2 TEMPLATE UNITS A B C R1 R1 R2 A B C R1: A(red) < B(red) A(green) < B(green) A(lum) < B(lum) A(blue) > A(green) A(blue) > A(red) TEMPLATE R2: B(red) > C(red) B(green) > C (green) B(blue) > C(blue) B(lum) > C(lum) 11/21/2018 Forest of Sensors

Example Recognition 11/21/2018 Forest of Sensors

Example Detection 11/21/2018 Forest of Sensors

Example Classification 11/21/2018 Forest of Sensors

Example Results 11/21/2018 Forest of Sensors

Classifying objects Using flexible templates to detect vehicles 11/21/2018 Forest of Sensors

Classifying activities Quantize state space of continuous observations by fitting Gaussians map sequence of observables to sequence of labels compute co-occurrence of labels over sequences -- defines joint probability cluster by E/M methods to find underlying probability distributions 11/21/2018 Forest of Sensors

Automatic activity classification 11/21/2018 Forest of Sensors

Long-term Activity Patterns 11/21/2018 Forest of Sensors

Multi-camera version 11/21/2018 Forest of Sensors

Mapping patterns to maps 11/21/2018 Forest of Sensors

Multi-camera coordination 11/21/2018 Forest of Sensors

Finding unusual events 11/21/2018 Forest of Sensors

Finding similar events 11/21/2018 Forest of Sensors

Future plans integration of pieces site models from dynamic tracking activity classification from multiple views variations on classification methods vocabulary of activities and interactions interactions of activity tracks fine scale actions 11/21/2018 Forest of Sensors