Automated Intruder Tracking using Particle Filtering and a Network of Binary Motion Sensors Jeremy Schiff EECS Department University of California, Berkeley Ken Goldberg IEOR and EECS Departments University of California, Berkeley Supported by NSF Grants: /

Outline Introduction Introduction Related Work Related Work Problem Formulation Problem Formulation Setup and Assumptions Setup and Assumptions Particle Filtering Particle Filtering Results Results Simulation Simulation Experimental Experimental Conclusion/Future Work Conclusion/Future Work

Motivation New class of technologies due to 9/11 New class of technologies due to 9/11 Automated Security Automated Security Wireless Sensor Networks Wireless Sensor Networks X10 PIR sensors - $25 X10 PIR sensors - $25 Robotic Webcams Robotic Webcams Pan, Tilt, Zoom Pan, Tilt, Zoom 500 Mpixels/Steradian 500 Mpixels/Steradian Increased computer processing speeds Increased computer processing speeds Enables Realtime Applications Enables Realtime Applications

Goal and Approach Wish to secure an environment Wish to secure an environment Low Cost Binary Sensors Low Cost Binary Sensors X10 ~ $25 X10 ~ $25 Optical Beam Optical Beam Floor Pad Floor Pad Manufactured in China Manufactured in China Noisy triggering pattern Noisy triggering pattern Refraction Refraction Use sensor triggering patterns to accurately localize an intruder Use sensor triggering patterns to accurately localize an intruder

Intuition Utilize Sensor Overlap Information Utilize Sensor Overlap Information

Intuition

Outline Introduction Introduction Related Work Related Work Problem Formulation Problem Formulation Setup and Assumptions Setup and Assumptions Particle Filtering Particle Filtering Experiments Experiments Simulation Simulation Real-world Real-world Conclusion/Future Work Conclusion/Future Work

Related Work Pursuer/Evader Games Pursuer/Evader Games Using line-of sight optical sensors Using line-of sight optical sensors [Isler, Kannan, Khanna 2004] [Isler, Kannan, Khanna 2004] Tracking Multiple Intruders Tracking Multiple Intruders [Oh, Sastry 2005] [Oh, Sastry 2005] Tracking Worn Devices Tracking Worn Devices Track Infrared Beacon Track Infrared Beacon [Shen et al. 2004] [Shen et al. 2004] Dynamic Shipment Planning using RFIDs Dynamic Shipment Planning using RFIDs [Kim et al. 2005] [Kim et al. 2005]

Related Work II Video Tracking Systems Video Tracking Systems [Micilotta and Bowden 2004] [Micilotta and Bowden 2004] Multiple Classes of Sensors Multiple Classes of Sensors Multiple exclusive modes Multiple exclusive modes [Cochran, Sinno, Clausen 1999] [Cochran, Sinno, Clausen 1999] Fuse data of multiple sensor types Fuse data of multiple sensor types [Jeffery et al. 2005] [Jeffery et al. 2005] Automated Camera Control Automated Camera Control [Song et al. 2005] [Song et al. 2005] Physical Devices Virtual Devices

Related Work III Probabilistic Tracking Approaches Probabilistic Tracking Approaches Kalman Filtering Kalman Filtering [Kalman 1960] [Kalman 1960] Extended Kalman Filtering Extended Kalman Filtering [Lefebvre, Bruyninckx, De Schutter 2004] [Lefebvre, Bruyninckx, De Schutter 2004] Particle Filtering Particle Filtering Book: [Thrun, Burgard, Fox 2005] Book: [Thrun, Burgard, Fox 2005] [Arulampalam et al. 2002] [Arulampalam et al. 2002]

Related Work IV Multiple humans controlling a camera Multiple humans controlling a camera [Song and Goldberg 2003] [Song and Goldberg 2003] [Song, Goldberg and Pashkevich 2003] [Song, Goldberg and Pashkevich 2003] Panorama Generation Panorama Generation [Song et al. 2005] [Song et al. 2005] Art Gallery Problem Art Gallery Problem [Shermer 1990] [Shermer 1990] [Urrutia 2000] [Urrutia 2000]

Outline Introduction Introduction Related Work Related Work Problem Formulation Problem Formulation Setup and Assumptions Setup and Assumptions Particle Filtering Particle Filtering Experiments Experiments Simulation Simulation Real-world Real-world Conclusion/Future Work Conclusion/Future Work

Setup and Assumptions Room Geometry Room Geometry List of nodes and edges List of nodes and edges Discretize space Discretize time

Setup and Assumptions II Intruder occupied world- space cell j in iteration Intruder occupied world- space cell j in iteration Sensor i triggered during iteration Sensor i triggered during iteration Sensor i experienced refraction period in iteration

Setup and Assumptions III Three Conditional Distributions Three Conditional Distributions Trigger while experiencing refraction Trigger while experiencing refraction Trigger from intruder Trigger from intruder Trigger from no intruder Trigger from no intruder

Output Estimated intruder location Estimated intruder location Objective: Objective: Minimize error between ground truth and estimation. Minimize error between ground truth and estimation.

Characterization Per sensor type Per sensor type Grid over sensor space Grid over sensor space Determine Determine Refraction period Refraction period False Negative Rate False Negative Rate False Positive Rate False Positive Rate

Deployment Convert to world-space Convert to world-space Overlay grid Overlay grid Transformed point to Cells Transformed point to Cells

Deployment II Determine potential non-zero characterization cells via convex hull Determine potential non-zero characterization cells via convex hull Inverse Distance Weighting Inverse Distance Weighting Interpolation according to distance Interpolation according to distance Determines values for cells without readings inside convex hull Determines values for cells without readings inside convex hull

Particle filters Non-Parametric Non-Parametric Sample Based Method (Particles) Sample Based Method (Particles) Particle Density ~ Likelihood Particle Density ~ Likelihood Tracking requires three distributions Tracking requires three distributions Initialization Distribution Initialization Distribution Transition Model (Intruder Model) Transition Model (Intruder Model) Observation Model Observation Model Determines Determines

Example

Example

Intruder Model State State Position, Orientation, Speed, and Refracting Sensors Position, Orientation, Speed, and Refracting Sensors Euler Integration for position Euler Integration for position Gaussian Random Walk for new speed and orientation Gaussian Random Walk for new speed and orientation Orientation change inversely proportional to speed Orientation change inversely proportional to speed Deterministic refraction periods Deterministic refraction periods Rejection Sampling to enforce room geometry Rejection Sampling to enforce room geometry

Intruder Model II Time between iterations: Time between iterations: Empirically determined constants: Empirically determined constants:

Intruder Model - Example Example state at iteration 0

Intruder Model - Example Accepted state for iteration 1

Intruder Model - Example Example state at iteration 1

Intruder Model - Example Accepted state for iteration 2

Intruder Model - Example Example state at iteration 2

Intruder Model - Example Rejected state for iteration 2

Intruder Model - Example Example state at iteration 2

Intruder Model - Example Rejected state for iteration 2

Intruder Model - Example Example state at iteration 2

Intruder Model - Example Accepted state for iteration 2

Sensor Model Evidence is vector of which sensors are triggering Evidence is vector of which sensors are triggering Triggering of sensors independent given intruder state implies Triggering of sensors independent given intruder state implies If sensor refracting If sensor refracting Otherwise Otherwise

Outline Introduction Introduction Related Work Related Work Problem Formulation Problem Formulation Setup and Assumptions Setup and Assumptions Particle Filtering Particle Filtering Experiments Experiments Simulation Simulation Real-world Real-world Conclusion/Future Work Conclusion/Future Work

Simulation Setup 22 Optical Beams 22 Optical Beams Perfect Perfect Optimal Performance Optimal Performance 14 Triangular Motion Sensor Perfect & Imperfect

Simulation Results Example Path Example Path Ground Truth Ground Truth Red Circles Red Circles Estimations Estimations Grey Circles Grey Circles

Simulation Results II Baseline Estimate Baseline Estimate Perfect Optical-Beam Sensors Perfect Optical-Beam Sensors P(E) Error E P(E)

Perfect Triangular Motion Sensors Perfect Triangular Motion Sensors Imperfect Triangular Motion Sensors Imperfect Triangular Motion Sensors Simulation Results III P(E) Error E P(E)

Error over Time – 4 Sec. Refraction, Imperfect Sensors Error over Time – 4 Sec. Refraction, Imperfect Sensors Density - 8 Sec. Refraction, Imperfect Sensors Density - 8 Sec. Refraction, Imperfect Sensors Simulation Results IV Error E Time (Seconds) Error E P(E)

In-Lab Results 8 Passive Infrared Sensors 8 Passive Infrared Sensors X10 X10 8 second refraction time 8 second refraction time Room 8x6 meters Room 8x6 meters.3 m /Cell dimension.3 m /Cell dimension Sampled every 2 seconds Sampled every 2 seconds 1000 Particles 1000 Particles

In-Lab Results II

Outline Introduction Introduction Related Work Related Work Problem Formulation Problem Formulation Setup and Assumptions Setup and Assumptions Particle Filtering Particle Filtering Results Results Simulation Simulation Experimental Experimental Conclusion/Future Work Conclusion/Future Work

Conclusions Real-time Tracking System Real-time Tracking System Binary Sensors with Refraction Period Binary Sensors with Refraction Period Particle Filtering for Sensor Fusion Particle Filtering for Sensor Fusion Conditional Probability Models Conditional Probability Models Models Models Intruder Velocity Intruder Velocity Room Geometry Room Geometry Sensor Characterization Sensor Characterization

Future Work Effects of varying different components Effects of varying different components Number Particles Number Particles Types of sensors Types of sensors Spatial arrangements of sensors Spatial arrangements of sensors Multiple intruders Multiple intruders Decentralize Decentralize Vision Processing Vision Processing Other applications Other applications Warehouse Tracking Warehouse Tracking

Thank You Jeremy Schiff: Jeremy Schiff: Ken Goldberg: Ken Goldberg: URL: URL: