Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring Ken Goldberg and Dezhen Song (Paul Wright and Carlo Sequin) Alpha Lab, IEOR and EECS University of California, Berkeley

Networked Robots

internet tele-robot:

RoboMotes: Gaurav S. Sukhatme, USC

Smart Dust: Kris Pister, UCB (Image: Kenn Brown)

Networked Cameras

Security Applications Banks, Airports, Freeways, Sports Events, Concerts, Hospitals, Schools, Warehouses, Stores, Playgrounds, Casinos, Prisons, etc.

Conventional Security Cameras Immobile or Repetitive Sweep Low resolution

New Video Cameras: Omnidirectional vs. Robotic Fixed lens with mirror 6M Pixel CCD $ 20.0 K 1M Pixel / Steradian Pan, Tilt, Zoom (21x) 0.37M Pixel CCD $ 1.2 K 500M Pixel / Steradian

Where to look?

Sensor Net Detects Activity “Motecams” Other sensors: audio, pressure switches, light beams, IR, etc Generate bounding boxes and motion vectors Transmit to Robot camera Activity localization

Block Motion Estimator Ron Fearing’s Xilinx FPGA board can compute motion vectors in hardware Extract: bounding frames

Viewpoint Selection Problem Given n bounding frames, find optimal frame

Related Work Facilities Location –Megiddo and Supowit [84] –Eppstein [97] –Halperin et al. [02] Rectangle Fitting –Grossi and Italiano [99,00] –Agarwal and Erickson [99] –Mount et al [96] Similarity Measures –Kavraki [98] –Broder et al [98, 00] –Veltkamp and Hagedoorn [00]

Problem Definition Requested frames :  i =[x i, y i, z i ], i=1,…,n

Problem Definition Assumptions –Camera has fixed aspect ratio: 4 x 3 –Candidate frame  = [x, y, z] t –(x, y)  R 2 (continuous set) – z  Z (discrete set) (x, y) 3z 4z

Problem Definition “Satisfaction” for frame i: 0  S i  1 S i = 0 S i = 1  =    i  =  i

Symmetric Difference Intersection-Over-Union Similarity Metrics Nonlinear functions of (x,y)

Intersection over Maximum: Requested frame  i, Area= a i Candidate frame  Area = a Satisfaction Metrics pipi

Intersection over Maximum: s i ( ,  i ) s i = Requested frame  i Candidate frame 

Satisfaction Function – s i (x,y) is a plateau One top plane Four side planes Quadratic surfaces at corners Critical boundaries: 4 horizontal, 4 vertical (for fixed z)

Objective Function Global Satisfaction: for fixed z Find  * = arg max S(  )

S(x,y) is non-differentiable, non-convex, but piecewise linear along axis-parallel lines. Properties of Global Satisfaction

Approximation Algorithm x y d Compute S(x,y) at lattice of sample points:

Approximation Algorithm –Run Time: –O(w h m n / d 2 )  * : Optimal frame : Optimal at lattice : Smallest frame at lattice that encloses  *

Exact Algorithm Virtual corner: Intersection between boundaries –Self intersection: –Frame intersection : y x

Exact Algorithm Claim: An optimal point occurs at a virtual corner. Proof: –Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur at boundaries

Exact Algorithm Exact Algorithm: Check all virtual corners  (mn 2 ) virtual corners  (n) time to evaluate S for each  (mn 3 ) total runtime

Improved Exact Algorithm Sweep horizontally: solve at each vertical –Sort critical points along y axis: O(n log n) –1D problem at each vertical boundary O(nm) –O(n) 1D problems –O(n 2 m) total runtime O(n) 1D problems

Examples

Summary Networked robots High res. security cameras Omnidirectional vs. robotic Motion Sensing Network Viewpoint Selection Problem Algorithms

Future Work Dynamic Version with motion prediction Multiple outputs: –p cameras –p views from one camera “Temporal” version: fairness –Integrate s i over time: minimize accumulated dissatisfaction for any frame request Obstacle Avoidance

Related Work Facility Location Problems –Megiddo and Supowit [84] –Eppstein [97] –Halperin et al. [02] Rectangle Fitting, Range Search, Range Sum, and Dominance Sum –Friesen and Chan [93] –Kapelio et al [95] –Mount et al [96] –Grossi and Italiano [99,00] –Agarwal and Erickson [99] –Zhang [02]

Related Work Similarity Measures –Kavraki [98] –Broder et al [98, 00] –Veltkamp and Hagedoorn [00] Frame selection algorithms –Song, Goldberg et al [02, 03, 04], –Har-peled et al. [03]

Problem Definition Assumptions –Camera has fixed aspect ratio: 4 x 3 –Candidate frame c = [x, y, z] t –(x, y)  R 2 (continuous set) – Resolution z  Z Z = 10 means a pixel in the image = 10×10m 2 area Bigger z = larger frame = lower resolution (x, y) 3z 4z

Problem Definition Requests : r i =[x l i, y t i, x r i, y b i, z i ], i=1,…,n ( x l i, y t i ) ( x r i, y b i )

Optimization Problem User i’s satisfaction Total satisfaction

Problem Definition “Satisfaction” for user i: 0  S i  1 S i = 0 S i = 1  = c  r i c = r i

Measure user i’s satisfaction: Coverage-Resolution Ratio Metrics Requested frame r i Area= a i Candidate frame c Area = a pipi

Comparison with Similarity Metrics Symmetric Difference Intersection-Over-Union Nonlinear functions of (x,y), Does not measure resolution difference

Optimization Problem

Requested Frame r i Candidate Frame c (for fixed z) Objective Function Properties

s i (x,y) is a plateau One top plane Four side planes Quadratic surfaces at corners Critical boundaries: 4 horizontal, 4 vertical Objective Function for Fixed Resolution 4z x y 3z 4(z i -z)

Objective Function Total satisfaction: for fixed z Frame selection problem: Find c * = arg max S(c)

S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines. Objective Function Properties 4z x y 3z 4(z i -z) 3z y sisi (z/z i ) 2 3(z i -z) x sisi 4z (z/z i ) 2 4(z i -z)

Plateau Vertex Definition Intersection between boundaries –Self intersection: –Plateau intersection : y x

Plateau Vertex Optimality Condition Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof: –Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur at x boundaries y S(y)S(y)

Fixed Resolution Exact Algorithm Brute force Exact Algorithm: Check all plateau vertices  (n 2 ) plateau vertices  (n) time to evaluate S for each  (n 3 ) total runtime

Improved Fixed Resolution Algorithm Sweep horizontally: solve at each vertical –Sort critical points along y axis: O(n log n) –1D problem at each vertical boundary O(n) –O(n) 1D problems –O(n 2 m) total runtime for m zoom levels O(n) 1D problems y S(y)S(y) x y

A New Architecture Activity and Video database Activity localization Activities Frame selection Active surveillance Control commands Videos

Software diagram TCP/IP Activity & video database Core (with shared memory segments) RPC module Communication Console/Log Activity server Activity generation Motescam Wireless Camera control Calibration Panoramic image generation Video server Panasonic HCM 280 Camera Visual C++ NesC + Tiny OS Gnu C++ MySQL

Database: indexing video data Activity Index –Timestamp –Speed (Or other sensor data) –Range Query video data using activity –Show video clips of moving objects with speed faster than 1 meter per second in zone 1 in last 10 days –Show video clips of zone 1 when CO 2 concentration exceeded the threshold in Jan (Assuming CO 2 sensor is used in detecting activity) Activity and Video database