Automated Motion Imagery Data Exploitation for Airborne Video Surveillance CGI Video Research Team Dr. He, Zhihai (Henry) - Electrical Engineering Dr. Palaniappan, Kannappan - Computer Science Dr. DeSouza, Guilherme N. - Electrical Engineering Dr. Duan, Ye - Computer Science

Outline Background and Project Overview Simulation environment setup and test video sets Moving objects detection and geo-location Video registration and moving objects tracking 3-D urban scene modeling from UAV videos Conclusion and discussion

? Background and Motivation - Airborne Surveillance Videos Massive (hundreds of hours of videos) A cognitive disaster for human analysts Need to develop algorithms to aggregate, filter, fuse, and summarize video data and extract important information. ? Video curtsey to AFRL Reviewed by human within an hour Automated Video Processing 100s’ hours of videos

Hierarchical Automated Motion Imagery Data Exploitation Long-Term Plan Data Visualization Decision Activity Mining and Abnormality Detection Knowledge Spatiotemporal Characterization of Objects Activity Moving Object Extraction and Geo-location Object Registration, Fusion, and Super-resolution Data Motion Imagery Data Signal

Fusion and Super-resolution Video Summarization and Activity Visualization Long-Term Plan Fusion and Super-resolution & Visualization Object Browsing Search for Objects of Interest Objects Information Database Motion Imagery Data Search A red vehicle moving southeast at a speed over 80 miles per hour at about 3:00pm on March 26th of 2006.

This Project / First Year 2006-2007 3-D scene modeling Trajectory extraction Multi-object tracking Geo-location Real-time video registration and mosaic

Meta data of in common format Research and Development Plan Preparing test video datasets Algorithm development and refinement Test in simulation environment Evaluate with flight test videos. Code optimization, speedup, documentation and transfer. Input Videos Motion Imagery Desktop Toolset Meta data of in common format

Preparing the Test Video Datasets Simulation test bed In-house video data collection Third-party test video datasets

UAV Simulation Setup Provide ground truth for Moving object detection and tracking Geo-location, speed, trajectory estimation of moving objects 3-D scene reconstruction

UAV Simulation Setup – A Close Look Need to Adjust the relative size of objects Add more structure and background texture.

In-House Aerial Video Collection Balloon UMC UAV Easy access High-quality video scenes with multiple moving objects No ground truth / metadata

Third-Party Test Video Sets DARPA test videos (no meta data) Crystal view test videos (with meta data) AFRL flight test videos (no meta data, few moving objects)

Two Scenarios High-Altitude Videos Dominant global camera motion Local moving objects – a small portion of the scene (< 20%) Ground object structure – negligible. Low-Altitude Videos Scene content change due to 3-D structure of ground objects (parallax) is significant. Local moving objects become a significant part of the scene.

Moving Object Detection and Geo-location

Reconnaissance and Surveillance in Urban Environments Limited visibility and resolution Complex and cluttered scenes High background activity

Motion Analysis and Detection of Regions of Interest (ROIs) (a) (b) Figure - Optical Flow, before (a) and after (b) removal of background motion (but before filtering of spurious flows in the image).

Motion Analysis and Detection of ROI’s Remove background motion and determine the dominant component in the foreground (a) (b) Figure – Histogram analysis of the Optical Flow. (a) Magnitudes and (b) Angles.

Correlation-based Tracking (a) (b) Figure – ROI (in red) highlighting the tracked object.

Preliminary Results

Preliminary Results

DARPA Sequence: Hollywood

Multi-Object Tracking with Optical Flow Analysis Note: Need further improvement, especially with fusion with the registration-based tracking technology to be presented next.

Video Registration and Multi-Object Tracking

Real-Time Registration of Videos Estimate the global camera motion parameters. Warp video frames into the same coordinate system – mosaic

Real-Time Registration of Videos Vehicle-Camera Motion Translational motion, Rotation, zoom, perspective change. Global Motion Equation [x, y] Frame n [X, Y] Frame n+1 after camera motion Need to estimate 8 parameters Theoretically, we only need to 8 equations (8 point-to-point correspondence)

Real-Time Registration of Videos Design Goals Generic – making no assumption about image content. The registration algorithm works in a wide range of environments. Robust to noise and errors. Low-complexity for real-time computation

Real-Time Registration of Videos Static Objects Dominant Motion Moving Objects Local Motion Structural Blocks Texture Blocks ……… With different reliability levels Camera Motion Estimation

Moving Objects Detection The motion of moving objects does not satisfy the global motion equation. Moving object can be detected based on this equation using hypothesis testing. Assumption Global motion estimation The moving objects are not a significant portion (<30%) of the video scene. Moving object detection and removal

Registration Result

Multi-Object Tracking 1. Detect moving objects in stabilized frames. 2. Predict locations of the current set of objects. 3. Match predictions to actual measurements. 4. Update object trajectories. 5. Update image stabilized ref coord system. Multi-object Detection and Tracking Unit Tracking VGoF Registration Into Common Coordinate System Moving Object Detection & Feature Extraction Data Association (Correspondence) Update Trajectories Object States Context Prediction Update Coord System

Dynamic State Estimation for Tracking System state Measurements State estimate Dynamic System Measurement System State Estimator State uncertainties System noise Measurement noise System Errors Agile motion Distraction/clutter Occlusion Changes in lighting Changes in pose Shadow (Object or background models are often inadequate or inaccurate) Measurement Errors Camera noise Framegrabber noise Compression artifacts Perspective projection State Error Position Appearance Color Shape Texture etc. Support map

Motion Detection - Structure and Flux Tensor Approach Typical Approach: threshold trace(J) Problem: trace(J) fails to capture the nature of gradient changes and results in ambiguities between stationary versus moving features Alternative Approach: Analyze the eigenvalues and the associated eigenvectors of J Problem: Eigen-decompositions at every pixel is computationally expensive for real time performance Proposed Solution: Flux tensor  time derivative of J

Motion Detection Flux Tensor vs Gaussian Mixture

Multi-object Tracking Stages Probabilistic Bayesian framework Features Used in Data Association: Proximity and Appearance-based Data Association Strategy: Multi-hypothesis testing Gating Strategies: Absolute and Relative Discontinuity Resolution: Prediction (Kalman filter), or Appearance models Filtering: Temporal consistency check and Spatio-temporal cluster check

Association Strategy Multi-hypothesis testing with delayed decision - Many matches are kept with evidence-based pruning Support for multiple interactions - one-to-one object matches, many-to-one, one-to-many, many-to-many, one-to-none, or none-to-one matches Corresponding low-level object tracking events Segmentation errors Group interactions (merge/split) Occlusion Fragmentation Entering object Exiting object ObjectMatchGraph

Exp Results: DARPA ET01 Video Frame #50 Registered Frame Motion Detection Results Foreground Mask Tracking Results

Registration and Tracking Results

Registration and Tracking Results

Registration and Tracking Results - Others

Registration and Tracking Results - Occlusion Before trajectory filtering After trajectory filtering

3-D Scene Reconstruction from UAV Videos

Multi-view Image based Modeling

Multi-View Image-Based Modeling Using Deformable Surfaces

3D Urban Scene Modeling Using Multi-view Aerial Imagery

3D Building Reconstruction Using Multi-view Aerial Images Processing Stages Feature exaction and matching Camera pose estimation Multi-view image-based modeling Texture mapped 3D urban scene visualization

User-Guided Feature Selection & Matching

Camera Calibration

3D Reconstruction

Texture mapping

Preliminary Results from Airborne Videos

Preliminary Results from Airborne Videos

R&D Tasks Completed During the Past 3 Months Improved the accuracy and robustness of optical flow estimation and moving object segmentation algorithm. Solved the problem of long-term registration and tracking over a large region using dynamic window and related data exchange and management problem. Developed a multi-path motion estimation and registration scheme to significantly improve the registration accuracy.

R&D Tasks Completed During the Past 3 Months Partially solved the long-term drifting error problem. Refine our multi-object tracking algorithm to deal with noise and 3-D structures. Improved the accuracy of feature matching and camera calibration for 3-D scene reconstruction.

Remaining R&D Tasks Need to compare the results of the geo-location algorithm against the ground truth and evaluate its performance Need to evaluate the performance of geo-location with test videos. Need to establishing trajectory continuity (object ID matching) across moving coordinate systems Customizing trajectory analysis for airborne video tracking with registration error, large platform motion, zooming, etc Need to solve the identification correspondence problem for tracking between segments. Morphological post processing filters

Remaining R&D Tasks Need to handle image noise, shadow, and 3-D structures. Need to evaluate the performance 3-D scene reconstruction using ground truth provided by our simulator. Visualization interface design (mosaic, trajectory, meta data). Code optimization and speedup with C/C++ Input / output interface design and data formats to convert our modules into desktop tools and transfer to NGA.

Conclusion and Discussion 3-D scene modeling Trajectory extraction Multi-object tracking Geo-location Real-time video registration and mosaic