Assisted Target Detection for Search & Rescue James Elder Department of Computer Science and Engineering Department of Psychology & Centre for Vision Research York University

Project Overview Improved sensing technologies have the potential to improve search and rescue performance. The degree of improvement depends on how accurately the data generated by these technologies are interpreted, either by computer or by the human visual system. The goal of the proposed project is to develop assisted target detection (ATD) algorithms that will improve human target detection rates by drawing visual attention to locations in the data stream where target probability is higher. The SAR simulator at DRDC Toronto will serve as a test-bed for evaluating the efficacy of all image enhancements. Project Start: Jan 1, 2006 Project Finish: OCE Funding expected to March 31, 2008

Project Team

York Search & Rescue ATD Team Researcher Position % Research Time Dr. James Elder PI 20% Vladimir Magdin M.Sc. Student 75% Dr. Bob Hou Senior Research Associate Alumni Researcher Position % Research Time Dr. Francisco Estrada Postdoctoral Fellow 10% Kevin Cannons PhD Student 20%

Industrial Partner Array Systems Computing Specializes in computationally intensive, real-time digital signal processing systems using multiple parallel processors. Provides high-performance software solutions in diverse application areas including sonar, radar and remote sensing, 3D modelling and simulation, and Software Defined Radio (SDR). Located in Toronto, close to DRDC Downsview and York University. Participants: Rajesh Jha Ray Obidowski Steven Truelove

Funding Projection Contributors: OCE cash $8 0,000 $8 0,000 $8 0,000 7 - Dec - 05 to 1 - Apr - 06 to 1 - Apr - 07 to Total of all 31 - Mar - 06 31 - Mar - 07 31 - Mar - 08 confirmed (confirmed) (propo sed) (proposed) and proposed c ontributions by sponsor : OCE cash $8 0,000 $8 0,000 $8 0,000 $24 0,000 Array cash $0 $0 $0 $ in - kind $ 68,400 $163,000 $163,000 $3 94,400 DRDC cash $ 50,000 $1 10 ,000 $ 80 ,000 $ 240 ,000 in - kind $20 9,000 $10,000 $10,000 $22 9,000 Total contributions by $ 4 5 2,400 $ 363 , 000 $ 333 ,000 $1, 148 ,400 project year:

Milestones Milestones Tasks Resources Start End Update Recruit personnel Recruit postdoctoral fellow Funding Nov 05 Jan 06 Delayed due to funding delay computer vision graduate student Fu nding Feb 06 V. Magdin started Sept Recruit psychophysics Mar Y . Morgenstern? Complete legal agreements Prepare and sign NDAs and IP agreements between York and Array, allowing sharing of source code. Time with institutional lawyers Dec Completed

Milestones Milestones Tasks Resources Start End Update Establish understan ding of problem and performance benchmarks Transfer C++ code for indoor surveillance to Array for integration and testing on GSP hardware. Array engineer Dec 05 Mar 06 Completed Review DND documents on SAR and AIMS, and review general literature on searc h - rescue Nov Obtain existing search rescue datasets with ground truth for system development and evaluation Existing DND database 5

Milestones Milestones Tasks Resources Start End Update Single - frame algorithm development and testing Develop and evaluate single frame st atistical target models. Dec 5 Jan 06 Completed frame adaptive background models. Feb Completed - Discuss Test frame by ATD algorithm in MATLAB on reserved flight data using ROC analysis Mar Evaluate statistical against AdaBoost methods Jan 07 In Progress Refine frame 07 Feb 07 Transfer MATLAB code to Array for integration and testing on GSP hardware Array engineer

Milestones Milestones Tasks Resources Start End Update Continuous algorithm development and testing Acquire new sear ch - and rescue datasets with ground truth. New DND database Apr 07 Mar Develop or adapt georeferencing algorithms for search rescue data streams. Georeferencing tools from AIMS project Jun 07 Incorporate motion cues and temporal integratio n into foreground and background models. Jun Jul 07 Test continuous ATD algorithm in MATLAB on reserved flight data using ROC analysis. Aug Refine continuous Sept Transfer MATLAB code to Array for integration and t esting on GSP hardware Array engineer Dec Complete report on ATD algorithms Oct07 Complete publication on ATD results

Milestones Milestones Tasks Resources Start End Update Develop and evaluate visualization methods Develop candidate for overlaying target probability data onto incoming visual data stream Dec 06 Mar 07 psychophysical methodology for evaluation using DRDC SAR Simulator Jun Conduct human - in the loop SAR simulator experiments DRDC simulator S ept Complete report on methodologies and results of SAR simulation experiments. Sept Complete publication on SAR simulation results 08

Milestones Milestones Tasks Resources Start End Update Final system integration and reporting Refine complete software system, incl uding ATD and visualization algorithms Dec 07 Jan 08 Transfer final MATLAB/C++ code to Array for integration and testing on GSP hardware Array engineer Jan 08 Feb 08 Complete final report on ATD system Feb Mar

Deliverables Assisted target detection algorithms Assisted target detection software (MATLAB and/or C++) A demonstration of ATD software integrated with Array Systems Scalable Generic Signal Processor technology for real-time operation. Results of human-in-the-loop experiments using the ATD technology and the DRDC SAR simulation system.

York Progress Report

Ground Truth Labeling Labeled over 30,000 objects Invested over 100 man-hours, split between 6 people Labeled portions of all 7 video sequences that contain targets

Ground Truth – Example Frame NIGHT EXERCISE_IR

Algorithm Evaluation Selected subset of ground truth data from a single IR video (Night Exercise) Not representative of search & rescue scenario Training set 100 frames 5000 +ve examples 5000 -ve examples Test set 185 frames 9000 targets

Algorithm Evaluation Hit Rate False Positives per Frame 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 5 10 15 20 False Positives per Frame Hit Rate Pixel-wise bayes Zero-mean Target Model Normal Model Static AdaBoost Linear SVM

Cross-Validation An effort is underway to evaluate the reliability of the ground truth data by cross-validating across human subjects.

Search and Rescue Simulator York is running a copy of the DRDC Search & Rescue simulator Do we need a Flybox? May need to buy higher-end computer.