[1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics Laboratory/Utah State University Presented at NATO IST-036/RWS-005 Halden, Norway September 2002

[2] Concept of Pixel-Level Fusion Active Sensor – LADAR Passive Sensor – Electro-Optic (EO) Radiation Fused into One Data Set RANGE DATA FROM MULTI- CHANNEL LADAR PASSIVE IR IMAGE DATA + FUSED RANGE IMAGE

[3] Enables Observation Through Partial Obscuration

[4] Employment Concept Individual Pixel r2r2 r3r3 r4r4 r1r1 Individual Pixel r1r1 r2r2 r3r3 r4r4 r5r5 Cloud Layers Ground Clutter Upper Canopy Lower Canopy

[5] Fused LADAR/EO Image 1 (Co-Registered and Georeferenced at the Pixel Level) LADAR Image EO Image

[6] LADAR Image EO Image Fused LADAR/EO Image 2 (Co-Registered and Georeferenced at the Pixel Level)

[7] Orthoprojection of the Two 3D Images Combined in Geographic Space First Image Second Image

[8] Profile Views in 3D Viewer First LADAR/EO Image First and Second LADAR/EO Images Combined

[9] Visualization Concept - Data Volume Exploration and Clutter Removal

[10] 3D Images of Cluttered Environment Image Plane Vegetation Canopy Ground Plane Terrain (per DTED or detected ground)

[11] Predict Ground Image Plane Bottom of Canopy Layer (Terrain + 3m) Ground Plane Terrain (per DTED or detected ground)

[12] Slice Volume at Bottom of Canopy Image Plane Bottom of Canopy Layer (Terrain + 3m) Ground Plane Terrain (per DTED or detected ground) Residual Shadowed Regions: P D  0

[13] Example of Canopy Slicing Orthoprojected Using LADAR – Trees Removed Oblique EO Scene Residual Shadowed Region: P D  0 Ground with Trees Removed

[14] Two Shots Combined – Trees Removed Residual Shadowed Region: P D  0 Corresponding EO Image

[15] Target Hidden Under Tree Hidden Target

[16] Target Exposed After Tree Removal Exposed Target

[17] Conclusions (1): Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction & Visualization Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments Clutter – tree canopies, clouds, etc. – can be removed through geometric filters

[18] Visualization Implication

[19] Some Visualization Implications Target Detection & Recognition >Probabilities of Detection >Probabilities of Target Presence (prior & posterior) >Exploitation of Negative Data Temporal Evolution of Estimation and Expectations >Out-of Sequence Data >Situation Projection >Expected Updates

[20] Prior & Posterior Detection Probabilities; e.g. Presentation of Negative Data Target Detectability Map – – 0.75 p D (x,t+n|t,A) Prior or Posterior Probability Density (e.g. Track Projection Uncertainty) x = Target state (type & location, etc.) t, t+n = Update times A = Action plan (sensor route, sensor controls, processing controls, etc.) p(x,t+n|t) Expectations Map Likely false alarm Likely missed detections p(x,t+n|t)p D (x,t+n|t,A)

[21] Some Visualization Implications Target Detection & Recognition >Probabilities of Detection >Probabilities of Target Presence (prior & posterior) >Exploitation of Negative Data Temporal Evolution of Estimation and Expectations >Out-of Sequence Data >Situation Projection >Expected Updates

[22] Revised Estimates of Track Histories (MTI + Imagery) Time ELEMENTARY SCHOOL MISSILE ASSEMBLY FACILITY MTI Reports Imagery Reports Latency Issues: Out of Sequence Data Initial Estimates of Track Histories (MTI only) Time ELEMENTARY SCHOOL MISSILE ASSEMBLY FACILITY

[23] Adaptive Information Exploitation Process Acknowledge uncertainties Estimate length of time for which decision can be deferred Estimate probability of resolving ambiguity in time >Given current collection plan >By redirecting current resources >By adding resource Estimate Cost & Net Utility of Candidate Action Plans Select & Implement Revised Plan

[24] Estimated HistoryEstimated Present SituationProjection Situation Situation Time Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Visualizing Time-Varying Situation Estimation & Expectation (1 of 3) Current Estimate of Present Situation Report Time (Plan A)

[25] Estimated HistoryEstimated Present SituationProjection Situation Situation Time Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Visualizing Time-Varying Situation Estimation & Expectation (2 of 3) Current Projection of Future Situation Report Time (Plan A)

[26] Visualizing Time-Varying Situation Estimation & Expectation (3 of 3) Report Time (Plan A) Estimated HistoryEstimated Present SituationProjection Situation Situation Time Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Expected Update of Present Situation

[27] Conclusions Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments >Clutter – tree canopies, clouds, etc. – can be removed over time through geometric filters Enables visualization of time-varying situation estimation & expectation >Presentation of Detection Probabilities (prior & posterior): e.g. Presenting Negative Data >Presentation of Temporal Evolution of Estimates and of Expectations –Out-of Sequence Data –Situation Projection –Expected Updates & Information Evolution

[28] Thank You Mange Takk!