1. [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics Laboratory/Utah State University Presented at NATO IST-036/RWS-005 Halden, Norway 10-13 September 2002

2. [2] Concept of Pixel-Level Fusion Active Sensor – LADAR Passive Sensor – Electro-Optic (EO) Radiation Fused into One Data Set RANGE DATA FROM MULTI- CHANNEL LADAR PASSIVE IR IMAGE DATA + FUSED RANGE IMAGE

3. [3] Enables Observation Through Partial Obscuration

4. [4] Employment Concept Individual Pixel r2r2 r3r3 r4r4 r1r1 Individual Pixel r1r1 r2r2 r3r3 r4r4 r5r5 Cloud Layers Ground Clutter Upper Canopy Lower Canopy

5. [5] Fused LADAR/EO Image 1 (Co-Registered and Georeferenced at the Pixel Level) LADAR Image EO Image

6. [6] LADAR Image EO Image Fused LADAR/EO Image 2 (Co-Registered and Georeferenced at the Pixel Level)

7. [7] Orthoprojection of the Two 3D Images Combined in Geographic Space First Image Second Image

8. [8] Profile Views in 3D Viewer First LADAR/EO Image First and Second LADAR/EO Images Combined

9. [9] Visualization Concept - Data Volume Exploration and Clutter Removal

10. [10] 3D Images of Cluttered Environment Image Plane Vegetation Canopy Ground Plane Terrain (per DTED or detected ground)

11. [11] Predict Ground Image Plane Bottom of Canopy Layer (Terrain + 3m) Ground Plane Terrain (per DTED or detected ground)

12. [12] Slice Volume at Bottom of Canopy Image Plane Bottom of Canopy Layer (Terrain + 3m) Ground Plane Terrain (per DTED or detected ground) Residual Shadowed Regions: P D  0

13. [13] Example of Canopy Slicing Orthoprojected Using LADAR – Trees Removed Oblique EO Scene Residual Shadowed Region: P D  0 Ground with Trees Removed

14. [14] Two Shots Combined – Trees Removed Residual Shadowed Region: P D  0 Corresponding EO Image

15. [15] Target Hidden Under Tree Hidden Target

16. [16] Target Exposed After Tree Removal Exposed Target

17. [17] Conclusions (1): Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction & Visualization Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments Clutter – tree canopies, clouds, etc. – can be removed through geometric filters

18. [18] Visualization Implication

19. [19] Some Visualization Implications Target Detection & Recognition >Probabilities of Detection >Probabilities of Target Presence (prior & posterior) >Exploitation of Negative Data Temporal Evolution of Estimation and Expectations >Out-of Sequence Data >Situation Projection >Expected Updates

20. [20] Prior & Posterior Detection Probabilities; e.g. Presentation of Negative Data Target Detectability Map 0.75 - 1.00 0 - 0.25 0.25 – 0.50 0.50 – 0.75 p D (x,t+n|t,A) Prior or Posterior Probability Density (e.g. Track Projection Uncertainty) x = Target state (type & location, etc.) t, t+n = Update times A = Action plan (sensor route, sensor controls, processing controls, etc.) p(x,t+n|t) Expectations Map Likely false alarm Likely missed detections p(x,t+n|t)p D (x,t+n|t,A)

21. [21] Some Visualization Implications Target Detection & Recognition >Probabilities of Detection >Probabilities of Target Presence (prior & posterior) >Exploitation of Negative Data Temporal Evolution of Estimation and Expectations >Out-of Sequence Data >Situation Projection >Expected Updates

22. [22] Revised Estimates of Track Histories (MTI + Imagery) Time ELEMENTARY SCHOOL MISSILE ASSEMBLY FACILITY MTI Reports Imagery Reports Latency Issues: Out of Sequence Data Initial Estimates of Track Histories (MTI only) Time ELEMENTARY SCHOOL MISSILE ASSEMBLY FACILITY

23. [23] Adaptive Information Exploitation Process Acknowledge uncertainties Estimate length of time for which decision can be deferred Estimate probability of resolving ambiguity in time >Given current collection plan >By redirecting current resources >By adding resource Estimate Cost & Net Utility of Candidate Action Plans Select & Implement Revised Plan

24. [24] Estimated HistoryEstimated Present SituationProjection Situation Situation Time -40-30-20-100+10+20+30+40 Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Visualizing Time-Varying Situation Estimation & Expectation (1 of 3) Current Estimate of Present Situation Report Time (Plan A)

25. [25] Estimated HistoryEstimated Present SituationProjection Situation Situation Time -40-30-20-100+10+20+30+40 Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Visualizing Time-Varying Situation Estimation & Expectation (2 of 3) Current Projection of Future Situation Report Time (Plan A)

26. [26] Visualizing Time-Varying Situation Estimation & Expectation (3 of 3) Report Time (Plan A) Estimated HistoryEstimated Present SituationProjection Situation Situation Time -40-30-20-100+10+20+30+40 Received ReportsExpected Coverage Tgt Selection DecisionTgt Engagement Decision Expected Update of Present Situation

27. [27] Conclusions Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments >Clutter – tree canopies, clouds, etc. – can be removed over time through geometric filters Enables visualization of time-varying situation estimation & expectation >Presentation of Detection Probabilities (prior & posterior): e.g. Presenting Negative Data >Presentation of Temporal Evolution of Estimates and of Expectations –Out-of Sequence Data –Situation Projection –Expected Updates & Information Evolution

28. [28] Thank You Mange Takk!