1. Dynamic Data Driven Applications System concept for Information Fusion Erik Blasch Guna Seetharaman, Kitt Reinhardt eeerik.blasch@gmail.com PM: Dr. Frederica Darema International Conference on Computational Science June 04-05 2013 Presented by Craig Douglas

2. Blasch/Seetharaman – ICCS 13 OUTLINE Motivation DDDAS Concept for Wide-Area Motion Imagery (WAMI) DDDAS match with Information Fusion DDDAS rich with applications to complex environmental modeling Information Fusion – Surveillance, Target Tracking Developments in motion imagery with large-data formats Focused on target tracking : need environment model in WAMI DDDAS processing for WAMI Statistical Mathematical modeling Developing results over different applications Summary: Future reporting in 2014 WAMI text

3. Blasch/Seetharaman – ICCS 13 DDDAS Mathematics Modeling Multimodal Cooperative Sensing Motivation Using the DDDAS concept (mathematics, modeling, and software); information fusion systems composing multi-modal sensor measurements can be enhanced to consider current trends in big data (e.g. large imagery), enterprise architectures (e.g. dynamic), and systems management for real-time cooperative sensing. The novelty of the work consists of developing a statistically optimal image perspective formation using 3D homographical mappings applied to Wide Area Motion Imagery for scene characterization, target tracking, and situational awareness. Applying the advanced scene characterization software in the AFRL PCPAD-X (Planning & Direction, Collection, Processing & Exploitation, Analysis & Production, and Dissemination eXperimentation) Program - that consists of multi-intelligence data fusion from streaming full-motion video (IMINT - Image) and operator textual reporting (HUMINT – human), the emerging situation can be understood for real-time mission management.

4. Blasch/Seetharaman – ICCS 13 4 DDDAS (Multimodal Cooperative Sensing) Dynamic Data Driven Application Systems (DDDAS) Theory Simulations Measurements Sensor Management Forecasting, Prediction Operational Condition Fidelity Filtering Mission Management User Refinement Situation Assessment Object ID and Tracking Full Motion Video Modeling Multi-scale Multimodal Data

5. Blasch/Seetharaman – ICCS 13 DDDAS Background Other works of interest DDDAS since its inception has been applied to numerous areas where complex real-world conditions are not predetermined by initialization parameters and data Darema, F., 2004. “Dynamic Data Driven Applications Systems: A New Paradigm for Application Simulations and Measurements. Computational Science DDDAS is related to Information Fusion in processing stochastic information Douglas, C. C., Efendiev, Y., Ewing, R., Ginting, V., and Lazarov, R., 2006. “Dynamic Data Driven Simulations in Stochastic Environments,” Computing, 77(4):321–333. DDDAS applied to Environmental modeling, Situation awareness, and Systems- level applications Environmental modeling Oceans [Patrikalakis, 2004], Wildfires [Chen M, 2005] Transportation [Fujimoto R. M; 2004], emergency medical response [Gaynor M; 05] Waste distribution [Parashar, M, 2006; Mahinthakumar, K, 2006]. Collaborative Adaptive Sensing of the Atmosphere (CASA) (Radar) formed by the National Science Foundation [Brotzge, J., 2006].

6. Blasch/Seetharaman – ICCS 13 OUTLINE Motivation DDDAS Concept for Wide-Area Motion Imagery (WAMI) DDDAS match with Information Fusion DDDAS rich with applications to complex environmental modeling Information Fusion – Surveillance, Target Tracking Developments in motion imagery with large-data formats Focused on target tracking : need environment model in WAMI DDDAS processing for WAMI Statistical Mathematical modeling Developing results over different applications Summary: Future reporting in 2014 WAMI text

7. Blasch/Seetharaman – ICCS 13 Information Fusion and DDDAS DDDAS Supports large (complex) data processing Reasons over stochastic uncertainty New Application: Enterprise Intelligence, Surveillance, and Reconnaissance Information Fusion Sensor Fusion (sensing) Info Fusion (enterprise) User Interaction (Report)

8. Blasch/Seetharaman – ICCS 13 Information Fusion and DDDAS DDDAS and Information Fusion Environmental modeling for object assessment, situation and impact assessment over mission needs E. P. Blasch, E. Bosse, and D. A. Lambert, High-Level Information Fusion Management and Systems Design, Artech House, Norwood, MA, 2012. Information Fusion Processing Levels : L0 data registration, L1 object assessment, (tracking, classification) L2 situation awareness L3 impact assessment (threat). L4 process refinement, L5 user refinement L6 mission management Applications : emergency response, sensor /user control (CASA), transportation.  HERE, focused on ISR tracking in imagery

9. Blasch/Seetharaman – ICCS 13 9 Point of View Cover Proc IEEE, Nov 2010

10. Blasch/Seetharaman – ICCS 13 OUTLINE Motivation DDDAS Concept for Wide-Area Motion Imagery (WAMI) DDDAS match with Information Fusion DDDAS rich with applications to complex environmental modeling Information Fusion – Surveillance, Target Tracking Developments in motion imagery with large-data formats Focused on target tracking : need environment model in WAMI DDDAS processing for WAMI Statistical Mathematical modeling Developing results over different applications Summary: Future reporting in 2014 WAMI text

11. Blasch/Seetharaman – ICCS 13 11 CLIF 2006/2007 EO UAV MWIR UAVLAIR Building mounted EO From Olga Mendozza-Schrock, Jim Patrick, E. Blasch, “Image Techniques for Layered Sensing,” IEEE NAECON09. Public Release No. RY-09-0449

12. Blasch/Seetharaman – ICCS 13 DDDAS Mathematics Modeling Multimodal Cooperative Sensing f(x, y; t)  f(x, y; t + 1) Surface (S1) (e.g. tennis court) Line (S2) (e.g. telephone pole) Wall (S3) (e.g. fence) Cube (S4) (e.g. building) COMPLEX (e.g. City) Complex Scene Analysis Data: Text (analyst reports) - “Find Building A” Data: Geospatial (terrain) - “3D surface” Dynamic: Perspective change from moving camera Surface Mapping Simple: 2D surface homography Complex: Different surfaces in 3D Least Squares corrupts 3D result Science Contribution Locate primitive surfaces Determine dynamic change through mixture of piecewise optimize mappings DDDAS

13. Blasch/Seetharaman – ICCS 13 13 DDDAS Applications Modeling (1) Multimodal Edge Detection WAMI Wide-Area Motion Imagery Road EW Road [ ℓ x, ℓ y, ℓ z ] WAMI Modeling Complex Imagery Changing Dynamics Model  testing, and update Decisions – Terrain Information (environment) Man-made objects (Buildings, targets) Environment Vanishing Point DDDAS WAMI MODEL Vanishing Point ( ℓ x, ℓ y) H-Test H0 / H1 Edges

14. Blasch/Seetharaman – ICCS 13 14 DDDAS Applications Modeling (2) Multimodal Edge Detection Sensor Vanishing Point WAMI Wide-Area Motion Imagery nadir zenith South  north West  East Z w y c X w Y w x c z c e y e z Road EW Road [ ℓ x, ℓ y, ℓ z ] Multimodal Inputs (only WAMI displayed) Text (analyst reports) Geospatial (terrain) Use primitives (through vanishing points ) Multimodal Difficult Text: “Find building A” Environment Vanishing Point Target Vanishing Point DDDAS

15. Blasch/Seetharaman – ICCS 13 DDDAS Mathematics and Stat Modeling Measurement based Optimality Non-Homography relation Transforms (between multimodal collections) f(x, y; t)f(x’, y’; t + 1) Surface (S1) (e.g. tennis court) Line (S2) (e.g. telephone pole) Wall (S3) (e.g. fence) Cube (S4) (e.g. building) COMPLEX (e.g. City)   Homogeneous z = - px – qy - s c  s  a  = {  S1,  S2,  S3,  S4 } s = each surface L 34 = S 3  S 4  = {  S3,  S4 } Mathematical Issues Working Must have tangential conformity Optimality Conditions (from terrain) If Imagery, how other sensors Joint DDDAS optimization L = direction cosine DDDAS

16. Blasch/Seetharaman – ICCS 13 DDDAS Systems Software PCPAD-X (Processing, Collection, and Dissemination)  Collaboration: RY (Sensors), RI (Information), RH (Human Effectiveness) Fusion Testbed Evaluation FMV Terrain User Visualization of complex scene Evaluation User (filter, analyze, report) Data Exploitation Multi-INT Fusion Viewer DDDAS Multiscale chat, text, doc Multiscale imagery

17. Blasch/Seetharaman – ICCS 13 OUTLINE Motivation DDDAS Concept for Wide-Area Motion Imagery (WAMI) DDDAS match with Information Fusion DDDAS rich with applications to complex environmental modeling Information Fusion – Surveillance, Target Tracking Developments in motion imagery with large-data formats Focused on target tracking : need environment model in WAMI DDDAS processing for WAMI Statistical Mathematical modeling Developing results over different applications Summary: Future reporting in 2014 WAMI text