Next Generation 4-D Distributed Modeling and Visualization of Battlefield Next Generation 4-D Distributed Modeling and Visualization of Battlefield Avideh Zakhor UC Berkeley June, 2001

Participants yAvideh Zakhor, (UC Berkeley) yBill Ribarsky, (Georgia Tech) yUlrich Neumann (USC) yPramod Varshney (Syracuse) ySuresh Lodha (UC Santa Cruz)

Battlefield Visualization zDetailed, timely and accurate picture of the modern battlefield vital to military zMany sources of info to build “picture”: yArchival data, roadmaps, GIS and databases: static ySensor information from mobile agents at different times and location: dynamic. yMultiple modalities: fusion zHow to make sense of all these without information overload?

Major Challenges zDisparate/conflicting sources of information resulting in large volumes of data. zData sources inherently uncertain: resulting models also uncertain. zData and models together with associated uncertainty need to be visualized on mobiles with limited capability. zAll of this is time varying, time dependent and dynamic.

Mobile AR Visualization Laser Lidar Radar Camera GPS Maps Gyroscope-- 3D model extraction Visualization Database Model update Mobiles with augmented reality sensors Decision Making UNCERTAINTY

Research Agenda z Model construction and update: y Radar, Laser, Cameras, Roadmaps, Satellite photos, GPS, Gyros, Speedometer, INS. yAugmented reality sensor tracking and registration required zMobile, real time visualization, interaction and navigation within the database : yAugmented reality sensor tracking and registration required; zUncertainty processing and visualization: ySensors, algorithms, models. zFusion used in all of the above.

Visualization Pentagon Information Fusion Information Fusion Uncertainty Processing/ Visualization Uncertainty Processing/ Visualization 4D Modeling/ Update 4D Modeling/ Update Visualization Database Visualization Database Tracking/ Registration Tracking/ Registration

Model Construction zDevelop a framework for fast, automatic and accurate 3D model construction for urban areas. zModels must be: y Easy, fast, automated, to compute; y Compact to represent; y Easy to render; yAmenable to photo realistic flythroughs. zStrategy: yFusion of multiple data sources: intensity, range, heading, speedometer, panoramic cameras. yIncorporate apriori models, e.g. aerial photos, and digital roadmaps.

3D Modeling: z Close-range modeling: y ground based vehicle with multiple sensors z Far-range modeling: y Aerial/satellite imagery z Fusion of close range and far range info at multiple levels: y Data and models.

Laser range scanners Digital roadmaps Aerial photos Ground Level Based Model Construction

Ground level based 3D model construction 3D point cloud from 2D scans Mesh generation Texture Mapping

Far Range Modeling Example of an aerial image and extracted disparity map of man- made structures

Tracking, Registration, & Autocalibration zNeeded for Augmentation or Visualization zMobile people with sensors provide textures and data for visualizations… yWhere are they? Where are they looking? yWhat are they doing? zTracking provides position and orientation yTracking is possible with landmarks in view yAutocalibration allows tracking in regions beyond/near landmarks – also can improve/refine models ySensor fusion (GPS, vision, gyroscopes) adds more tracking information for wider areas

Mobile Testbed

Registration Using Mutual Information Original IRS and RADAR image pair Registered RADAR image

Temporal Aspects zTime critical applications: ySequential methods for information fusion yTradeoff between speed and accuracy of results(QoS) zFuse information arriving at different times: yPerform independent time-based calculations, followed by fusion/integration

Fusion of Asynchronous Information t new Fusion Projection Source 1 Source 2 Source 1 Source 2 t1t1 t2t2 t new Decision maker

Mobile Visualization  Synchronized data bases of dynamic data in collaborative environments  Peer to peer mode  Sharing geo-spatial data, annotations, triggering events  Mobility:  Development of mobile visualization capability using laptop version of global geospatial system  Gesture recognition capability  New hierarchical volume rendering technique to weather and uncertainty Dynamic, Synchronized Databases Real-time acquisition and insertion

Uncertainty Computation and Visualization

Uncertainty in Visualizations of Terrains zTerrain visualization is important in a battlespace environment zTerrain compression needed due to -- real-time interaction -- limited screen space -- progressive transmission requirements zHow much uncertainty is introduced due to data compression? zCan we trust compressed terrain images?

Monterey Bay originalunconstrained, 20%unconstrained, 70% constrained, 70%constrained, 80%

Target Uncertainty Visualization in 2D Galaxy Diffuse Color Speed:Mean 1.0; Std Dev 0.2 Direction: Mean 0.0; Std Dev 45

Transitions (1) z Established close relationships with both government agencies and industry: y Government agencies xAFRL Information Directorate, TRADOC Analysis Center xOffice of Naval Research (ONR) xNASA (Houston Space Center) xDARPA  CERTIP (Center for Emergency Response Technology, Instrumentation, and Policy), yIndustry: xHRL, Boeing, Raytheon, General Electric, HP xOlympus America xNiagara Mohawk Power Corp, ICT x Carrier Corp, Andro consulting, Kitware

Publications (1) z Hassan Foroosh, “Conservative Optical Flow Fields”, submitted to International Journal of Computer Vision, October z Hassan Foroosh, “A General Motion Model based on The Theory of Conservative Fields”, submitted to International Conference on Computer Vision and Pattern Recognition, May z Hassan Foroosh, “A Closed-Form Solution For Optical Flow By Imposing Temporal Constraints”, accepted in International Conference on Image Processing, January z Hassan Foroosh and Avideh Zakhor, “Virtual Cities From Aerial Images By Fusion of Segmentation And Stereo Vision in a Bayesian Framework”, submitted to International Conference on Computer Vision and Pattern Recognition, May z Hassan Foroosh and Avideh Zakhor, “Projective Rectification Using a Single Homography”, submitted to International Conference on Computer Vision and Pattern Recognition, May z Christian Frueh and Avideh Zakhor, “Fast 3D Model Generation in Urban Environments”, accepted in Conf. on Multisensor Fusion and Integration for Intelligent Systems, February 2001.

Publications (2) z Christian Frueh and Avideh Zakhor, “3D model generation for cities using aerial photographs and ground level laser scans”, submitted to International Conference on Computer Vision and Pattern Recognition, May z N.L. Chang and A. Zakhor, “A Multivalued Representation for View Synthesis”, in Proceedings of the International Conference on Image Processing, Kobe, Japan, October 1999, vol. 2, pp ; Also submitted to International Journal of Computer Vision, January 2000 zT.Y. Jiang, William Ribarsky, Tony Wasilewski, Nickolas Faust, Brendan Hannigan, and Mitchell Parry,“Acquisition and Display of Real-Time Atmospheric Data on Terrain,” to be published, Proceedings of Eurographics-IEEE Visualization Symposium zMitchell Parry, Brendan Hannigan, William Ribarsky, T.Y. Jiang, and Nickolas Faust, “Hierarchical Storage and Visualization of Real-Time 3D Data,” to be published, Proceedings of SPIE Aerosense zMitchell Parry, William Ribarsky, Chris Shaw, Tony Wasilewski, Nickolas Faust, T.Y. Jiang, and David Krum, “Acquisition, Management, and Use of Large-Scale, Dynamic Geospatial Data,” submitted to IEEE Computer Graphics & Applications

Publications (3) zJ. W. Lee, S. You, and U. Neumann. “Large Motion Estimation for Omnidirectional Vision,” IEEE Workshop on Omnidirectional Vision, with CVPR, June 2000 zJ.W. Lee, U. Neumann. "Motion Estimation with Incomplete Information Using Omni- Directional Vision," IEEE International Conference on Image Processing (ICIP'00), Vancouver Canada, September 2000 zS. You, U. Neumann. "Fusion of Vision and Gyro Tracking for Robust Augmented Reality Registration," IEEE VR2001, pp.71-78, Yokahama Japan, March 2001 zB. Jiang, U. Neumann, “Extendible Tracking by Line Auto-Calibration,” submitted to ISAR 2001 zAntonio Hasbun, Pramod K. Varshney, Kishan G. Mehrotra and C.K. Mohan, “Uncertainty Introduced by Quantization in Image Processing Applications”, Under preparation. zKishan G. Mehrotra, C.K. Mohan and Pramod K. Varshney, “Theory and Applications of Random Sets: A Review”, Under preparation.

Publications (4) zSuresh K. Lodha, Krishna M. Roskin, and Jose Renteria,``Hierarchical Topology Preserving Compression of 2D Terrains'', Submitted for publication to Computer Graphics Forum. zSuresh K. Lodha and Arvind Verma, ``Spatio-Temporal Visualization of Urban Crimes on a GIS Grid'', Proceedings of the ACM GIS Conference, November 2000, ACM Press, pages zSuresh K. Lodha, Jose Renteria and Krishna M. Roskin, ``Topology Preserving Compression of 2D Vector Fields'', Proceedings of IEEE Visualization '2000, October 2000, pp zChristopher Campbell, Michael Shafae, Suresh K. Lodha and D. Massaro, ``Multimodal Representations for the Exploration of Multidimensional Fuzzy Data", Submitted for publication to Behavior Research Methods, Instruments, and Computers.

Publication (5) zQ. Zhang and P. K. Varshney, “Decentralized M-ary Detection via Hierarchical Binary Decision Fusion”, Information Fusion, Vol. 2, pp. 3-16, March zHua-mei Chen and Pramod K. Varshney, “A pyramid approach for multimodality image registration based on mutual information”, Proceedings of 3 rd international conference on information fusion, vol. I, pp. MoD3 0-15, Paris, July z Hua-mei Chen and Pramod K. Varshney, “Generalized partial volume interpolation for image registration based on mutual information”, presented at 2000 WESTERN NEW YORK, IMAGE PROCESSING WORKSHOP, University of Rochester,October 13, zHua-mei Chen and Pramod K. Varshney, ”A cooperative search algorithm for mutual information based image registration,” Proc. of Aerosense 2001, Orlando, April zAndrew L. Drozd et al., “ Towards the development of multisensor integrated display systems,” To be presented at Fusion 2001, Montreal, Aug zDavid Krum, William Ribarsky, Chris Shaw, Larry Hodges, and Nickolas Faust, “Situational Visualization,” submitted to ACM Virtual Reality Software and Technology.

Cross Collaboration UCBUSCG.T.SYRUCSC Model const. & update X x x x Tracking & reg. x X x x Mobile visual. database x x X x x Uncertain. processing x x X x Uncertain. Visualizati on. x x X

Outline of Talks zU.C. Berkeley, "3D model construction for urban environments“ zUSC, " Tracking and Visualization with Models and Images" z Georgia Tech, "Mobile Visualization in a Dynamic, Augmented Battlespace" zSyracuse, "Uncertainty Processing and Information Fusion for Visualization" zU.C. Santa Cruz, "Uncertainty Quantification and Visualization: Terrains and Targets"

3D Model Construction for Urban Environments z Close-range modeling: y ground based vehicle with multiple sensors z Far-range modeling: y Aerial/satellite imagery z Fusion of close range and far range info at multiple levels: y Data and models.