Next Generation 4-D Distributed Modeling and Visualization of Battlefield Next Generation 4-D Distributed Modeling and Visualization of Battlefield Avideh Zakhor UC Berkeley

MURI on Mobile Augmented Battlespace Visualization zParticipants: yAvideh Zakhor, (UC Berkeley) yBill Ribarsky, (Georgia Tech) yUlrich Neumann (USC) yPramod Varshney (Syracuse) ySuresh Lodha (UC Santa Cruz)

Battlefield Visualization zDetailed, timely and accurate picture of the modern battlefield vital to military zMany sources of info: yeye witness, aerial photographs, sonar, Synthetic Aperture Radar (SAR), Multi- Spectral Imaging (MSI), Hyper-Spectral Imaging (HSI). Foliage PENetration (FOPEN) radar, Electro-Optic (EO), Infra- Red (IR), Moving Target Imaging (MTI)

Major Challenges zDisparate/conflicting sources of info must be combined. zImpossible for ONE individual to collect and comprehend zSpecially trained technicians for each info source zEffectiveness of information combining and fusion determined by its usability. zMust avoid information overload in presenting the data.

Historical Perspective zSand box: ybox filled with sand shaped to replicate battlespace terrain. yCommanders moved around small physical replicas of battlefield objects to direct situation.

Historical Perspective zPaper maps and acetate: yAs intelligence arrives, technicians use grease pencils to mark new info on acetate. yCommanders draw on the acetate to plan battlefield situations. yTime consuming: several hours to print, distribute and update. yMany opportunities for introducing errors.

Historical Perspective zJoint Maritime Command Information System (JMCIS): ycomputerized battle space visualization ytwo fundamental limitations: xclutter when displaying too much info. xTwo dimensional display: 3D info is lost.

Historical Perspective z Responsive workbench with Dragon: yInherently 3D. yWorkstation renders 3D, back-projected on a horizontal screen. yUsers in the viewing area interact with the bench through 3D mouse, pinch gloves, speech recognition. yStereographic display with LCS shutter glass.

Improvements over WRB/Dragon zWorkbench not suitable for mobile soldier with PDA. zAugmented reality can enhance Dragon/WRB: ysensors distributed among the soldiers, can be used both to navigate and to update the visualization database. zNeed to deal with uncertainty: represent, compute, and visualize uncertainty information without cluttering. zTime should become the 4th dimension. 4D model construction. Play back/visualize the last 24 hours.

Agile, Mobile, Collaborative Testbed  A networked, collaborative whole Earth terrain database linking workstations, large projected displays, and mobile handheld systems.  Mobile users carry systems with handheld or augmented displays providing 3D terrain visualizations.  Mobile users receive, record, and transmit information about the world.  Users of stationary 3D displays collect and evaluate information provided by mobile users and route data and intelligence back to mobile users.  Collaboration is through annotation of the virtual geo-spatial database.

Central Terrain Database NAVEVirtual Workbench Mobile Displays Collaboration Channels Wireless Terrain Datalink Terrain Datalink Position, Terrain Markup, Route Finding, Weather, Friendly/Foe advisories, etc. w/GPS, bearing, tilt sensors and wireless data. Local databases download and cache data according to bandwidth, movement, and rendering speed of each platform. Wireless Architecture: Agile, Mobile, Collaborative Testbed

Research Agenda z Model construction: y initially constructed by registering sensor imagery to reference imagery, maps, elevation data, etc. yFour dimensional: time treated on the same footing as space. zModel update: Distributed mobile or stationary users contribute to updating the database via their sensors. zMobile, real time visualization, interaction and navigation within the database; augmented reality sensors tracking and registration required; zUncertainty processing for model construction and update, as well as uncertainty visualization.

Tracking & Registration Database Visualization Model Update & Construction Uncertainty processing Uncertainty visualization Technical Challenges

3-D and 4D Model Construction zDevelop a framework for fast, automatic and accurate 3D model construction for objects, scenes, rooms, buildings (interior and exterior), urban areas, and cities. Incorporate time element (4-D). zModels must be easy to compute, compact to represent, suitable for insertion in large hierarchical visualization databases, to facilitate high quality view synthesis and visualization from views that were not necessarily captured during data collection process. zStrategy: yFusion of multiple data sources: intensity, range, GPS, panoramic cameras. yIncorporate apriori models, e.g. CAD, DEM, DTED, elevation data, maps

Geo-registration and Tracking zDevelop techniques for unemcumbered, wide area and real time tracking for (a) augmentation and (b) visualization. zStrategy: Estimate real time 6-DOF tracking by fusion of multiple data streams with variable uncertainty: GPS, vision, inertial gyros, accelerometers, compass, laser range finders zReal time prototype will use the resulting tracking algorithms on an augmented reality PDA to geo-register and navigate the user within the geo-spatial database.

Mobile Visualization zVGIS: general framework for global geo-spatial data and visualization. zHierarchical, scalable data structures with fast access that include (a) time; (b) uncertainty; c) all varied products. zDynamic data structures with fast update: Real time data is put in dynamic cache until system finds time to integrate it with the online data base. zAutomated detail management for (a) uncertainty; (b) all visual products; zIntelligent retrieval and visual data mining. zMulti-modal interaction in multiple environments.

Multi-modal Interaction in Multiple Environments Environments Desktop Laptop Mobile (palm, HMD, larger handheld or wearable) Virtual Workbench Liveboard NAVE Others Interaction Modes Mouse Joystick or Spaceball Pen-based or touch screen Wired 3D or wireless 3D Head tracking, hand-tracking, gesture recognition Voice Others What are the environmental affordances? What are user needs? Is 3D interaction necessary? How best to navigate large virtual spaces (2D or 3D)? How to most effectively combine modes? How to evaluate?

Technical Challenges: Uncertainty Computation and Visualization

Uncertainty Processing zRepresentation and computation issues: ymany formalisms: probability, possibility, evidence theory; Transformations; zFusion: data, feature and decision levels yuncertainty aware fusion algorithms for dynamic distributed networks; various topologies for fusion networks: serial, parallel, tree network, non-tree feedback network. ydecentralized statistical inference algorithms zTime Critical computation and quality of service issues: yData continually arriving, requiring re-computation. ytradeoff between precision and speed of uncertainty computation; fast, yet imprecise answers.

Uncertainty Visualization and Validation zPresent uncertainty in an intuitive uncluttered way; zDisplay devices: screen space, mobility zModality: vision, audio, haptics zData types: scalar/vector/tensor, discrete/continuous, static/dynamic zUncertainty visualization techniques: glyphs, deformation, transparency, texture, superimposing/backgrounding, augmented reality. zValidation with novice and expert users: yTasks for mobile battlefield; measure accuracy and speed performance of user; conduct stat. analysis.

Proof of Concept : Multi-modal Interactions for PDA z Extend interaction methods for workstations to PDA; zWired (electromagnetic) and unwired (vision) interaction modes, hand gestures zVisible and infrared sensors added to PDA zDemonstrate a hybrid vision/inertial tracking PDA ydemonstrate VGIS visualization on portable display context zMultiple mobile users collaborate on identifying and locating features or targets.

Facilities and Equipment zG. Tech: GVU Future Computing Lab (FCL); Mobile augmented test-bed. zUSC: IMSC lab; CGIT lab. zUCSC: VizLab zSyracuse: Sensor Fusion Laboratory (SFL) zBerkeley: Video and Image Processing (VIP) Lab.

Collaboration and Management issues zDr. Hassan Foroosh currently with University of Maryland, will be the technical liason amongst the five universities. zTelephone conference amongst team members periodically to discuss status of the project. zLive seminars broadcast via video conferencing amongst five campuses, once every two months. zAnnual workshops and retreats for PI and graduate students and all other researchers to exchange ideas.

Transitions zAir Force Research Lab (Graniero) zArmy Research Lab (Emmerman, Tocarcik) zNavy Research Lab (Rosenblum) zLawernce Livermore Lab (Ucelton) zSun Microsystems (Sowizraj) zIntel (Liang) zPlanet 9 (Colleen) zGeometrix (Zwern) zHughes Research Lab (Azuma)

Cross Collaboration UCBUSCG.T.SYRUCSC Model const. & update X x x x Tracking & reg. x X x x Mobile visual. database x x X x x Uncertain. processing x x X x Uncertain. Visualizati on. x x X

Outline of Talks z3D model construction for visualization (UC Berkeley) zGeo-registration and tracking for augmentation and visualization(USC) zMobile visualization in dynamic, augmented battlespace (Georgia Tech) zUncertainty processing and information fusion (Syracuse) zUncertainty visualization and validation (UC Santa Cruz)