Testbed for Mobile Augmented Battlefield Visualization: Summing Up May 10, 2006

Xin Zhang, Tazama Upendo St Julien, Ramesh Rajagopalan, William Ribarsky, Pramod Varshney, Chilukuri Mohan, and Kishan Mehrotra. Dynamic Decision Support for Mobile Situational Visualization. AppliedVis William Ribarsky, co-editor, Special Issue on Haptics, Telepresence, and Virtual Reality, IEEE Transactions on Visualization and Computer Graphics (November, 2005). Justin Jang, Peter Wonka, William Ribarsky, and C.D. Shaw. Punctuated Simplification of Man-Made Objects. To be published, The Visual Computer. Tazama St. Julien, Joseph Scoccinaro, Jonathan Gdalevich, and William Ribarsky. Sharing of Precise 4D Annotations in Collaborative Mobile Situational Visualization. Submitted to IEEE Symposium on Wearable Computing. Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and William Ribarsky. Using Urban Legibility to Produce Completely Navigable Large Scale Urban Models. To be published, ACM SIGGRAPH 2006 Short Papers. Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and William Ribarsky. Hierarchical Simplification of City Models to Maintain Urban Legibility. Submitted to IEEE Transactions on Visualization and Computer Graphics. Xin Zhang, Tazama Upendo St Julien, Ramesh Rajagopalan, William Ribarsky, Pramod Varshney, Chilukuri Mohan, and Kishan Mehrotra. An Integrated Path Engine for Mobile Situational Visualization. To be submitted. Publications in the Last Year

Matrix of Project Activities and Results

Transitions Presented mobile situational visualization and its relation to homeland security at invited talks at a special session of the AAAS meeting on the National Visualization and Analytics Center (February, 2005), at AppliedVis 2005 (May, 2005), and at an invited presentation for the DHS Regional Visualization and Analytics Centers (January, 2006). Using work in urban terrain analysis begun here as a foundation, began work on a project funded by ARO for eye-point dependent models applied to terrain analysis and applications such as line-of-sight. Made a proposal for the DTO ARIVA project that will use, among other things, urban infrastructure visualization. The proposal is now in Phase 2 evaluation. Established the Southeastern Regional Visualization and Analytics Center, funded by DHS. Among other things, the SRVAC will be looking at critical infrastructure simulations for disaster relief planning and emergency response. The terrain visualization and modeling capabilities developed here will be used.

Visualization and Analytics Centers RVAC Univ. of North Carolina Charlotte, Georgia Tech RVAC Penn. State DHS GVAC NVAC : Pacific Northwest National Laboratory RVAC Stanford University Scholars A Partnership with Academia, Industry, Government Laboratories Detecting the Expected -- Discovering the Unexpected TM RVAC University of Washington RVAC Purdue University IVAC Consortium

RVAC University of Washington RVAC Purdue University RVAC Univ. of North Carolina Charlotte, Georgia Tech Bank of America RVAC Penn. State DHS GVAC(s) NVAC : Pacific Northwest National Laboratory RVAC Stanford University Scholars Consortium IVAC Detecting the Expected -- Discovering the Unexpected TM Alaska New Zealand Australia Hawaii Europe Canada Pacific Rim Indiana Univ. School of Medicine Drexel University NY Emergency Management Port of Authority NSF Visualization and Analytics Centers A Partnership with Academia, Industry, Government Laboratories

Mobile Situational Visualization User equipped with mobile situational visualization system Mobile Situational Visualization: An extension of situation awareness that exploits and integrates interactive visualization, mobile computing, wireless networking, and multiple sensors: Mobile users with GPS, orientation sensing, cameras, wireless User carries own 3D database Servers that store and disseminate information from/to multiple clients (location, object/event, weather/NBC servers) Location server to manage communications between users and areas of interest for both servers and users Ability to see weather, chem/bio clouds, and positions of other users Accurate overviews of terrain with accurately placed 3D buildings Ability to mark, annotate, and share positions, directions, speed, and uncertainties of moving vehicles or people Ability to access and playback histories of movement

Drawing Area Buttons Pen Tool Mobile Team Collaboration Example collaborators Shared observations of vehicle location, direction, speed Mobile Situational Visualization

Everybody has a location in space and time in the Virtual World Geographic server lookup approach –Users –Location Servers –Data Servers Weather Server User Location Server Traffic Server Annotation Server GeoData Server Mobile Sitvis Collaborative Environment

Everybody has a location in space and time in the Virtual World Geographic server lookup approach –Users –Location Servers –Data Servers User Location Server Traffic Server Annotation Server GeoData Server Weather Server Mobile Sitvis Collaborative Environment

Testbed Development Accurate placement of modeled buildings and trees from multiple sources Georgia Tech campus -Navigable environment Detailed urban component Realistic urban buildings x

Scenarios and Results Scenario 1 A commander, out of sight of his unit, directs it. He creates waypoints and paths through the mobile sitvis system that individuals in the unit move to. Scenario 2 Three individuals in a unit track a moving subject. They must keep it in sight and coordinate their tracking activities. In both scenarios, working with the full mobile sitvis system is compared with a traditional method (individuals with only radio communication and maps). Tracked subjects followed paths (set for about equal length and number of turns) that were not known to unit members.

Scenarios and Results Scenario 2: History of all users’ locations and annotations over a 45 minute session. (Tracked subject is in red.)

Scenarios and Results What did we find? Mobile sitvis works! It does as well as traditional method (radio + map) for tracking a moving subject. It is better than traditional method for command operations that direct multiple units. It provides significant new capabilities: -Significantly more accurate location than GPS alone. -Specific digital annotations in space-time that can be shared immediately. -Overviews of several moving, annotated entities that can be understood all at once -Histories for tracking and analysis This work suggest new scenarios of greater impact.

Mobile Situational Visualization + Dynamic Decision Support 3D Interaction Visualization Decision Support Module 4D Geospatial Server Sensors User Input Collaboration with the Syracuse team

Mobile Situational Visualization + Dynamic Decision Support

We have fully integrated the dynamic decision support engine with mobile situational visualization, providing the following capabilities: a structure for shared interaction and collaboration among mobile users, general methods for heterogeneous spatiotemporal sensor organization and display, a decision support module supporting activity recognition, response planning, and behavioral modeling that is integrated with the mobile visualization structure and will accept the mobile users as collaborating agents, a prototype mobile situational visualization system that employs the decision engine to produce meaningful responses in one or more urban scenarios. Mobile Situational Visualization + Dynamic Decision Support

Dynamic Decision Support: Grid-Based Approach The region to be traversed is laid out in a grid, balancing computation cost (no. of grid cells) versus accuracy (edges or nodes in the same grid cell). Edge relaxation is used to choose vertices. Vertices connected by valid edges are considered, and those with the best value of a quality metric, Q, are chosen. A probability risk model is applied. A simple zero mean Gaussian distribution with a finite range is used to model point risks. Default values are given for grenades, rifles, etc. If multiple risks overlap in a grid cell, the multiple threat is computed as [1 - (1 - P1) x (1 – P2) x …… x (1 - Pn)] where there are n threats. Djikstra’s single source, single destination shortest path algorithm is used.

Dynamic Decision Support: Grid-Based Approach Red boundary shows selected area in dense urban area (mid-town Atlanta).

Green path (highlighted with green dots): optimal (in this case low risk) path for selected balance between risks and path length. Our initial urban scenario is route planning under dynamic threats. Threats of different extents and risks are sighted, placed, and shared by mobile users. The decision engine provides a real-time path that balances on a continuous scale between risk and shortest path (where the mobile user can select the balance). Mobile Situational Visualization + Dynamic Decision Support

The mobile decision engine produces fast, accurate, and usable results Mobile Situational Visualization + Dynamic Decision Support Safe route Some risk Shortest route

Future Directions Combine dynamic route planning with line-of-sight to take into account obstructions in determining risk. Scale up to larger areas. Take into account moving risks or risks that change in other ways. Support other decisions.

Interactive Visualization of Very Large Urban Spaces How can one freely navigate very large urban spaces? -A medium size city can have several hundred thousand buildings; a large city can have millions of buildings. -Even if the simplest building models are rendered, there could still be an overwhelming amount of geometry and textures. View of Xinxiang, China with over 26,000 buildings. What should be rendered? Apply knowledge from urban planning: urban legibility.

Urban legibility embodies concepts from urban planning about what makes an urban space understandable and more easily navigable. (For example, depict the city around the concepts of paths, edges, districts, nodes and landmarks.) Can we find an automated way to embody these concepts and thus keep the city legible (and recognizable) at all scales? Interactive Visualization of Very Large Urban Spaces

Yes, we can shape our automated urban analysis to embody the urban legibility principles. Original (textured) District Simplification with our method Simplification with Qslim Our simplified model with textures applied Interactive Visualization of Very Large Urban Spaces

Skyline at full resolution Skyline with 7% polygons Landmark preservation Full resolution Interactive view with 18% polygons and greatly simplified textures View-dependent rendering Perceptual errors are not very noticeable because conceptual structure (i.e., what’s important) is retained. Interactive Visualization of Very Large Urban Spaces

View-Dependent Rendering of Very Large Collection Bounding box Selected LOD Hierarchical multiresolution organization View-Dependent LOD for large collections of 3D models Q QQQQ Q QQQQ Q QQQQ Q N Levels Linked Global Quadtrees Viewpoint Screen

Knowledge Visualization: Very Large Urban Spaces Video

Organizing Large Collections of 3D Models for Interactive Display Merging of different types and formats Automated replacement of lower resolution duplicate structures Common format and organization for different types Q QQQQ Q QQQQ Q QQQQ QQQQ Linked Global Quadtrees

Paging, Culling, and Fast Rendering Quadcell Block QQQQ Linked global quadtree Block Out-of core Storage

Hierarchical, Multiresolution Organization Quadtrees LODs

Collections of simple geometry Quadtree interior nodesQuadtree leaf nodes collection1 collection2 collection3 model1 tree2sg3 Hierarchical, Multiresolution Organization Urban Legibility Detailed Hierarchical Simplification

Questions?

Publications from Previous Years Ernst Houtgast, Onno Pfeiffer, Zachary Wartell, William Ribarsky, and Frits Post. Navigation and Interaction in a Multi- Scale Stereoscopic Environment. IEEE Virtual Reality Nickolas Faust and William Ribarsky. Integration of GIS, Remote Sensing, and Visualization. Invited paper, Proc. Remote Sensing 2003 (Barcelona, 2003). William Ribarsky, editor (with Holly Rushmeier). 3D Reconstruction and Visualization of Large Scale Environments. Special Issue of IEEE Computer Graphics & Applications (December, 2003). David Krum, Olugbenga Omoteso, William Ribarsky, Thad Starner, and Larry Hodges. Evaluation of a Multimodal Interface for 3D Terrain Visualization. pp IEEE Visualization Justin Jang, William Ribarsky, Christopher Shaw, and Peter Wonka. Appearance-Preserving View-Dependent Visualization. IEEE Visualization 2003, pp William Ribarsky, Zachary Wartell, and Nickolas Faust. Precision Markup Modeling and Display in a Global Geospatial Environment. Proceedings SPIE 17th International Conference on Aerospace/Defense Sensing, Simulation, and Controls (2003). William Ribarsky. Virtual Geographic Information Systems. The Visualization Handbook, Charles Hanson and Christopher Johnson, editors (Academic Press, New York, 2003). Zachary Wartell, Eunjung Kang, Tony Wasilewski, William Ribarsky, and Nickolas Faust. Rendering Vector Data over Global, Multiresolution 3D Terrain. Eurographics-IEEE Visualization Symposium 2003, pp Peter Wonka, Michael Wimmer, Francois Sillion, and William Ribarsky. Instant Architecture. Siggraph 2003, pp (2003). Tony Wasilewski, William Ribarsky, and Nickolas Faust. From Urban Terrain Models to Visible Cities. Vol. 22(4), pp , IEEE CG&A (2002). David Krum, Rob Melby, William Ribarsky, and Larry Hodges. Isometric Pointer Interfaces for Wearable 3D Visualization. ACM CHI William Ribarsky, “Towards the Visual Earth,” Workshop on Intersection of Geospatial Information and Information Technology, National Research Council (October, 2001). William Ribarsky, Christopher Shaw, Zachary Wartell, and Nickolas Faust, “Building the Visual Earth,” to be published, SPIE 16th International Conference on Aerospace/Defense Sensing, Simulation, and Controls (2002).

Publications from Previous Years David Krum, William Ribarsky, Chris Shaw, Larry Hodges, and Nickolas Faust. Situational Visualization. pp , ACM VRST 2001 (2001). David Krum, Olugbenga Omoteso, William Ribarsky, Thad Starner, and Larry Hodges. Speech and Gesture Multimodal Control of a Whole Earth 3D Virtual Environment. Eurographics-IEEE Visualization Symposium Winner of SAIC Best Student Paper award. “Acquisition and Display of Real-Time Atmospheric Data on Terrain,” T.Y. Jiang, William Ribarsky, Tony Wasilewski, Nickolas Faust, Brendan Hannigan, and Mitchell Parry, Proceedings of the Eurographics-IEEE Visualization Symposium 2001, pp “Client-Server Modes of GTVGIS,” Nick Faust, William Ribarsky, and Frank Jiang, Vol. 4368A, SPIE 15th Annual Conference on Aerosense (2001). “Hierarchical Storage and Visualization of Real-Time 3D Data,” with Mitchell Parry, Brendan Hannigan, William Ribarsky, T.Y. Jiang, and Nickolas Faust, Proc. SPIE 15 th Annual Conference on Aerosense 2001, Vol. 4368A. “Semiautomatic Landscape Feature Extraction and Modeling,” Matthew Grimes, Tony Wasilewski, Nickolas Faust, and William Ribarsky, Proc. SPIE 15th Annual Conference on Aerosense (2001), Vol. 4368A. “Real-Time Global Data Model for the Digital Earth,” William Ribarsky, Nickolas Faust, William Ribarsky, T.Y. Jiang, and Tony Wasilewski, Proceedings of the INTERNATIONAL CONFERENCE ON DISCRETE GLOBAL GRIDS (2000). Development of Tools for Construction of Urban Databases and Their Efficient Visualization,” Nickolas Faust and William Ribarsky, Modeling and Visualizing the Digital Earth, Mahdi Abdelguerfi, Editor (Kluwer, Amsterdam, 2001). Computers & Graphics, Special Issue on Data Visualization (Vol. 24, no. 3, June, 2000), Editors Eduard Groeller, William Ribarsky, and Helwig Loeffelmann.

Students Who Worked on Project Remco Chang Tom Butkiewicz Caroline Ziemkiewicz Xin Zhang Justin Jang Tazama St. Julien David Krum Olugbenga Omoteso Jaeil Choi Weidong Shi Guoquan (Richard) Zhou Eunjung Kang Brendan Hannigan Mitchell Parry Matthew Grimes Ernst Houtgast Onno Pfeiffer Joseph Scoccinaro Jonathan Gdalevich