Distributed Spatial Databases The MIDCARB Carbon Sequestration Project This talk was presented at the 2002 Digital Mapping Techniques Conference in Salt Lake City, Utah on Monday May 20. The conference is hosted by the U.S. Geological Survey to exchange technical expertise concerning geologic mapping and the creation of geologic map databases. Nathan Eaton Indiana Geological Survey Jerry Weisenfluh Kentucky Geological Survey Jim McDonald Ohio Geological Survey

Overview MIDCARB objectives and design SDE design and functionality ArcIMS deployment The presentation included a brief discussion of the MIDCARB project design and objectives, the use of ESRI’s SDE database technology for managing distributed databases, and the use of ESRI’s ArcIMS software for integrating those data.

MIDCARB Consortium The MIDCARB project is a consortium of the five state geological surveys of Ohio, Indiana, Illinois, Kansas, and Kentucky. It is funded by DOE’s National Energy Technology Laboratory.

MIDCARB Project Goals Characterize Major CO2 Sources Quantity, Quality, Location Characterize Potential Sequestration Sites Geology and Reservoir Characteristics Develop Relational/Spatial Databases Local and Regional Reporting Levels Supply this data to the public For use as tools in cost/feasibility analyses, etc. The goals of the MIDCARB project are 1) to develop and organize critical information relative to CO2 sources (primarily electric generating facilities) and potential sequestration sites for it, 2) to develop the information technology needed to access, query, model, analyze, display and distribute natural resource data related to carbon management across five states, various organizations and numerous databases, and 3) to make this information accessible to users on the world wide web.

... MIDCARB Data Relationship Model Internet (html browser accessible) Query & Reporting Applications (ArcIMS, SDE, & Java) Each state has well established relational and spatial databases that relate to potential sequestration reservoirs. However, each of these databases have different design specifications and, moreover, are continuously updated by the respective state organizations. The project strategy was, therefore, to utilize a distributed database structure to facilitate data maintenance, and develop software tools to integrate and interact with the data in a web environment. ...

Spatial Database Design ArcView ArcIMS RDBMS SDE CLIENTS ArcMap ESRI’s Spatial Database Engine (SDE) is a database technology for managing spatial data. The software utilizes an existing RDBMS (e.g. Oracle or SQLServer) for data storage, and manages requests for information from a variety of other ESRI clients. The project elected to utilize this software solution because each of the organizations were using some or all of the ESRI products and because of the diverse off-the-shelf client support. ArcExplorer

KGS Spatial Database SDE Basemaps Hazards Geology Land Use Water Data Petroleum Data Coal Data The MIDCARB project coincided with efforts at the Kentucky Geological Survey to construct its own SDE database. Several design elements evolved during the development process. Existing spatial data covered a wide spectrum of natural resource themes and were large in number. It was decided to maintain these data in a single SDE database, using subtables for organizing the data thematically. This approach alleviated the necessity for users to make multiple database connections required by a multiple database scenario. Data layers were prepared in a single coordinate system and datum, NAD83 decimal degrees, to simplify integration. Features were added as SDE layers - no geodatabase functionality has yet been enabled. The KGS enterprise point databases (oil, gas, coal, and water well and measurement locations) that are maintained in the RDBMS were spatially enabled in SDE by adding their location information as SDE layers. Infrastructure Sequestration

Database Organization Using a single SDE database presents logistical challenges with respect to data access. This image shows an example of the connection dialog for ESRI’s ArcMap software. The potentially large number of SDE themes makes it difficult for users to identify which themes to use. KGS developed a table naming schema to alleviate this problem. Thematic groupings are added to SDE tables named with a four character code (e.g., COAL, GEOL). Feature classes are named with a two letter state prefix, followed by a three character scale integer, and a meaningful theme name. This format results in a connection list that is sorted by theme type, followed by scale and name, with each designation nearly in vertical alignment. Most RDBMS systems limit such table names to approximately 32 characters.

Pre-clipped features to facilitate queries Another challenge for managing SDE data relates to the potentially large size of state-wide databases. For example, the KGS 1:24,000 geologic map database, when complete, will comprise707 detailed vector datasets that have been edgematched and joined. While SDE does support spatial queries using a tiling scheme, it will not be effective with such databases because many merged features can cover as much as 30% of the state. KGS decided to pre-intersect these complex feature classes with commonly queried geographic extents (i.e., county and quadrangle outlines). This not only facilitates faster queries, but simplifies the process of preparing finished map layouts.

Custom Geologic Map Query This slide shows a geologic map of two Kentucky counties produced by a single SDE query of the seamless geologic map database. Woodford County Fayette County

Future SDE Work Application tools for querying multiple themes Symbol palettes derived from database A lot of testing for robustness Constructing custom maps by SDE query is efficient, but if multiple themes are involved, the query needs to be issued for each theme. A query tool for each client application that collects the query criteria, then iterates through selected themes would be useful. Large, seamless databases, like the Kentucky geologic maps, present challenges for feature symbolization, particularly for polygon map units. Because each of the ESRI clients uses a different style format, this problem is exacerbated. This calls for a database solution that can construct symbol palettes on the fly. Most of the MIDCARB feature themes are relatively simple - point locations and simple geographic outlines. Serving complex and large spatial databases from distributed nodes will require extensive testing for robustness.

MIDCARB ArcIMS Deployment For MIDCARB, we have chosen to start with an html ArcIMS template and customize it using html and some JavaScript. This slide shows the MIDCARB home page with a graphic link to the IMS service. www.midcarb.org

Spatial Database Connections RDBMS Data Distribution and Analysis SDE Illinois GIS Integration Web browser SDE Indiana SDE ArcIMS Kansas The MIDCARB project integrates spatial data by connecting to each state’s enterprise-scale geodatabase using ArcIMS. ArcIMS simply displays the map layers in a single map interface. It is transparent to the user that map layers are being loaded from more than one location (both from a design and efficiency perspective.) SDE Kentucky SDE Ohio

Table of Contents Customization One of the most difficult issues associated with integrating multiple datasets from several states is creating a user interface that simplifies that list of available map layers. To overcome this problem we created a hybrid Table of Contents (TOC) and Legend using html and some simple JavaScript. Table of Contents layers have been grouped into subject areas to help users find layers of interest. The subject titles at the top of the TOC have hotlinks to scroll to the respective part of the legend. Click Now to demonstrate the scrolling We also added additional functionality to the TOC by incorporating hotlinks to metadata files (where metadata is available) from the layer names. Eventually, we will add hot links to metadata for all layers listed in the TOC.

Legend Customization GIF-based legends To provide additional capability from the TOC, we have added GIF images which show preformatted legends. When a layer is checked on, its symbol appears below the layer name in the TOC. To make this work, we created a GIF image of each layers symbology. The GIFs are then loaded dynamically based on whether on not a particular layer is checked on. This enhancement saves screen space and gives us more room for the main map frame.

Metadata Access We are currently experimenting with ways to distribute the GIS datasets associated with the layers shown in the interactive map. Right now we have an html download page where users can download a few datasets and access metadata.

Tabular Data Access In addition to integrating spatial data from five states, the MIDCARB project has successfully integrated tabular RDBMS data from each state in the consortium. The following slides illustrate how we are integrating tabular data. Lets look at CO2 sources in IL. First, zoom in to an area of interest using the Zoom In Tool. Next, choose the Identify Tool and click on a power plant. Note the information listed in the Results Window. These are the geodatabase featureclass attributes. Now, for some real data integration. First select the hotlink tool and click on the power plant again. This time a ColdFusion request is sent to a ColdFusion application server in Kansas. The ColdFusion application server sends a request to the appropriate database, in this case an Oracle server in Illinois. The Kansas ColdFusion application server then generates a report based on the Illinois data sends the report back to the Web client in the form of an html report. Finally, data from the Illinois RDBMS is returned to the Kansas server for creation of a real time graph.

ColdFusion Data Pathways RDBMS SDE Illinois (Oracle) Return Results Query RDBMS SDE Indiana (SQL Server) ColdFusion Request ColdFusion Report Web browser CF Server SDE Kansas (Oracle) MIDCARB has successfully linked tabular databases from the five cooperating states using a Web application development tool called ColdFusion. Recently purchased by Macromedia, formerly owned by Alliare, ColdFusion provides the developer with a custom set of html (CFML) tags that allow the developer to easily interact with databases to build dynamic Web pages. The first report on the previous slide is generated dynamically using a ColdFusion report. The diagram shows how requests from the Web browser, in this case initiated by clicking on a power plant with the hotlink tool, travel back to the ColdFusion server in Kansas. The ColdFusion server then queries appropriate databases, in this case the Illinois Oracle database. Results of the query are returned to the ColdFusion server in Kansas and then delivered as a pure html report to the Web browser. ArcIMS SDE Kentucky (SQL Server) Ohio SDE (SQL Server)

Distributed Coal Themes Some kinds of feature classes are not conducive to distributed data storage. This structure contour map of the Illinois basin Springfield coal was compiled from three separate shapefiles and is stored on a single server at Kentucky.

ArcIMS - Level of Detail At the present time, the MIDCARB project is using a single ArcIMS instance (at Kansas, with replication in Indiana) to integrate information from the five SDE databases. Future developments may include multiple IMS services, where the initial site passes control to another server at some zoom level to show more detailed information. KS ArcIMS OH ArcIMS IL SDE IN SDE KY SDE OH SDE OH SDE

Distributed Petroleum Fields Indiana Illinois Petroleum field outlines are a good example of how similar datasets from different states are being integrated across state lines. Note symbology in TOC and consistency of data across state lines. The value in being able to compare this type of data has huge implications for petroleum exploration around the margins of states (which would be one of many secondary benefits of projects like MIDCARB).

Future IMS Work Database views to fully integrate themes Pre-IMS query interface to simplify the user view ArcIMS 4.0 metadata facility While the distributed nature of the data in MIDCARB is relatively transparent to the user, the fact that similar themes are coming from different feature classes is not, as each has an entry in the TOC. It would be advantageous to show only one legend item for the collection of related feature themes, and SDE database views may be one way of achieving this. The number of potential themes available to MIDCARB users is large and continuing to grow. Much work needs to be done to simplify the user interface so that only desired themes are shown. This could be accomplished with some kind of query interface to collect information from users about what they want to see. The current implementation of the MIDCARB IMS service uses version 3.x software. When version 4.0 is available, its new capability that accesses metadata from an SDE database will be explored.

Summary ArcIMS and ArcSDE appear to be a robust environment for distributed data SDE promises interesting opportunities for institutional data sharing (outside of IMS) Challenge for sharing large and complex databases is design as much as technology The use of SDE and IMS to integrate distributed spatial and tabular databases for the MIDCARB project has proven to be quite robust. Moreover, the ease of sharing spatial data across institutions using this technology promises new opportunities for natural resource organizations. While there are technical challenges for implementing such a system, the greatest challenge is in creating a design that is clear and easy to use.