Jeremy D. Bartley Kansas Geological Survey An Introduction to an Index of Geospatial Web Services

Jeremy D. Bartley Kansas Geological Survey Mapdex Outline Advantages of building an index of dynamic map services Discovering published mapservices Index mapping capabilities via XML storage Summarize results Search Index Updating the index (new servers, new layers, statistics, stability)

Jeremy D. Bartley Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Why do Geographers/planners/scientists care if there is an easy way to find spatial data over the internet? “Early GIS was impeded by the lack of good algorithms and powerful computers to analyze data and by the difficulties associated with digitizing, which were such that 80% or more of a project’s resources were often consumed in the task of converting paper records to digital databases. Today, a new source of frustration has emerged. Despite all of the on-line digital data now available and the vastly increased power of GIS, it is still common to spend 80% or more of a project’s resources on searching, discovering, retrieving, and reformatting data.” Forward by Michael Goodchild in Internet GIS by Peng and Tsou 2003

Jeremy D. Bartley Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Many groups have built catalogs of spatial data. –State Clearinghouses –Federal Government Initiatives They all rely on manual publishing of metadata –More Metadata, more information –Less actual data, more work for the provider

Jeremy D. Bartley Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services There are several various catalogs of spatial data available to the user –Geospatial One-Stop, Geography network (Government sponsored) (industry sponsored) –Requires users to publish metadata Geospatial One-Stop FGDC and GOS: Working as One to Build the NSDI Date: June 7, 2004 Sharon Shin

Jeremy D. Bartley Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services The fundamental difference between Mapdex and other geospatial catalogs is that Mapdex searches ALL available map services that are openly published on the Internet via ArcIMS or through WMS compliant servers serving through version Other data catalogs rely on the owners of the data to “publish” metadata about the mapservices and the underlying data to the catalog ArcIMS server administrators can enable authentication to not allow Mapdex (and anybody else) to harvest (or use) information about their mapservices without a username and password

Jeremy D. Bartley Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Current Mapdex Statistics Successfully Indexed: –Almost 1,500 servers containing… –Approximately 25,000 live mapping services (ArcIMS Mapservices spanning… –Over 370,000 spatial layers (300,000 feature layers, 70,000 raster layers) –Over 100 WMS 1.1.x services over 60 servers

Jeremy D. Bartley Kansas Geological Survey How Map Services Work From ESRI:

Jeremy D. Bartley Kansas Geological Survey How Map Services Work ArcXML programmers guide ESRI Systemdesginstrategies2004.pdf

Jeremy D. Bartley Kansas Geological Survey How Map Services Work

Jeremy D. Bartley Kansas Geological Survey How Map Services Work

Jeremy D. Bartley Kansas Geological Survey How Map Services Work Heteractis magnifica

Jeremy D. Bartley Kansas Geological Survey How Map Services Work Each server has multiple mapservices available from it: If you post a getclientservices XML request to the ArcIMS server… You get an XML response containing key information about the mapservices that are available to be used by any client.

Jeremy D. Bartley Kansas Geological Survey How Map Services Work Each mapservice has multiple spatial layers that are available from it: If you post a getserviceinfo XML request to the ArcIMS server… You get an XML response containing key information about the mapservice (in this case Geology). Information like: extent, projection, layer envelope, layer name, layer type, column name, column type, etc.

Jeremy D. Bartley Kansas Geological Survey The Next Step: Building of the Index These XML documents that describe the inner workings of the mapservice are a wealth of knowledge If we could somehow harvest the mapserver metadata we would have a vast database of: servers, services, layers, columns, etc… If we know the name of the server that is running the mapserver instance (ArcIMS), we can harvest the rest of the information programmatically

Jeremy D. Bartley Kansas Geological Survey Finding servers that publish mapservices Number one indexing application: GOOGLE The main way Google builds its index is by finding pages that link to other pages Mapdex finds map servers that publish data in a specific format (web service) and are indexed by Google Most dynamic mapping applications can by identified by either vendor specific content or common key words (zoom, pan, map) within the html page

Jeremy D. Bartley Kansas Geological Survey Finding servers that publish mapservices In order to build the index of servers that contain mapservices we need to programmatically search the internet for possible servers We can get close by using a traditional search engine like Google. We can automate the search by making use of Google’s API (application programmers interface) web service –We can search Google programmatically by using the API

Jeremy D. Bartley Kansas Geological Survey Discovering Published mapservices 1.Search Google for possible mapserver sites. 2.For each server, check to see if it is publishing mapservices. 3.If it is a match, then save the response into the database.

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities To make the database of mapservers searchable/queryable we need to either: Parse the xml file and store the key results into a database Or Store the xml file directly into a database that fully supports XML integration

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities Mapdex is using Oracle’s XML DB version 9.2 From:

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities In order to store ArcXML into Oracle’s XML DB, schemas (definition of the xml data model) had to be built from the ground up to match ArcXML syntax and protocol –ArcIMS uses dtd’s as its definition Once stored….

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities This Becomes This….

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities Once we store the server capabilities xml document we can produced some interesting statistics about the state of mapservers…

Jeremy D. Bartley Kansas Geological Survey Storage of Server Capabilities Server domain statistics

Jeremy D. Bartley Kansas Geological Survey Storage of mapservice configuration We employ the same technology to store information about the mapservices that are contained in the ~1,500 distinct mapservers For each mapservice within each mapserver we request detailed mapservice information and store it within the database.

Jeremy D. Bartley Kansas Geological Survey Storage of mapservice configuration So this XML document becomes…

Jeremy D. Bartley Kansas Geological Survey Storage of mapservice configuration Now we have access to over 370,000 GIS layers This includes approximately 300,000 vector layers (point, line, polygon) and 70,000 raster layers (DEM, orthophotos, satellite images, scanned maps)

Jeremy D. Bartley Kansas Geological Survey Spatial envelopes for mapservice layers Each mapservice includes a global envelope. This envelope is a bounding box that identifies the maximum extent of all the layers in the mapservice. Each layer contains the envelope for all the features for a particular layer.

Jeremy D. Bartley Kansas Geological Survey Spatial Envelopes for mapservice Layers The catch--not all mapservices declare their projection In terms of layers that means of the 365,000 layers that Geodiscovery has indexed, approximately 150,000 layers either have their projection set or are assumed geographic

Jeremy D. Bartley Kansas Geological Survey Spatial envelopes for mapservice layers Why is this important… If we know the mapservice projection we can reproject the various layer envelopes to a common projection (geographic- latitude/longitude) Now we can query mapservice layers by geographic location (all layers within state, county, zip code, etc.)

Jeremy D. Bartley Kansas Geological Survey Let’s Search It! A simple front end was created to allow easy searching and quick mapping of the spatial data

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey Let’s Search It!

Jeremy D. Bartley Kansas Geological Survey Let’s Search It!

Jeremy D. Bartley Kansas Geological Survey Let’s Search It!

Jeremy D. Bartley Kansas Geological Survey Interesting Statistics

Jeremy D. Bartley Kansas Geological Survey Interesting Statistics

Jeremy D. Bartley Kansas Geological Survey Updating the Index Mapdex is continually updating and discovering new services Mechanisms are in place to: –Discover new servers –Discover new mapservices –Discover new layers –Remove old servers & services –Update envelope index

Jeremy D. Bartley Kansas Geological Survey Future Trends After repeated indexing and searching we will have the ability to provide: –Server stability statistics (its up time) We are currently building an integrated mapviewer that can be used to view multiple layers from multiple servers in realtime We are building the infrastructure that will allow people to build dynamic targeted portals that package mapserver layers from multiple mapservices –USGS DEM, landcover, & local data

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey

Jeremy D. Bartley Kansas Geological Survey