1. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey An Introduction to an Index of Geospatial Web Services

2. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Mapdex Outline Advantages of building an index of dynamic map services Discovering published mapservices Index mapping capabilities via XML storage Summarize results Search Index Updating the index (new servers, new layers, statistics, stability)

3. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Why do Geographers/planners/scientists care if there is an easy way to find spatial data over the internet? “Early GIS was impeded by the lack of good algorithms and powerful computers to analyze data and by the difficulties associated with digitizing, which were such that 80% or more of a project’s resources were often consumed in the task of converting paper records to digital databases. Today, a new source of frustration has emerged. Despite all of the on-line digital data now available and the vastly increased power of GIS, it is still common to spend 80% or more of a project’s resources on searching, discovering, retrieving, and reformatting data.” Forward by Michael Goodchild in Internet GIS by Peng and Tsou 2003

4. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Many groups have built catalogs of spatial data. –State Clearinghouses –Federal Government Initiatives They all rely on manual publishing of metadata –More Metadata, more information –Less actual data, more work for the provider

5. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services There are several various catalogs of spatial data available to the user –Geospatial One-Stop, Geography network www.geo-one-stop.gov (Government sponsored) www.geographynetwork.com (industry sponsored) –Requires users to publish metadata Geospatial One-Stop FGDC and GOS: Working as One to Build the NSDI Date: June 7, 2004 Sharon Shin sharon_shin@fgdc.gov

6. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services The fundamental difference between Mapdex and other geospatial catalogs is that Mapdex searches ALL available map services that are openly published on the Internet via ArcIMS or through WMS compliant servers serving through version 1.1.1. Other data catalogs rely on the owners of the data to “publish” metadata about the mapservices and the underlying data to the catalog ArcIMS server administrators can enable authentication to not allow Mapdex (and anybody else) to harvest (or use) information about their mapservices without a username and password

7. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Advantages of Building an Index of Dynamic Map Services Current Mapdex Statistics Successfully Indexed: –Almost 1,500 servers containing… –Approximately 25,000 live mapping services (ArcIMS Mapservices spanning… –Over 370,000 spatial layers (300,000 feature layers, 70,000 raster layers) –Over 100 WMS 1.1.x services over 60 servers

8. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work From ESRI: www.esri.com/software/internetmaps/index.html

9. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work ArcXML programmers guide ESRI Systemdesginstrategies2004.pdf

10. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work

11. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work

12. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work Heteractis magnifica

13. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work Each server has multiple mapservices available from it: If you post a getclientservices XML request to the ArcIMS server… http://seamless.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=catalog&ClientVersion=4.0.1 You get an XML response containing key information about the mapservices that are available to be used by any client.

14. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey How Map Services Work Each mapservice has multiple spatial layers that are available from it: If you post a getserviceinfo XML request to the ArcIMS server… http://seamless.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=geology&ClientVersion=4.0.1 You get an XML response containing key information about the mapservice (in this case Geology). Information like: extent, projection, layer envelope, layer name, layer type, column name, column type, etc.

15. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey The Next Step: Building of the Index These XML documents that describe the inner workings of the mapservice are a wealth of knowledge If we could somehow harvest the mapserver metadata we would have a vast database of: servers, services, layers, columns, etc… If we know the name of the server that is running the mapserver instance (ArcIMS), we can harvest the rest of the information programmatically

16. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Finding servers that publish mapservices Number one indexing application: GOOGLE The main way Google builds its index is by finding pages that link to other pages Mapdex finds map servers that publish data in a specific format (web service) and are indexed by Google Most dynamic mapping applications can by identified by either vendor specific content or common key words (zoom, pan, map) within the html page

17. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Finding servers that publish mapservices In order to build the index of servers that contain mapservices we need to programmatically search the internet for possible servers We can get close by using a traditional search engine like Google. We can automate the search by making use of Google’s API (application programmers interface) web service –We can search Google programmatically by using the API

18. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Discovering Published mapservices 1.Search Google for possible mapserver sites. 2.For each server, check to see if it is publishing mapservices. 3.If it is a match, then save the response into the database.

19. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities To make the database of mapservers searchable/queryable we need to either: Parse the xml file and store the key results into a database Or Store the xml file directly into a database that fully supports XML integration

20. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities Mapdex is using Oracle’s XML DB version 9.2 From: http://otn.oracle.com/tech/xml/xmldb/9.2.0.2.0/index.html

21. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities In order to store ArcXML into Oracle’s XML DB, schemas (definition of the xml data model) had to be built from the ground up to match ArcXML syntax and protocol –ArcIMS uses dtd’s as its definition Once stored….

22. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities This Becomes This….

23. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities Once we store the server capabilities xml document we can produced some interesting statistics about the state of mapservers…

24. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of Server Capabilities Server domain statistics

25. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of mapservice configuration We employ the same technology to store information about the mapservices that are contained in the ~1,500 distinct mapservers For each mapservice within each mapserver we request detailed mapservice information and store it within the database.

26. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of mapservice configuration So this XML document becomes…

27. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Storage of mapservice configuration Now we have access to over 370,000 GIS layers This includes approximately 300,000 vector layers (point, line, polygon) and 70,000 raster layers (DEM, orthophotos, satellite images, scanned maps)

28. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Spatial envelopes for mapservice layers Each mapservice includes a global envelope. This envelope is a bounding box that identifies the maximum extent of all the layers in the mapservice. Each layer contains the envelope for all the features for a particular layer.

29. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Spatial Envelopes for mapservice Layers The catch--not all mapservices declare their projection In terms of layers that means of the 365,000 layers that Geodiscovery has indexed, approximately 150,000 layers either have their projection set or are assumed geographic

30. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Spatial envelopes for mapservice layers Why is this important… If we know the mapservice projection we can reproject the various layer envelopes to a common projection (geographic- latitude/longitude) Now we can query mapservice layers by geographic location (all layers within state, county, zip code, etc.)

31. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Let’s Search It! A simple front end was created to allow easy searching and quick mapping of the spatial data

32. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

33. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Let’s Search It!

34. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Let’s Search It!

35. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Let’s Search It!

36. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Interesting Statistics

37. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Interesting Statistics

38. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Updating the Index Mapdex is continually updating and discovering new services Mechanisms are in place to: –Discover new servers –Discover new mapservices –Discover new layers –Remove old servers & services –Update envelope index

39. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey Future Trends After repeated indexing and searching we will have the ability to provide: –Server stability statistics (its up time) We are currently building an integrated mapviewer that can be used to view multiple layers from multiple servers in realtime We are building the infrastructure that will allow people to build dynamic targeted portals that package mapserver layers from multiple mapservices –USGS DEM, landcover, & local data

40. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

41. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

42. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

43. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

44. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

45. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

46. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

47. Jeremy D. Bartley www.mapdex.org Kansas Geological Survey