10/31/2007 Effective discovery of geospatial data: a geospatial catalogue perspective Dr. Yuqi Bai Research Assistant Professor Center for Spatial Information Science and Systems George Mason University Washington/Northern Virginia Chapter of IEEE/GRSS Technical Meeting Wednesday, October 31, 2007

Page 2 CSISS Center for Spatial Information Science and Systems Contents Geospatial data discovery problems Geospatial data discovery systems –System architectures –Referenced Metadata standards –Referenced Catalogue Service standard Geospatial Catalogue Federation –Case study: GMU CSISS CFS product –Main Challenges –Proposed federation strategies –Product system –Discussion GMU CSISS CSW/CFS Applications Summary

Page 3 CSISS Center for Spatial Information Science and Systems Background Large volume of geospatial data –has been accumulated over the last several decades through mapping, survey and observation Petabyte level –NASA EOSDIS project is expected to archive one petabyte per year of raw data that are distributed among data centers. –On November 20, 2003, the NASA Land Processes Distributed Active Archive Center (LP DAAC) data archive holdings crossed the one petabyte threshold in volume*. –1 petabyte = 1*10**15 bytes = 8*10**6 Second * 1 Gb/s (~ days) = 8*10**7 Seconds * 100 Mb/s (~925.9 days) *

Page 4 CSISS Center for Spatial Information Science and Systems Significant Problem and Question Problem –Large volume of geospatial data has to be maintained in few data centers, while these data are highly needed in all research carried out by research staff, professors and students in every college, university and government agencies. geospatial data Question –How can users be helped in evaluating the fitness of a particular data set, among hundreds of collections and millions of granules, for user for their specific decision or assessment? End user

Page 5 CSISS Center for Spatial Information Science and Systems Geospatial Data Discovery Mechanism Overview –Step 1: Organizing textual information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of every piece of data set Metadata (data about data) geospatial data geospatial metadata discovery interface End user –Step 2: Providing catalogue discovery interface for this metadata information to end users –Step 3: Enabling direct data download or customization through online software modules, or “services”

Page 6 CSISS Center for Spatial Information Science and Systems Geospatial Metadata The metadata required for effective data management varies with the type of data and context of use. –Standards + Profiles geospatial data geospatial metadata discovery interface End user Standards: –ISO ISO 15836:2003 –Dublin Core metadata element set –Stage code: ( ) ISO 19115:2003 –Geographic information -- Metadata –Stage code: ( ) ISO 19115:2003/Cor 1:2006 –Stage code: ( ) ISO –Geographic information -- Metadata -- Part 2: Extensions for imagery and gridded data –Under development –Stage code: ( ) ISO 19119:2005 –Geographic information -- Services –Stage code: ( ) ISO 19139:2007 –Geographic information -- Metadata -- XML schema implementation –Stage code: ( )

Page 7 CSISS Center for Spatial Information Science and Systems Geospatial Metadata (Cont.) The metadata required for effective data management varies with the type of data and context of use. –Standards + Profiles geospatial data geospatial metadata discovery interface End user Standards: –US FGDC-STD –Content Standard for Digital Geospatial Metadata FGDC-STD –Content Standard for Digital Geospatial Metadata: Extensions for Remote Sensing Metadata –NASA ECS Science Metadata

Page 8 CSISS Center for Spatial Information Science and Systems Geospatial Metadata Discovery Interface geospatial data geospatial metadata discovery interface End user The discovery interface varies with the type/structure of the underlying metadata and context of use. Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS

Page 9 CSISS Center for Spatial Information Science and Systems Geospatial Metadata Discovery Interface (Cont.) The discovery interface varies with the type/structure of the underlying metadata and context of use. geospatial data geospatial metadata discovery interface End user Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS –Web page navigation with limited search functionalities E.g. NASA GCMD

Page 10 CSISS Center for Spatial Information Science and Systems Geospatial Metadata Discovery Interface (Cont.) The discovery interface varies with the type/structure of the underlying metadata and context of use. geospatial data geospatial metadata discovery interface End user Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS –Web page navigation with limited search functionalities E.g. NASA GCMD –Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG)

Page 11 CSISS Center for Spatial Information Science and Systems Geospatial Data Discovery Process through EOS Data Gateway (EDG) System

Page 12 CSISS Center for Spatial Information Science and Systems Geospatial Metadata Discovery Interface (Cont.) The discovery interface varies with the type/structure of the underlying metadata and context of use. geospatial data geospatial metadata discovery interface End user Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS –Web page navigation with limited search functionalities E.g. NASA GCMD –Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG) LP DAACGES DISC

Page 13 CSISS Center for Spatial Information Science and Systems Geospatial Metadata Discovery Interface (Cont.) The discovery interface varies with the type/structure of the underlying metadata and context of use. geospatial data geospatial metadata discovery interface End user Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS –Web page navigation with limited search functionalities E.g. NASA GCMD –Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG) –Web-based GUI with enhanced search functionalities, with proprietary API interface E.g. NASA ECHO –IIMSAQL Query Language ECHO Service Core GES DISCLP DAAC GMU CSISS ECHO OGC Wrapper

Page 14 CSISS Center for Spatial Information Science and Systems geospatial data geospatial metadata discovery interface End user 15 Terabytes Images GMU CSISS OGC Catalogue Service Core ebRIM WrapperISO WrapperOGC Core Data Download GeoBrain Online Analysis System (GeOnAS) Geospatial Metadata Discovery Interface (Cont.) The discovery interface varies with the type/structure of the underlying metadata and context of use. Fromthe user’s point of view: –Simple web page navigation with no search functionality E.g. THREDDS –Web page navigation with limited search functionalities E.g. NASA GCMD –Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG) –Web-based GUI with enhanced search functionalities, with proprietary API interface E.g. NASA ECHO –IIMSAQL Query Language –Web-based GUI with enhanced search functionalities, with open API interface E.g. GMU CSISS/LAITS CSW

Page 15 CSISS Center for Spatial Information Science and Systems GMU CSISS/LAITS CSW - Designed and Developed from Aug Support OGC CSW and 2.0.2

Page 16 CSISS Center for Spatial Information Science and Systems Geospatial Catalogue Service Standard OGC Catalogue Service is the only available standard –specifies the interfaces between clients and catalogue services through the presentation of abstract and implementation-specific models. –Catalogue Service and its clients OGC’s perspective: –Catalogue Service supports the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. –Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. –Catalogue services are required to support the discovery of and binding to registered information resources within an information community. geospatial data geospatial metadata discovery interface End user Catalogue Service Catalogue Service Client

Page 17 CSISS Center for Spatial Information Science and Systems Geospatial Catalogue Service Standard (Cont.)

Page 18 CSISS Center for Spatial Information Science and Systems Current Status of Geospatial Catalogue System geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper

Page 19 CSISS Center for Spatial Information Science and Systems New Problems and Questions geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper

Page 20 CSISS Center for Spatial Information Science and Systems New Problems and Questions geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper

Page 21 CSISS Center for Spatial Information Science and Systems New Problems and Questions geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper

Page 22 CSISS Center for Spatial Information Science and Systems New Problems and Questions geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper

Page 23 CSISS Center for Spatial Information Science and Systems New Problems and Questions Different agencies have developed their own geospatial catalogues to facilitate discovery, access, and sharing of large volumes of geospatial data, either observed satellite images or simulation data. These geospatial catalogues are becoming accessible online through their query interfaces. Scientists who conduct multi-disciplinary research may need to search multiple catalogues in order to find the data they need. Such work is very time-consuming and tedious, especially when the catalogues may use different metadata models and catalog interface protocols. It is very desirable if those catalogues can be integrated into a catalogue federation, which will present a well-known metadata model and interface protocol to users and hide the complexity and diversity of the affiliated catalogues behind the interface. With the federation, users only need to work with the federated catalogue to find the data they need instead of working with catalogues individually. Catalogue federation service - integrating multiple legacy catalogues to facilitate distributed and integrated data discovery.

Page 24 CSISS Center for Spatial Information Science and Systems Federation Context geospatial data geospatial metadata discovery interface End user GMU CSISS OGC Catalogue Core ebRIMISOOGC Core ECHO Service Core GMU CSISS ECHO OGC Wrapper Catalogue Federation

Page 25 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System Federation Case Study – GMU CSISS CFS System Community Catalogues End user GMU CSISS Catalogue Federation Service NASA ECHO GMU CSISS OGC CSW DOE Earth System Grid Simulation Data Catalogue DOE Earth System Grid Simulation Data Catalogue Discovery Interface GMU GUI Third Party System

Page 26 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) We analyzed the following aspects of each catalogue –Metadata Conceptual Model –Query Language –Communication Protocol

Page 27 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: –1. Protocol Adaptation GMU CSW and the ESG catalogue support HTTP protocol (GET/POST) binding, while NASA ECHO uses SOAP to maintain the connection with the clients. The federation server should use the correct protocol when communicating with each Catalogue service. The protocol the clients may use to talk to the federation server itself is another concern. After all protocols have been defined and identified, the federation server should support protocol adaptation internally.

Page 28 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: –2. Query Dispatching The federation server is responsible for dispatching a query to the affiliated catalogue services. A dispatching model should be defined to deal with the following issues: Transparency: Whether the federation user is aware of these affiliated catalogue services and whether users can define which catalogue services are of interest in their queries. Sequence: Whether the federation server dispatches the users’ queries to these affiliated catalogue services in a predefined sequence, whether this sequence can be changed in runtime, and whether the federation users can define this sequence in their queries.

Page 29 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: –3. Query Translation: The translation of queries is another major issue. The federation has to deal with the following problems: Metadata Query Objects: The metadata objects queried using one set of query criteria may not have counterparts in another schema. For example, the federation service cannot fulfill queries for objects defined in GMU CSW and NASA ECHO for those simulation- specific metadata objects referenced only in the ESG catalogue schema. Another issue is that the same registry object has different names, in different schemes, e.g., Granule in NASA ECHO versus DataGranule in GMU CSW. Query Format: Both GMU CSW and the ESG Catalogue accept queries in OGC Filter format, while ECHO only accepts IIMSAQL format. The federation server needs to transform an individual query into the different proprietary formats. The spatial query criterion and temporal query criterion are expressed differently in the NASA ECHO granule query payload and the GMU CSW granule query payload. Query Language Functionality: Some complex query predicates in one query language cannot be identically expressed in another one. For example, the OGC Filter specification supports nested Boolean queries. Such queries can be supported at best with difficulty on ECHO IIMSAQL, and some cannot be supported at all.

Page 30 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: –4. Results Integration: Catalogue query results from multiple Catalogue Services may need to be integrated before being sent back to users. As these different sets of metadata results may not use the same schema, the rules the federation server uses to re-organize metadata information while keeping the original content should be well designed. Furthermore, whether the clients can define the format of the query result of interest and, if so, how, also needs to be addressed.

Page 31 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) We propose the following federation strategies: –1. Protocol Adaptation As this federation is supposed to provide a single access point to multiple, autonomous information sources, it may follow the mediator-wrapper architecture, where the federation works as a mediator, and wrappers may be deployed for communicating with specific catalogue services if protocol adaptation is needed. –2. Query Dispatching 1) Opaque: In this scenario, the federation service fully controls the distributed query process, with the clients having no awareness of the affiliated Catalogue Services. 2) Translucent: The federation service may expose the affiliated Catalogue Services to the users, but the users can define neither which Catalogue Services their query can be forwarded to nor the sequence of queries. 3) Transparent: The federation service may expose the affiliated Catalogue Service to the users, and the user chooses which catalogue service shall be used for each inquiry and the order of the inquiries.

Page 32 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Proposed federation strategies: –3. Query Translation Query Translation in the federation has two aspects: semantic and syntactic. A federation usually maintains a global schema that is exposed to end-users. Metadata attribute terms in user queries always follow this global schema. Before being dispatched to an underlying affiliated catalogue service, they should be transformed appropriately. This transformation logically involves four layers: metadata term, query criterion, query criteria, and query payload, as shown in the following picture.

Page 33 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Proposed Federation Strategy –4. Query Result Integration A federation service needs to integrate query results from multiple underlying Catalogue Services before sending them back to the clients. It may choose to implement one of three kinds of integration mechanisms. –Opaque: In this case, the federation service defines, maintains and advertises a unique information model. Each query result from affiliated Catalogue Services should, if necessary, be transformed to this information model. The original metadata information can be kept in the final transformed query results. –Translucent: The federation service does not maintain a complete, unique information model but defines a common subset of metadata objects that are supported by all the affiliated Catalogue Services, such as name, and spatial and temporal range. The federation service transforms only this part of the metadata information, while the remaining embedded original metadata information remains unchanged in the final response. –Transparent: The federation service has no role in metadata integration. All the query results from affiliated Catalogue Services are simply grouped together, keeping the original metadata formats. In this scenario, the users are supposed to analyze each result fetched from federation service, since the results may not all conform to the same schema even though grouped together in one response.

Page 34 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Federation System Architecture The GMU CFS consists of two types of components: Mediator and Wrapper. The Mediator is a key component of the GMU CFS. It accepts user’s queries through an OpenGIS CSW query interface. For better modularity and sustainability, the GMU CFS has externalized the query translation and the result transformation module for ECHO to make a wrapper, i.e., OGC CSW for ECHO. This wrapper accepts client queries, in OGC Filter format, that are forwarded by the Mediator. They are transformed to ECHO IIMSAQL format by the Query Transformation module.

Page 35 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Federation System Context

Page 36 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Discussion –This GMU CFS system can integrate NASA ECHO, GMU CSW and the DOE ESG Simulation Data Catalogue. One advantage of its design is that CFS follows the OpenGIS Catalogue Service standard as the communication protocol with the underlying affiliated catalogue services. As long as new catalogue services follow this standard, they can easily be integrated into the federation system. –However, integrating new legacy catalogue services cannot be plug and play. Abstracting the specific information models, the catalogue registration mechanism, and query orchestration would be new issues to consider when scaling this federation beyond these three catalogue services.

Page 37 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Discussion (Cont.) –In the strategy described here, the specific information models need to be carefully evaluated before incorporation into the global schema and subsequent exposure to client users, when including new catalogue services. Efforts such as the ISO Technical Committee (TC) 211 Geospatial Metadata Standard 19115, FGDC Content Standard for Digital Geospatial Metadata (FGDC, 1998) and Dublin Core (DCMI, 2006) are attempting to standardize the geospatial metadata information model, but, in many cases, their use is voluntary. –A metadata crosswalk could be of help when mapping two distinct models, but is not very suitable for one-to-many mapping. –An ontology-based approach would provide a new way to create a global schema.

Page 38 CSISS Center for Spatial Information Science and Systems GMU CSISS CFS System (Cont.) Federation Case Study – GMU CSISS CFS System (Cont.) Discussion (Cont.) –New catalogue services must be registered manually. In the design presented here, the federation service discovers the underlying catalogue services at design time, rather than at run time. This strategy greatly simplifies the mechanism for federation, and lowers the complexity of implementation. –In fact, without an automatic way to integrate the information model, it does not make much sense to register the catalogue service automatically.

Page 39 CSISS Center for Spatial Information Science and Systems Publications Journal papers –Towards a Geospatial Catalogue Federation Service Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei. Photogrammetric Engineering and Remote Sensing (PE&RS). 2007, Vol.73, No.6, pp –A Taxonomy of Geospatial Services for Global Service Discovery and Interoperability Y. Bai, L. Di. Computers & Geosciences. (Under review). Book chapters –Catalogue Information Models L. Di, Y. Bai. Encyclopedia of Geographical Information Science. –Geospatial Image Metadata Catalogue Services Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei. Encyclopedia_of_Geoinformatics. Conferences –Serving Satellite Remote Sensing Data to User Community through the OGC Interoperability Protocols L.Di, W.Yang, Y.Bai. AGU 2005 Fall Meeting. –GEOSS Registry System: Enabling the Registering and Discovering of Geospatial Web Services Worldwide Y. Bai, L. Di, N. Doug, Y. Wei. AGU 2008 Fall Meeting.

Page 40 CSISS Center for Spatial Information Science and Systems GMU CSISS CSW/CFS Applications NASA NEHEA GeoBrain project –Mobilizing NASA EOS data and information through Web service and knowledge management technologies for higher-education teaching and research –PI: Liping Di at GMU –Co-I: 9 Educational partners –Task Lead: Peisheng Zhao at GMU

Page 41 CSISS Center for Spatial Information Science and Systems GMU CSISS CSW/CFS Applications (Cont.) NASA AIST Grid/OGC project –Integration of OGC and Grid Technologies for Earth Science Modeling and Applications –PI: Liping Di at GMU –Co-I: Piyush Mehrotra –NASA Ames Research Center Dean Williams –DOE LLNL –Task Lead: Aijun Chen at GMU

Page 42 CSISS Center for Spatial Information Science and Systems GMU CSISS CSW/CFS Applications (Cont.) GEOSS Component and Service Registry –Maintenance and enhancement of the GEOSS Registry for earth observation –PI: Liping Di at GMU –Task Lead: Yuqi Bai at GMU

Page 43 CSISS Center for Spatial Information Science and Systems GMU CSISS CSW/CFS Applications (Cont.) GEOSS GeoNetwork Clearinghouse Candidate System –Catalogue Service Interface for Web Portals Task Lead: Yuqi Bai at GMU

Page 44 CSISS Center for Spatial Information Science and Systems Summary Metadata and Catalogue system work behind the scenes to support geospatial data discovery Online discovery of textual metadata information has greatly facilitated direct discovery of large volumes of geospatial data Multiple metadata standards/profiles exist at the international level and national level OGC specifications (baseline/profiles) are main players regarding the definition of the the Catalogue’s behavior Heterogeneous Catalogue systems prevent further integration and interoperability It is desirable to have a Catalogue Federation to –provide a single interface to users, while –hiding the complexity and diversity of the affiliated catalogues behind the interface GMU CSISS Catalogue Federation product has been presented –Research findings –System design and implementation –Lessons learned GMU CSISS CSW and CFS systems have been successfully used in several national and international projects (funded by NASA, FGDC, and GEO)

Page 45 CSISS Center for Spatial Information Science and Systems Acknowledgements This study was supported by grants from NASA through the REASoN program (NNG04GE61A; Professor Liping Di, Principal Investigator). I appreciate my teammates for their kind support over the last three years. Thank you for your attention this afternoon. Happy Halloween !