1. ESIP Federated Search Cluster
Federated Space-Time Query for Earth Science Data Using OpenSearch Conventions
ESIP Federated Search Cluster
Chris Lynnes
Bruce Beaumont
Ruth Duerr
Hook Hua
et al.

2. Outline Finding Earth science data: why so difficult???
Federated search, past and present
Employing OpenSearch
Community conventions within OpenSearch
Client and server developments

3. Use Case: Volcanic Ash Plume
Studying spatio-temporal extent of the volcanic ash plume from Chaiten eruption, 2 May 2008
“Get me all available aerosol data for 2 May 2008, in the vicinity of 43 S, 73 W”
CNES (POLDER)
DataFed (Multiple)
ASDC (MISR, CALIPSO)
GES DISC (OMI, HIRDLS)
MODAPS (MODIS)
OBGP (SeaWiFS)
Scientists (experimental CALIPSO)
Aeronet (station-based)…

4. Finding satellite datasets today is tedious, hit-or-miss
Step 1: Search through multiple directories for the right datasets
“Did I find them all?”
Steps 2-N:
Foreach data_provider
Learn_search_interface()
Search_for_data_files()
Fetch_data_files()
Load_data_into_analysis_tool()
End foreach

5. EOSDIS Version 0 was an early federated search system
Earth Observing System Data and Information System
7 distributed data centers with search servers following the same protocol
Dataset level
File level
Drawbacks at the time
Slow
Today we have big pipes
Idiosyncratic protocol (Object Description Language over sockets)
Today we have http and a wealth of standards
Limited clients due to protocol and technology
Today we have web technologies galore

6. Federated search can be better today
Simple: facilitates adoption
Standards-based, but extensible
Machine-callable: enables clients
Embeddable
In web pages, documents, workflows, tools…

7. OpenSearch is a simple, extensible, embeddable, machine-callable convention
“a collection of simple formats for the sharing of search results”
OpenSearch Description Document
Describes a search engine so that it can be used by search clients (incl. Firefox and IE)
OpenSearch response elements
Extend syndication formats (e.g., RSS and Atom) with extra metadata in search results
Extensions
Have been proposed for Geospatial and Time queries

8. ESIP Conventions for Federated Space-Time Query
Earth Science Information Partners
Consortium of >90 organizations that collect, interpret and develop applications for remotely sensed Earth observation information
Clusters: focus groups to promote information exchange among organizations
Federated Search cluster to develop ESIP community conventions
Two-Step (Recursive) OpenSearch
Atom responses

9. Space-Time Data Query is a Two-Step Process
Search for datasets, then for files within selected datasets
Most dataset-level queries have
low "precision": precision = desiderata / (desiderata + dreck)
small results set (dozens)
Space-time granule queries for a given dataset have
large results set (tens of thousands), but
high precision
Combining the two in one step produces
mammoth results set (dozens * tens of thousands)
with low precision

10. Recursive OpenSearch begins with a dataset discovery phase
Granule Search
Client
Dataset Query Engine
OpenSearch Description Document Store
Granule Query Engine
dataset query
dataset results
with link to OpenSearch Description Document

11. Dataset results link to OpenSearch Description documents
Dataset Query Engine
OpenSearch Description Document Store
Granule Query Engine
Client
dataset query
Dataset
Discovery
dataset results
OpenSearch Description Request
OpenSearch Description Document
with template for granule queries
Granule Search

12. Templates from OpenSearch Description Documents enable granule query construction
Dataset Query Engine
OpenSearch Description Document Store
Granule Query Engine
Client
dataset query
Dataset
Discovery
dataset results
OpenSearch Description Request
OpenSearch Description Document
Granule Search
granule query
granule results

13. A client can be as simple as an XSLT
Attach a stylesheet to the Dataset OpenSearch Description Document
Renders the document in the browser as a search form

14. Several groups are developing both servers and clients
Goddard Earth Sciences Data and Information Services Center
National Snow and Ice Data Center
Global Hydrology Resource Center
ACCESS-NEWS
MODIS Adaptive Processing System
EOS Clearinghouse
Clients
Mirador (GES DISC)
Talkoot (University of Alabama--Huntsville)

15. Next Steps Integration with services Develop and recruit clients
Format conversion, subsetting, standard data protocols (OPeNDAP, OGC)
Servicecasting: Atom-based approach to advertising services for ESIP data
Develop and recruit clients
Reference “micro-client” for testing and cloning