AIRS Meeting GSFC, February 1, 2002 ECS Data Pool Gregory Leptoukh

February 1, 2002Gregory Leptoukh2 AIRS Mission Support at the GDAAC Data Support Team MODIS Data Support Team Atm. Dyn. SSI&T SW dev Data Support Team Hydrology Data Support Team Hydrology Data Support Team Hydrology Data Support Team Hydrology

February 1, 2002Gregory Leptoukh3 Outline What is Data Pool? How does Data Pool help the distribution? What is the Data Pool project status? WHOM for Data Pool Future plans

February 1, 2002Gregory Leptoukh4 EOS Data Pools Concept

February 1, 2002Gregory Leptoukh5

February 1, 2002Gregory Leptoukh6 Data Pool Goals Data Pool –Support increased electronic distribution –Reduce need to pull data from archive –Reduce need for order submissions –Give science and applications users timely, direct access to data, including machine access –Allow users to tailor their data views to more quickly locate the data they need

February 1, 2002Gregory Leptoukh7 Data Pool goals, cont. Spatial Subscription Server –Data Pool is populated by subscriptions –Subscriptions set product importance and expiration priority –Allow subscriptions to correctly qualify data on extended set of metadata attributes, especially spatial and temporal attributes –Increase subscription processing capacity –Design server to be extensible for new action types

February 1, 2002Gregory Leptoukh8 Initial Data Pool Parameters –Data Pool Sizing GSFC: 14 TB + (23 TB to help with data transfer to MODIS L2+ processing) –Contents Data files XML-based metadata files; consistent with BMGT DTD Browse files in jpeg format (via hdf2jpg) –Directory Structure /datapool/ / / / e.g., /datapool/OPS/ASTT/AST_L1B.001/

February 1, 2002Gregory Leptoukh9 Gateways to Data Pool Web: –GES DAAC Search&Order Interface (WHOM) – working on it –Earth Data Gateway (EDG) – no plans yet –ECS/Landover WHOM-like interface – delayed Anonymous FTP:

February 1, 2002Gregory Leptoukh10 GUIs and Utilities Spatial Subscription Server GUI - allows operators to place qualified (spatially, temporally, or by parameter) subscriptions for: 1) notification of ECS insert/delete/metadata update events ; or 2) distribution from the ECS archive Data Pool Maintenance GUI - allows operators to set or change Data Pool configuration parameters, including which data types are eligible for insertion, and to monitor Data Pool insert processes Update Granule Expiration utility - command line utility which allows operators to update the expiration date and optionally the retention priority of specific Data Pool granules Cleanup utility - a command line utility which allows operators to cleanup expired granules from the Data Pool database and disks Access Statistics utility - a command line utility which collects Data Pool access statistics from the FTP and Web Server logs, and summarizes this information by subscription ID in the Data Pool database Data Pool User Web Pages - allows end users to navigate data pool contents, and browse and download data products

February 1, 2002Gregory Leptoukh11 Current GDAAC DP status Hardware on the floor 14 TB of disk space mounted on REID Spatial Subscription Server has been delivered Data Pool insertion s/w + database to be delivered in February 2002 WHOM-like web interface delivery delayed GDAAC starts developing WHOM extension to Data Pool

February 1, 2002Gregory Leptoukh12 Initial capabilities Initial Data Pool capabilities: 1 Gigabit Ethernet One extra “X” distribution capability Approximately 2 weeks of data On-line data access via Web and FTP DAACs control Data Pool contents and data residency Initial Spatial Subscription Server capabilities: Spatial, temporal and PSA qualifiers Electronic (push and pull) data delivery

February 1, 2002Gregory Leptoukh13 Data Pool and DAAC DAAC does support the very DP idea Data Pool management tools too powerful – it easy to delete important (or even all) data Data Pool design was not coordinated with DAACs, and it would be difficult to use it as delivered. DAAC Unique Extensions (DUE) needed DAAC wants to use Data Pool to strengthen and optimize L1 data transfer to MODAPS

February 1, 2002Gregory Leptoukh14 WHOM for Data Pool Utilize existing WHOM paradigm, structure and software: Hierarchical Structure and Navigation: –simple point & click –flexible, alternative hierarchies (views) Pages Generated from Templates: –consistent look and feel throughout interface –configuration management made easier due to content being separate from software Modular Structure: –Easy to add new products Fast search Query Preview paradigm – no zero hits! Convenient Calendar View Geographical presentation of granule coverage Attribute filtering

February 1, 2002Gregory Leptoukh15 WHOM for Data Pool, cont. Several release approach: 1.Short term (2 months): have a usable prototype at the GDAAC 2.Slightly longer term (3 - 4 months): DP view prototype (in parallel) 3.Long term (6 months): have a version of WHOM with ECS archive and DP holdings integrated.

February 1, 2002Gregory Leptoukh16 Future GDAAC Plans for DP Port Level 1 browse to DP Port Level 2+ browse from SIPS to DP Tie user subscriptions with DP Tie subsetting capability to DP Tie data mining and similar activities to DP

February 1, 2002Gregory Leptoukh17 Down The Road? Expanded Data Pool Data Providers Other Data Pools Data Producers Data Services Data Items Data Service Developers Archive Hierarchical Storage Management (if required) Metadata Export View Access OGC Access ECHO User-specific Navigation Clients Data-driven Processing and Distribution (internal and external)