05 December, 2002HDF & HDF-EOS Workhop VI1 ESDIS Status Richard Ullman ESDIS Project

05 December, 2002HDF & HDF-EOS Workhop VI2 Outline ESDIS General Status HDF-EOS Plans Website

05 December, 2002HDF & HDF-EOS Workhop VI3 ESDIS Science Data Services Today EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. –EOSDIS finalizing preparations to support ICESat and SORCE –In all, EOSDIS is providing data processing, archival, and/or distribution for over 15 Earth science satellite missions. EOSDIS has set a new benchmark for data management. The total volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals.

05 December, 2002HDF & HDF-EOS Workhop VI4 FY94/95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 Science Operations Supported Heritage Missions T/P, UARS, TOMS, ERBE Terra ACRIMSat TRMM Landsat 7 QuikSCAT Aqua GRACE SAGE III JASON SeaWiFS Radarsat Products Distributed 15 Million Archive Growth Archive Volume has doubled each year for the past three years 2 Petabytes Science Operations Timeline

05 December, 2002HDF & HDF-EOS Workhop VI5 EOSDIS DAAC Data delivery FY’02 October 1, 2002 through September 30, 2002 Over 1.8 million distinct users 16,013,980 products delivered Notes to charts on following slides: –Distinct users includes users accessing DAAC web pages, including web-crawlers –Distinct user type is based on addresses of users or URLs –Product delivered is defined as the smallest deliverable unit of data –Product delivery breakdown is based on addresses of users receiving ECS and Non-ECS data –“FTP Delivery” are to URLs not mapped to specific domains –“Foreign Other” includes foreign addresses whose country is known but domain-type could not be determined

05 December, 2002HDF & HDF-EOS Workhop VI6 16,013,980 Data Products Delivered October 1, September 30, 2002

05 December, 2002HDF & HDF-EOS Workhop VI7 Over 1.8 Million Distinct Users October 1, September 30, 2002

05 December, 2002HDF & HDF-EOS Workhop VI8 ESDIS Status After years of development, ECS is operational and generally recognized as successful. Primary ECS development contract is essentially completed. ECS Maintenance and Development (EMD) will emphasize maintenance more than development. Already the majority of ESDIS budget has shifted from development to operations. Still two major areas of new capability (see posters at AGU for more information): –Data Pools –ECHO

05 December, 2002HDF & HDF-EOS Workhop VI9 HDF-EOS Plans HDF-EOS 5 development is nearly complete. Continue to maintain, port to newer operating systems, bug-fix. Need the advise of this community workshop - we will discuss this afternoon. –What tools or capabilities are now needed? HDF-EOS 2 and HDF-EOS 5 –When is it the right time to press EOS science teams to migrate to HDF-EOS 5? –What steps should NASA take to facilitate?

05 December, 2002HDF & HDF-EOS Workhop VI10 Data Pools Concept: The Right Data, the Right Way, Right Now! Value-Added Providers Data Providers Data Tailoring Workflow Management End Users User-defined Views, Presentations and Data Access Requests Dynamic Web and FTP Data Views, User-specified Data Access Other Data Pools Data Producers Data Services Data Items Data Service Developers Geo PIPE PIPE = Personalized Information Presentation Engine

05 December, 2002HDF & HDF-EOS Workhop VI11 General Capabilities –The Right Data: Data Location Groups, themes, bookmarks and views Navigation and machine-based location Science views (e.g., science metadata) Applications views (e.g., OGIS coverage server) Location aids (e.g., geopolitical overlays) External location-support services –The Right Way: Data Tailoring Data reduction, manipulation and reformatting services Virtual data products Workflow management and execution monitoring External tailoring services –Right Now: Rapid Access Low latency data transfers Secure remote file access On-the-fly data compression Automated request routing and load balancing Near real-time data

05 December, 2002HDF & HDF-EOS Workhop VI12 EOS ClearingHOuse (ECHO) ECHO is a metadata clearinghouse –A single Internet portal for Earth science metadata search –Index of data provider inventory-level data holdings metadata..ECHO is a data order broker –Forwards orders for data discovered to the data providers to fill. –Data providers retain customer fulfillment service ECHO is a data service broker –Registered service are associated with registered datasets –Four kinds of service association Advertised, Context Passing, Brokered, Order Option ECHO is an open client API for custom user clients –The EOS Data Gateway (EDG) is ESDIS’ ECHO client

05 December, 2002HDF & HDF-EOS Workhop VI13 ECHO Data Providers Current ECHO Data Providers –EOSDIS Core System DAACs (17% thus far) EDC Land Processes DAAC Goddard DAAC NSIDC DAAC –ORNL DAAC (100%) ECHO holds the metadata for over 3 million granules, and growing

05 December, 2002HDF & HDF-EOS Workhop VI14 HDF-EOS Tools and Information Web Site

05 December, 2002HDF & HDF-EOS Workhop VI15

05 December, 2002HDF & HDF-EOS Workhop VI16 Website status A resource for discovering about hdf-eos in particular. –This workshop series’ presentation archive –Links to hdf-eos tools Site has been revamped according to comments received at the last workshop. –Tools download page now has opportunity for user feedback. –Workshop presentations are keyword searchable New features planned –Better introductory material. Post and organize documentation of HDF-EOS. Better navigation to NCSA site for HDF –Host hdf-eos “web forum” Incorporate the listserv

05 December, 2002HDF & HDF-EOS Workhop VI17 HDF-EOS Profile Richard Ullman ESDIS Project

05 December, 2002HDF & HDF-EOS Workhop VI18 History September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities Dec ECS Engineering Support Directive to create HDF-EOS June 1996, HDF-EOS v1.0 library released Upgrades every 6 mo., –Current version 2.8 on HDF 4 HDF5 support (called HDF-EOS 5) beginning November 2000 –Current version on HDF5

05 December, 2002HDF & HDF-EOS Workhop VI19 HDF-EOS Data Objects Point Swath Profile (Swath subtype) Grid Zonal (HDF-EOS 5 only)

05 December, 2002HDF & HDF-EOS Workhop VI20 HDF-EOS 5 Based on HDF5, a complete rewrite of HDF4 with a different interface. –First released in Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. –Support same data structures –Added prefix ‘HE5_’ to HDF-EOS 2 functions. –Doesn’t preclude HDF5 functionality. –Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG

05 December, 2002HDF & HDF-EOS Workhop VI21 HDF-EOS 5 Functionality Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface Global (File), Group & Local Attributes External Data Files Subsetting Unix/Linux Support Threadsafe Version FORTRAN, C, C++ General Table Interface (proposed)

05 December, 2002HDF & HDF-EOS Workhop VI22 The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. STRUC. METADATA Global (file) Attributes HDFEOS SWATHGRIDPOINT Root -- “/” ADDITIONAL HDFEOS INFORMATION Top Level of HDF-EOS 5

05 December, 2002HDF & HDF-EOS Workhop VI23 SwathName Data FieldsGeolocation Fields Attribute Dataset Data Field.1 Data Field.n Longitude Latitude Group Attribute : Global Attribute : Local Attribute : CoLatitudeTime Shaded Objects are implemented in a fixed way. User doesn’t have direct access via the interface Each Data Field can have Attributes and/or Dimension Scales Profile Fields Profile Field.1 Profile Field.n Group Swath Structure

05 December, 2002HDF & HDF-EOS Workhop VI24 HDF-EOS Point Intended use –Discrete points in time and/or location. –Table of data linked to table of geographic information. –8-level Hierarchical, each level may contain indices to the level below

05 December, 2002HDF & HDF-EOS Workhop VI25 HDF-EOS Point Hierarchical links: – Every level in a Point data set must be linked into the hierarchy. – Before two levels can be linked, a link field must exist

05 December, 2002HDF & HDF-EOS Workhop VI26 HDF-EOS Swath Intended use –Across track scanning instruments. –Sounding instruments –Level 1: Geolocated Sensor Units –Level 2: Geophysical Parameters

05 December, 2002HDF & HDF-EOS Workhop VI27 “Longitude” “Latitude” Dimension Name: Geotrack Size: 21 Dimension Name: Geotrack Size: 21 Dimension Name: Scan Size: 16 Dimension Name: Scan Size: 16 Dimension Name: Track Size: 42 Dimension Name: Track Size: 42 Map1 DataDimension: “Track” Geodimension: “Geotrack” Offset: 1 Increment: 2 Map1 DataDimension: “Track” Geodimension: “Geotrack” Offset: 1 Increment: 2 “Brightness Temperature” “Time” HDF-EOS Swath Data FieldsGeolocation Fields

05 December, 2002HDF & HDF-EOS Workhop VI28 HDF-EOS Grid Intended use –Variables mapped on uniform space-time grid scales –Level 3 - Gridded single measurement parameters – Level 4 - Modeled or derived from multiple measurements

05 December, 2002HDF & HDF-EOS Workhop VI29 HDF-EOS Grid Xdim Size: 2000 Ydim Size: 800 Projinfo Altitude Size 30

05 December, 2002HDF & HDF-EOS Workhop VI30 HDF-EOS Grid Projections Supported Geographic Transverse Mercator Universal Transverse Mercator Hotine Oblique Mercator Space Oblique Mercator Polar Stereographic Lambert Azimuthal Equal Area Lambert Conformal Conic Polyconic Interrupted Goode’s Homolosine Integerized Sinusoidal Compression Methods Run-Length Encoding Adaptive Huffman Gzip

05 December, 2002HDF & HDF-EOS Workhop VI31 Product Levels Level 0 - Reconstructed, unprocessed instrument/pay- load data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed. Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data. Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent). Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data. Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency. Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements.

05 December, 2002HDF & HDF-EOS Workhop VI32 Resources –HDF-EOS on the web: –HDF and HDF5 on the web: –HDF-EOS and HDF via