NPP / NPOESS Product Profile of HDF5 Richard Ullman NASA / Goddard NPOESS Integrated Program Office

NPP/ NPOESS Product Data Format Richard E. Ullman NOAA/NESDIS/IPO NASA/GSFC/NPP Algorithm Division System Engineering Data/Information Architecture

NPOESS NPOESS is the National Polar Operational Environmental Satellite System –Managed by the Integrated Program Office (IPO) of three agencies: NASA, DoD and NOAA. –Next Generation Sun-Synchronous environmental observations for operational use. –Continuity of measurements with the NASA EOS Launch circa 2014

NPP NPP key program objectives: –To provide NASA with continuation of a group of global change observations initiated by the Earth Observing System (EOS) TERRA, AQUA, and Aura missions –To provide the NPOESS operational community with pre-operational risk reduction demonstration and validation for selected NPOESS instruments, and algorithms, as well as the ground system –Launch June 2010.

NPP (13:30 LTAN) NPOESS C1 (13:30 LTAN) NPOESS C2 (17:30 LTAN) Instruments VIIRSXXX CrISXX ATMSXX Micowave Imager X OMPSXNadir Only CERESERBSXX SESSSEM Only Services Search & RescueXX A-DCSXX HRD BroadcastXXX LRD BroadcastXX Payload Complement Comparison

Mission Segments and Responsibilities Space Segment -NASA -IPO -NOAA Command, Control & Communication Segment (C3S) Command, Control & Communication Segment (C3S) Manage Mission Manage Satellite Operations Space/Ground Communication Data Routing & Retrieval Manage Mission Manage Satellite Operations Space/Ground Communication Data Routing & Retrieval Launch Support Segment (LSS) Launch Support Segment (LSS) Launch vehicle Launch Support Payload Processing support Launch vehicle Launch Support Payload Processing support Space Segment Spacecraft ATMS CERES VIIRS CrIS OMPS VIIRS CrIS OMPS Interface Data Processing Segment (IDPS) Interface Data Processing Segment (IDPS) Ingest and validate raw SMD Process RDRs, SDRs, EDRs Perform operations cal proc Distribute data records Ingest and validate raw SMD Process RDRs, SDRs, EDRs Perform operations cal proc Distribute data records Archive & Distribution Segment (ADS) Archive & Distribution Segment (ADS) Ingest & validate data records Manage archive Interface with users Ingest & validate data records Manage archive Interface with users Science Data Segment (SDS) Science Data Segment (SDS) Ingest and validate RDRs Support Climate Research Ingest and validate RDRs Support Climate Research

Interface Data Processing System The IDPS is a single production system for all NPP/NPOESS products –Delivery to institutional users (Centrals - NESDIS, AWFA, NAVOCEANO, FNMOC) and to CLASS –Low latency –Configuration controlled algorithms –Raw, Sensor Radiances, and Environmental Records

NPP/NPOESS Product Definition Product content is specified by the Algorithm Theoretical Basis Documents (ATBDs) consistent with program requirements and controlled by the NPOESS Algorithm Review Board (ARB) Product encoding is specified by the Common Data Format Control Book-External (CDFCB-X) controlled by the IPO Configuration Control Board. Product encoding for all delivered products is HDF5 according to a profile defined in the CDFCB-X.

HDF5 for NPOESS Hierarchical Data Format 5 (HDF5) is the format for delivery of processed products from the National Polar-orbiting Operational Environmental Satellite System (NPOESS) and for the NPOESS Preparatory Program (NPP). HDF5 is a general purpose library and file format for storing scientific data. Two primary objects: Dataset, a multidimensional array of data elements Group, a structure for organizing objects Efficient storage and I/O, including parallel I/O. Free, open source software, multiple platforms. Data stored in HDF5 is used in many fields from computational fluid dynamics to film making. Data can be stored in HDF5 in an endless variety of ways, so it is important to standardize how NPOESS product data is organized in HDF5.

Format Strengths Mature technology. The encoding technologies used in NPP/NPOESS data products are: –HDF5 product encoding: –XML product metadata: –Bit packed field representation of quality information: –Scaled integer compression: Consistent HDF5 group structure –Organization for each product is the same as all others. – Data “payload” is always in a product group within All_Data group. Allows for flexible temporal aggregation –Granules are appended by extending dataset dimension.

Format Challenges Geolocation appears in a separate product group and may be in separate HDF5 file. Field metadata, used to interpret data (similar to netCDF CF) are in separate product profile file. Quality flags must be parsed before they can be interpreted. Information needed for un-scaling scaled integers is not obvious. HDF5 indirect reference link API, used to link metadata to the data in NPOESS’ use is complex and not supported by all analysis COTS implementations.

Information Model UML Diagram

An Example Product Group In this example product group: –Five datasets constitute the product. –There are two common dimensions. –There are three congruent datasets. –Two datasets contain scale and offset values. –One dataset contains quality flags by element. –There are two granules in this aggregation. – Dimension “alongTrack” crosses the “granule boundary.”

Example extracted from VIIRS Sea Surface Temperature EDR Field NameDescriptionData Type DimensionsUnits BulkTemp Sea Surface Bulk Temperature 16-bit unsigned integer [ N*768, 3200 ] Kelvin / Unitless SkinTemp Sea Surface Skin Temperature 16-bit unsigned integer [ N*768, 3200 ] Kelvin / Unitless QF1_VIIRSSSTEDR Land/Water Background 1-bit[N*768, 3200 ]Unitless SST Skin Quality2-bitUnitless SST Bulk Quality2-bitUnitless Aerosol Correction 3-bitUnitless SSTBulkFactorsBulk SST Scale32-bit float[ N*2]Unitless Bulk SST Offset32-bit floatKelvin SSTSkinFactorsSkin SST Scale32-bit float[ N*2]Unitless Skin SST Offset32-bit floatKelvin N in Dimension is number of granules

NPOESS Product Group QF1_VIIRSSSTEDR Granule 1 Granule 0 crossTrack along Track SkinTemp Granule 1 Granule 0 BulkTemp Granule 1 Granule 0 S O S O SSTSkinFactors SSTBulkFactors S O S O Granule 0 Granule 1 Granule 0 Granule 1 Example Product Group

Dimensions Dimensions are defined for each field. Fields are related by congruency and common dimensions. Common dimensions are given the same name. One dimension crosses the granule boundary. When multiple granules are “aggregated” the “granule boundary” dimension is extended. Dimension names and attributes are provided in the product profile.

Scaled Integer Storage For storage efficiency floating point data values may be stored as scaled integers. To re-generate the data value, the dataset element must be multiplied by a supplied scale factor and an integer offset added. The scale factor and offset are provided, one pair for each granule as a separate dataset. The scale and offset value is the same for all granules produced with a given version of an algorithm - not dynamic scaling. The fact that a dataset is a scaled value and the association between the data dataset and the scale factor dataset is contained in the product profile.

Quality Flags bit 2-bit 1-bit Quality Flags by Element Most NPOESS products contain multiple indicators of quality on an element by element basis. Quality flags are associated by congruency (shared dimension) with a data array. Multiple Flags of less than 8-bits are “packed” into structures aligned on 8-bit boundaries.

Geolocation Geolocation products are constructed using the same conventions as SDRs and EDRs. Geolocation datasets have a congruence relationship with the same dimensions as the datasets to which they apply. The association between a data product with its geolocation product is made on one of two ways: –The geolocation product may be packaged as a separate product group within the same HDF5 file. –The name of a separate geolocation product file may be stored in the N_GEO_Ref attribute on the root HDF group. –Choice of “as a product group” or “as a separate file” is made upon order from the NPOESS IDPS.

Common Geolocation Fields for VIIRS Products Field NameCommentsDimensionsUnitsData Type StartTime since epoch 1/1/1958 [per scan or swath]microseconds 64-bit signed integer MidTime since epoch 1/1/1958 [per scan or swath]microseconds 64-bit signed integer SCPositionECR coordinates[per scan or swath]meters32-bit float SCVelocityECR coordinates[per scan or swath]meters/second32-bit float Latitude[per cell]degrees32-bit float Longitude[per cell]degrees32-bit float SolarZenithAngle[per cell]degrees32-bit float SolarAzimuthAngle[per cell]degrees32-bit float SensorZenithAngle[per cell]degrees32-bit float SensorAzimuthAngle[per cell]degrees32-bit float Heightgeoid or terrain[per cell]meters32-bit float SatelliteRange[per cell]meters32-bit float

Product Profiles XML documents provide definition of product fields. –Product Profile is delivered as part of the product documentation. –Contains metadata such as units of measure, dimension names, legend entries, etc –A separate profile per product, but each conforms to the same NPOESS Product document type definition (dtd) and XML schema definition (xsd). –A style sheet is provided that can render the profile for a web browser. –Example:  VIIRS_SST_EDR.xmlVIIRS_SST_EDR.xml

Attribute NameTypeComments DataTypeString String format is: “ %d-bit %s ”, where %d is the number of bits and %s is one of: osigned integer ounsigned integer ofloating point o (a bitfield) DescriptionStringA descriptive text Dimension_GranuleBoundary Set of Boolean True (1) indicates that this is dimension extends when granules are appended. Dimension_Name set of string Name match indicates that this dimension is congruent with the same dimension names in other datasets in this product group. Field_NameStringThe name of the HDF5 dataset that contains the field values. FillValue_Name Set of string FillValue_Value Set of number Data type matches type of dataset. LegendEntry_Name Set of string LegendEntry_Value Set of number Data type matches type of dataset. 17 field attributes in XML product profile (1-9)

Attribute NameTypeComments MeasurementUnitsStringConsistent with SI naming and Unidata ’ s “ udunits ” package NumberOfDimensionsIntegerInteger greater than zero. NumberOfFillValuesInteger If zero, then no FillValue_Name and FillValue_Value attributes are present. Fill Values are used for primary data fields only. NumberOfLegendEntriesInteger If zero, then no LegendEntry_Name and LegendEntry_Value attributes are present. Legend entries are used for quality fields only. RangeMaxNumber Maximum expected value of field elements in the product, not just this dataset instance. Data type matches type of dataset. RangeMinNumber Minimum expected value of field elements in the product, not just this instance. Data type matches type of dataset. Scaled Boolean True indicates that the dataset is scaled. Note that fill values are in the dataset type and so must be tested before un- scaling. ScaleFactorNameString the name of the HDF5 dataset that contains scaling coefficients. To un-scale the elements, first multiply the scaled element by the first element and then add the second element. If the dataset is not scaled, Scale_AttributeName will not exist. 17 field attributes in XML product profile (10-17)